YT YouTube2Text

YouTube2Text → Transcripts → How to Deploy, Secure, and Automate Full-Stack Web Apps – Course for Beginners

How to Deploy, Secure, and Automate Full-Stack Web Apps – Course for Beginners — Transcript

by freeCodeCamp.org · 100,405 words · 13,243 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Taking a web application from local
0:01development to a live secure production
0:04environment can be daunting if you've
0:06never looked under the hood. In this
0:09comprehensive step-by-step course, you
0:12will learn to manually configure an
0:15Ubuntu server, set up essential runtimes
0:18and search services, and establish
0:21automated CI/CD pipelines. By doing
0:24everything hands-on before introducing
0:27automation, you will gain a deep
0:29foundational understanding of full stack
0:32deployment and how to reliably host
0:35complex systems. Immad Sadi developed
0:38this course. Hi everyone, I hope you are
0:41all doing great. In this course, we are
0:44going to talk about full stack
0:46deployment. This is a long course but
0:49trust me if you stick around and follow
0:52each chapter slowly you will learn a lot
0:55of things because I try to do things
0:57manually and to go step by step even if
1:00you haven't tried anything and the
1:03problem that here that I'm trying to
1:05solve and that I will try to teach you
1:07is how to take a web application that
1:09you have developed locally and deploy
1:12it. I have done this exercise three
1:14times for websites that I developed for
1:17myself and that I deployed for users to
1:20use and I will try to teach you all the
1:22things that I have learned across
1:24different domains. So let's look at the
1:27slides and let's see what we will learn.
1:30Let's define the problem in detail.
1:32First you have your web application.
1:35That's the prerequisite for watching
1:37this course. You can develop it using
1:40Python, Vue.js JS or other web
1:42frameworks and you want this application
1:45to be used. Here we have users so they
1:49they need to access the web application
1:51but they can't because you are running
1:54it locally on your laptop so they can't
1:57access it. What do you need to do? You
1:59need to take your source code and deploy
2:02it somewhere. So this is what we call
2:04the production environment. Here you
2:07rent a server where you can launch your
2:10application code. You can use platforms
2:13like Digital Ocean, Cloudflare,
2:16Lenode or others. There are plenty of
2:18platforms that we can use. And now your
2:22users can access the application that
2:24you have deployed and they are going to
2:26be happy. This is the problem. How do we
2:29do this? We have the web application. So
2:31we need a few steps to follow in order
2:34to deploy the web application and this
2:37is exactly the problem that we are going
2:38to solve in this course. What is the
2:41solution? In our case, we are going to
2:44rent as I said a server from let's say
2:48digital ocean, but you can do the same
2:51thing with other platforms. And this
2:54server can be either virtual or
2:56physical. We are going to explain this
2:57later. And some of these managed
2:59platforms simplify the process for us.
3:02So for example, you can have your code
3:04on GitHub and you just connect it to
3:07this to one of these platforms and they
3:09will deploy the code for you. You have
3:11platforms like Netlifi, Cloudflare,
3:15Lenode as I said and others. But the
3:18problem if you are a beginner and go
3:20directly to one of these managed
3:22platforms is that you won't understand
3:25exactly what is happening under the hood
3:28because they have a layer of abstraction
3:30that removes all the burden and all of
3:34the things that you need to configure
3:35manually. And this is great for people
3:38that have the experience with Linux and
3:41have deployed things manually across the
3:44years. But if you are be a beginner, I
3:47will highly suggest to not start with
3:49these platforms and try to do things
3:51manually to understand what is happening
3:54because transitioning from a manual
3:56process to using one of these platforms
3:59is very easy. So we are going to look
4:02behind the scenes here. As as you can
4:04see from the slide, I am banning these
4:06platforms because we are going to rent a
4:09server, a brand new server. We are going
4:12to install the operating system on it.
4:15In this case, we are going to use Abuntu
4:17and I will show you how to install the
4:20required software. So, we can install
4:22Python, NodeJS and other software that
4:26we need to run the application. And we
4:28will focus also on making sure that the
4:31web server is secure and that no one can
4:34access it. And we will see other things
4:36as well in this course. And here I want
4:38to mention or I want to emphasize this
4:41core philosophy that we will follow
4:43during the course is that before we
4:46automate anything we have to do the
4:49process manually because that will teach
4:51us why we need automation because if you
4:53don't do that you won't see the benefit
4:55of automating processes and also I have
4:58developed a book that goes uh with this
5:01course is uh not something the
5:04theoretical as I said I have applied
5:06this three times s on three websites
5:09that I have deployed and I am trying to
5:12give you all that knowledge in a
5:14distilled format. So I used a lot of
5:16resources like Google, I read books, I
5:20also found some code snippets and I
5:22filtered all of that into one simp one
5:26course that you can follow with all the
5:28things that I have learned and also the
5:30book if you want to read it. And as I
5:33have said, I have worked on this course
5:36for more than 8 months now to try to
5:39give you something that you might find
5:41helpful and to go in to see everything
5:44manually and to learn about everything
5:46that happens behind the scenes. And in
5:49this course, I will focus on deploying a
5:53specific web application that I have
5:55used and that you can find the source
5:56code online on GitHub that uses fast API
6:00in the back end and Vue.js JS in the
6:02front end and as an extra layer we added
6:05a search engine using M search but this
6:08is just an example to show you that you
6:10can deploy a complex system but if you
6:13are using Ruby or other programming
6:15languages the process will be the same
6:17you just need to change the software
6:19that you need to install and the
6:21commands to run it. This course is
6:24divided into six modules. In the first
6:27one which I call the foundation we are
6:30going to as I said in this case we are
6:32going to use digital ocean but you can
6:34do the same thing with the other
6:35platforms. We are going to rent a
6:37server. I will teach you what is an SSH
6:40key. This is something that we are going
6:42to use in order to authenticate to uh
6:44the machine that we are going to rent.
6:47And here something very important. This
6:49is for security reasons. We are going to
6:51remove the ability to connect using a
6:54password. Instead, we will only allow
6:56using an SSH key. And we will also block
7:00users from connecting as the root user
7:04because if someone is able to use that,
7:08they might do something nefarious in
7:10your server and we don't want to allow
7:12that. Instead, we are going to create a
7:14new user that you can use in order to
7:16connect to the virtual machine. And on
7:18top of that, we are going to configure
7:20firewall to block anyone who is trying
7:24to access your your web server. Why is
7:26that? Because once you once you create a
7:30web server, it is going to be available
7:31online and anyone can see it. So we need
7:35to set up the firewall and block people
7:37that are trying to access your web
7:40server. In this second module which is
7:42called the application runtime, here we
7:45are going to work with our code. So we
7:47are going to take it from let's say
7:49GitHub or from our laptop and we are
7:52going to deploy it to the server. So we
7:55are going to copy it and paste it there.
7:58Because we are using Python, we need to
8:00install we need to install it and
8:02configure the virtual environment. And
8:04something very important here. We are
8:06going to start with a small server. It's
8:09not it is not going to have a lot of
8:11RAM. So we will end up having some
8:14problems with memory. So we will
8:16eventually when we try to install
8:18software that needs compilation that
8:20will that will make the uh Linux run out
8:23of memory. So in that case we are going
8:26to learn something new which is swap
8:27memory which will allow us to take some
8:31some space from the SSD and help to in
8:35order to help us to install some some
8:37software that requires a lot of RAM. And
8:39in order to manage the back end we are
8:41going to use a mix of these software. We
8:44are going to use supervisor, J unicorn
8:46and UV core. Don't worry about that. We
8:49are going to learn why do we need them
8:52in module 2. So I'm just showing you
8:54what the things that we are going to
8:56learn. And after that we are going to
8:58use engineext as a reverse proxy and we
9:01are going to connect it to the back end
9:04using Unix sockets. In the third module,
9:07we are going to focus on media search
9:09because this is the data that we that
9:12the web application needs. I will show
9:14you how to export and dump the data that
9:17we have locally so that you can take it
9:19from your local machine and put it on
9:21the server. We are going to self-host
9:23mail search. We are not going to use a
9:26cloud offering that will give us the
9:28ability to get me search. No, we are
9:31going to install it and we are going to
9:32manage it on the same server. And here
9:35we are going to create another user that
9:37will manage me search for us. And we are
9:39going to use systemd to create a
9:42background service that will launch me
9:45search for us and that we can use in
9:47order to manage me search. After that we
9:49are going to use or we are going to
9:51create a batch script that will
9:53automatically create backups for us so
9:55that if something bad happens we can
9:58just read the backup and load it in
10:00order to continue. And finally, we are
10:03going to I will show you a nice
10:05dashboard that you can use in order to
10:07manage the data you have on me search.
10:10And we are going to use something very
10:12awesome which is called SSH tunneling
10:15that will allow us to connect this
10:17dashboard to your server without having
10:21to open a port or something like that.
10:25In module four, we are going to talk
10:27about global delivery and app security.
10:31And here we are going to start by
10:33registering a domain and configuring
10:36DNS. After that we are going we need a
10:39certificate so that we can get HTTPS
10:42that will make uh the connection between
10:46your users and your web server secure
10:48because if you don't do that you they
10:50will only see HTTP and their browser
10:53will flag that connection as insecure.
10:56And with let's encrypt the certificates
10:59are only valid for 3 months. So here I
11:02will show you how to create a script
11:03that will automatically renew the
11:05certificate for you without having to
11:07run any commands. After that we are
11:10going to use cloud player to serve our
11:13front- end assets because it is static
11:16and that will help us reduce the latency
11:19across uh the world because when when
11:23you pick a server you pick a location in
11:25in my example I picked Germany. So that
11:28means people from China or other
11:31countries that are very far away they
11:33are going to see high latency. In order
11:36to fix this problem, we use CDNs and in
11:39this case we are going to use cloud
11:41player. In module 5, we will focus a lot
11:43on automation. Here we are going to
11:46connect to GitHub because up to now we
11:50have cloned the project using a command.
11:55So we copied the code and we pasted it
11:57on the virtual machine and we ran the
12:00commands to start the back end and front
12:01end servers etc. We did everything
12:03manually. But in this module, we are
12:06going to automate that. So we will push
12:09the code to GitHub and we will just hit
12:11a button to deploy the web application
12:13without having to go to the server. So
12:15this is what we are going to learn here.
12:18But first we need to learn how to
12:19protect a branch. So if you are working
12:22on a codebase, you have let's say your
12:25main or master branch, you should never
12:27push the code directly to it. You need
12:29to create a new branch and then merge
12:31the code. So I will show you how to
12:33protect that main branch so that you
12:35cannot so that you don't push code that
12:38is broken accidentally. After that we
12:40are going to use a project called
12:41pre-commit that will allow us to create
12:45some rules that we can run when we
12:47commit code. For example, one of the
12:50rules that we can add is checking
12:51linting and formatting so that if your
12:53code is not formatted well, you will not
12:55be able to commit. After that, we are
12:58going to create a pipeline using GitHub
13:00actions. And that will allow us to run
13:03the same um tests like linting,
13:06formatting, running unit tests, etc.,
13:10but on GitHub. And that will allow us to
13:13block a merge request or a pull request
13:16if it doesn't satisfy all these tests.
13:20And that's how we can detect if our code
13:22is broken or not. If it isn't, if
13:24everything is green, then we can merge
13:26to master and from that we can deploy
13:28and we will do other things. Uh we will
13:30add integration tests, end to end tests,
13:33etc. So don't worry about that. We will
13:35focus a lot. Uh this this model is going
13:37to be very long, but it is going to be
13:40very very useful. And finally in the
13:43last module, we will focus on
13:45optimization and maintenance. Here I
13:47will show you how to read the enginex
13:49logs and display that data nicely using
13:51a project called go access. After that I
13:54will show you how to monitor the memory
13:56usage using btop++. This is an amazing
14:00project. I really like it and you will
14:02see once we once you reach module 6 you
14:04will see how btop can help us understand
14:08what is happening on our web server and
14:11I will show you how to ban IP addresses
14:14that are hammering your website. So
14:16sometimes when you analyze the logs
14:19using go access you will see some IP
14:21addresses that are trying to reach your
14:24server thousand um more than a thousand
14:27times a day or a minute and that's that
14:29is not human. So a human will not will
14:32not go to your website and try to
14:34refresh the page a thousand times.
14:35That's impossible. So these are
14:37automated bots that are trying to steal
14:39credentials or to see if there are
14:41weaknesses in your web server. So we are
14:43going to block them in order to first
14:46not waste time serving them things that
14:48they they are not going to read and to
14:51yeah reduce the the burden on our
14:54server. And we will also use an amazing
14:56tool called locust. This is a Python
14:59library that allows us to simulate users
15:02trying to access our website because
15:04this will allow us to estimate how many
15:07requests per second or per minute we the
15:10server can sustain before crashing or
15:13not being able to serve as many users as
15:16possible. So this is going to allow us
15:18to understand the weaknesses of our
15:21server and we will do other things. is
15:23we will analyze the disk usage um
15:26perform some log retention policy
15:28changes etc. So as you can see this
15:30course is packed with information. So it
15:34is going to be more than 10 hours long.
15:36So go through it step by step and don't
15:38try to rush. I I have tried to make it
15:41accessible to anyone. So even if you are
15:44a beginner don't worry about that
15:46because this is why I have made the
15:48course in order to teach you how to
15:50deploy things. And at the end you will
15:52see that you can also go automate uh
15:56more stuff but I just try to pick your
15:59interest and show you what you can do.
16:01And after that as I said if you don't
16:03want to do this process manually at
16:05least you did it once. So now you
16:07understand after that you can use
16:08platforms like Netifi get GitHub pages
16:11to deploy your applications without
16:13having to worry about renting a server
16:16or something like that. And in terms of
16:18resources don't worry about that.
16:20Everything is open source. I will make
16:23sure to put the links in the video
16:24description. You have two GitHub
16:26repositories. The first one, which is
16:28what I'm showing you right now, contains
16:31the code that we will deploy and the
16:34second GitHub repository contains the
16:36content of the book. So if you want to
16:40just read and you don't prefer watching
16:42videos, you can go to that repository,
16:46click on the book folder and you'll find
16:48all the chapters right there with the
16:50code. Everything everything is there in
16:52a nice format. And if you need some help
16:55or if you just want to say something
16:58after watching the course, feel free to
17:00connect with me. You can find my
17:02LinkedIn on the video description or you
17:05can send me a an email if you want to.
17:08You will find all my information in the
17:11video description. I hope that I have
17:13motivated you enough to watch the
17:15course. I hope that you enjoy it and
17:17that you learn a lot from it. Now, let's
17:19start with the first module.
17:22Welcome to module one where we will set
17:25up the foundation of the cloud
17:27environment. This is where your code
17:30will live. In this sub chapter, we will
17:33focus on cloud setup and access. Let's
17:36get started. Before we deploy any code,
17:40we need a secure environment. In this
17:44course, I will be using digital ocean.
17:47But these concepts apply everywhere.
17:50Whether you are using AWS, Lenode, or
17:54even a Raspberry Pi at home, the Linux
17:57commands remain the same. By the end of
18:00this video, you will have a server
18:03running Ubuntu 2404 LTS. We will
18:07generate secure SSH keys, connect to the
18:12server, and most importantly, we will
18:15completely disable password
18:18authentication.
18:19This ensures that your server is locked
18:22down from the very first moment. Now
18:25let's go to digital ocean in order to
18:28create our first droplet.
18:31Before creating the droplets, I want you
18:34to know that we have a GitHub repository
18:37where you will find everything I'm
18:40sharing with you in this course. So for
18:42example here inside the book folder we
18:45have the first subchapter and you have
18:49all the steps here. So if you prefer
18:52reading articles you will find those
18:55inside the book folder and the slides
18:58and everything else are available in the
19:01repository as well. You will find the
19:03link in the video description. If you
19:06have never used Digital Ocean before, if
19:09you click on the link in the video
19:10description, you will be able to get
19:13$200 for free just to test their
19:16platform. And that amount is available
19:18for 60 days. Click on get started. That
19:22will take you to a login page where you
19:26can create your account. In my case, it
19:29it logs me directly because I already
19:31have an account. And here we need to
19:34click on projects. As you can see, I
19:36already have two projects. I have my
19:39personal website is hosted here. And in
19:43your case, this list will be empty.
19:46First, click on new project.
19:50In this form, give the project a name.
19:53Select the environment type and the
19:55purpose of this project. I will call the
19:58project test. The environment type,
20:02let's say development. Here in the
20:05purpose drop-down menu, I'll select it
20:08web application and the description is
20:10optional. So click on create a project.
20:14Let me zoom in a little bit. Here they
20:16say move resources into test. So here we
20:20don't have any resources. So we will
20:22skip this for now. And as you can see
20:25the project has been created. And now we
20:28can create resources. And the resource
20:32that we are going to create is a
20:34droplet. Droplets are machines that that
20:37can run our code in the cloud. This
20:40means that we will take our
20:43application's source code put it on this
20:46droplet that is hosted by digital ocean
20:49and the code will be running inside
20:52inside of that machine. So now you can
20:55click on create droplets or you can do
20:58it from here. Create droplets. It's the
21:02same thing.
21:05Now we are in the create droplets page.
21:08The first thing that we need is to
21:11choose a region. For example, in my
21:14case, I live in Morocco. So I will
21:16choose Frankfurt because Germany is
21:20close to Morocco. That's why I chose it.
21:23you can do the same thing. So for
21:24example, if you live in Europe, you can
21:26select either London or Frankfurt or
21:29also Amsterdam. Those three will work
21:31perfectly. But if you are trying to
21:35deploy this application next to users in
21:38another country, then try to choose the
21:40the region based on where your users are
21:44going to be uh located. And here for
21:47example in Frankfurt, we have just one
21:49data center. So there there are there
21:51aren't other options. Next, choose the
21:55operating system. Here we have a lot of
21:58options, but I will select Ubuntu. And
22:02here we have plenty of versions. I will
22:04choose the LTS version, and I will
22:07select the latest one. Now you need to
22:10choose the size of your virtual machine.
22:14You can pick between shared CPU and
22:18dedicated CPU. Inside each option, you
22:22must decide the type of of storage, how
22:26much memory you want, uh how many CPU
22:30cores you want to have in this machine,
22:32etc. And here if you select dedicated
22:35GPU CPU, uh you will see that the price
22:39increases drastically because inside
22:43this option, the resources are reserved
22:46only for you. But shared CPU, you will
22:49be sharing resources with other people.
22:52That's why it's not as expensive as
22:55dedicated CPU. Just for testing
22:58purposes, select choose shared CPU,
23:02regular, and then select the lowest
23:06tier, $4 a month. And if you have just a
23:09simple web application that you want to
23:11host, then this option is fine. uh for
23:16example I have de deployed many websites
23:18using just this plan and it works
23:21wonderfully if you see that the
23:24application can't handle this the high
23:26traffic that you are seeing after that
23:28it's possible to to to get more
23:31resources so this doesn't if you select
23:33this now it doesn't mean that you are
23:35stuck in this offer and you'll have to
23:37recreate everything from the beginning
23:40no you you can still uh upsize the the
23:43droplet and add more resources If you
23:46need next, so here we have additional
23:48storage. You can see that here we get 10
23:50GB of storage. If you think that this is
23:53not enough, you can click on add volume
23:56and select the amount that you want to
23:58add. But this is not free. So you will
24:00be paying $1 per month per 10 GB. In my
24:04case, I don't need this, so I'll just
24:06remove it. 10 GB is enough. Uh if you
24:10want to have backups, you can click on
24:12this. And again, this is not free. So uh
24:16if you need this enable it and choose
24:19the right options but in any I don't
24:22think this is needed for testing
24:24purposes. So uncheck it. And now we
24:28arrive at the authentication section. So
24:31here we have two options. We can use SSH
24:35keys which is as you can see the default
24:37option selected by Digital Ocean or if
24:40you want you can use passwords. But as
24:43you can see here, we have a warning that
24:45says please store your password securely
24:48because if because passwords can be cast
24:51if they are not strong, but SSH keys are
24:54much stronger. So let's use SSH keys and
24:58I'll show you how to create one uh next.
25:01So now we need to create an SSH key. So
25:04I already have one here, but I will
25:06create one by clicking on this new SSH
25:09key button. You can see that they give
25:12us here the steps that we need to
25:15follow. I will show you how to do that.
25:17Now one once we generate the SSH key, we
25:21will put it here. Give it a name and add
25:24it. But before I create this, I want to
25:26go back to the slides in order to
25:28explain how uh authentication with SSH
25:32keys work. Let's look at what actually
25:36happens when you try to connect to your
25:39droplet. As I said, we will be using SSH
25:42keys instead of passwords because they
25:44are much harder to steal. The process
25:48starts with two files that we call
25:50private key and public key. And as you
25:54can see, we have two containers. This
25:56one represents your machine, your laptop
26:00that you have next to you. And here we
26:02have the server that we are creating on
26:05Digital Ocean or you can create on AWS
26:09or other cloud providers. The private
26:11key lives on your laptop but the public
26:14key will be inside the server. So the
26:18private key here think of it as your
26:20physical house key. You never share this
26:23with anyone. So this needs to be kept
26:26secret. Then you give the server a
26:31public key. Think of this as the lock on
26:34the door. Anyone can see the lock, but
26:37because they don't have the key, they
26:39cannot open it. When you run the SSH
26:42command, your laptop contacts the server
26:46and says, "Hello, I would like to
26:49connect using this specific ID." When
26:52the server receives this request, it
26:55replies with a challenge because the
26:58server is suspicious. It it doesn't know
27:00if you are the real person or if this is
27:02a hacker that wants to hack the the
27:04server and extract valuable information
27:07from it. So the server sends a challenge
27:11basically a random string of characters
27:13that we call a nons and basically says
27:17prove you own that private key. sign
27:20this random string for me. So if the the
27:24requesttor is legitimate, that means
27:28that it has the private key, it will be
27:31able to basically sign this string and
27:35send it back to the server. So here the
27:40the laptop will say or the real person
27:42will say I signed this using my private
27:44key. Here is the signature. Now the
27:47server will receive this and will try to
27:50do the math to verify if this is uh if
27:53the private key is correct. If that's
27:56the case, the server will grant access
27:59because it verified that the signature
28:01is correct. Otherwise, it's going to
28:04block this request and the anyone except
28:08you will not be able to log to your
28:11machine. So this is how SSH SSH
28:14authentication works. And now I'll show
28:17you how to create the these two pairs of
28:20keys uh the private and public keys.
28:23Paste this command. As I told you if you
28:26go to the to GitHub you will find this
28:31markdown file and here you will have
28:33everything. Um, so if I scroll down a
28:35little bit, you can see that I'm here we
28:37we have the same steps that we followed
28:40and here you can see that we had the
28:42commands. So don't worry about them. You
28:44will find the everything in here so that
28:46we can just copy copy them easily
28:48without typing them manually. So the SSH
28:52key gen command is used to generate SSH
28:56keys and we have plenty of algorithms
28:59that we can use to generate these pairs
29:01of keys and one of those is called
29:04ED25519.
29:06So this is the one that we are going to
29:07use. It's faster and more secure than
29:11RSA but you can also use RSA if you
29:14want. And here - F is basically the file
29:18name. All SSH keys are stored in the SSH
29:22directory. And if you don't give it a
29:24name, you might risk overriding your
29:27keys. So here I will call it test SSH
29:32key. And here we can add a comment. This
29:36is helpful if you want to identify the
29:38keys. And now I can hit enter. And as
29:42you can see it says generating public
29:44key with this algorithm. And here I need
29:47to enter a passphrase for this specific
29:51key. And here they tell me if I don't
29:53want to add a passphrase, I can just hit
29:55enter to move to the next uh step. When
29:59should you add a passphrase and when
30:02should you ignore this? I strongly
30:04recommend adding a passphrase for your
30:07personal key. So if you want to log to
30:10the machine, type some commands etc. do
30:14do some maintenance in that case you
30:16will be logged in manually. So in that
30:19case it's great to have a passphrase
30:21because even if someone was able to get
30:25your key they won't be able to login
30:27because the server will ask for a
30:29passphrase. Basically it's another layer
30:31of security that will prevent anyone
30:34from using your ser or accessing your
30:37server. But if you want to have a a
30:42machine or a script that will
30:44automatically deploy your application,
30:47in that case you don't want to have a
30:48passphrase because the script will be
30:51blocked. I will show you how to create
30:54two keys. So this one that we are
30:57creating is going to be used manually
31:00but we'll have another one that will
31:02call it let's say GitHub or something
31:05that we will use in GitHub to deploy our
31:08application automatically and in that
31:11case we don't need a passphrase. So for
31:13now I will add the passphrase and I will
31:15show you that we will be asked to enter
31:18it when we want to log the droplets. I
31:23have entered the passphrase and here
31:26they tell me to enter it again. Great.
31:29So as you can see the key random image
31:32is so that means that the the keys were
31:35generated. As you can see I have listed
31:38all the keys inside the SSH directory
31:41and I search just for this and you can
31:44see that we have two files. This one pub
31:48means public key and this one is the
31:50private key. This one should stay on
31:52your laptop but this one we will copy it
31:55and put it inside the droplet. Great. So
31:59now copy this and we will use the catch
32:03command to see the content of this
32:06public key. So here make run this and
32:09copy everything and after that I will
32:13show you where to put that uh on digital
32:15ocean. I have copied the key and now I
32:18will put it here and I will give it a
32:21name. So I will start by entering the
32:24name. So test SSH key manual. So here
32:30you paste your SSH key and after that
32:32click on add SSH key. I have done that
32:35and as you can see I have another SSH
32:37key here. And now if I want to add
32:40another key, I can click on new SSH key
32:42and repeat the same steps in the
32:45terminal to generate another one. But
32:47make sure to have another name and to
32:51not enter a passphrase because that one
32:54we will use later to deploy the
32:57application automatically. Or if you
32:59don't want to do this here, I will show
33:01you how to do it within the droplet.
33:04Here we can select this first option add
33:07improved metric monitoring. It's free so
33:10why not? And here finally we have this
33:14section. So here we can create multiple
33:16droplets if you want to but in our case
33:20uh one is enough. You can change the
33:23host name. So this is the name that will
33:26appear inside the droplet. We can we can
33:30change that. Let's call it uh let's keep
33:33it as Ubuntu test project. If you have
33:37multiple droplets, using tags will help
33:40you filter them quickly. And after that,
33:43we should assign this to the test
33:45project that we created. So everything
33:48is okay. Now click on create droplet.
33:53Open your list of projects and click the
33:56project that you have created. As you
33:58can see here is the droplet and here
34:01this is the running indicator. So this
34:04means that this droplet is in the
34:08creation phase but once this turns green
34:12it means that it's running. You can
34:14click on the name and as you can see now
34:17it's running. We have no data because we
34:20have just created this. You have your IP
34:23address. You have everything here. And
34:26as you can see, if you want to upsize
34:29this later, you can click on the upsize
34:31droplets button. And here you can in add
34:34more memory, more storage or in uh
34:38increase the the number of CPU cores if
34:41you want to. So I have refreshed the
34:43page and as you can see now it says
34:45active instead of uh new. Great. Now I
34:49want you to copy this IP address. Just
34:52click on the copy button. And you might
34:55be wondering why am I blurring the IP
34:58address. Even though this is just a test
35:01droplets that I will delete later, it's
35:04a good habit to keep your server IPs
35:06private to prevent unwanted traffic or
35:10attacks. So, copy the IP address and
35:13open your terminal. We are going to use
35:16the SSH command to SSH to the droplet.
35:21So first we type SSH
35:24I and here we need to specify the
35:28private key and the keys are stored in
35:32the SSH directory and here I have called
35:35mine test SSH key and after that we
35:38specify the user that we want to log in.
35:43So in the machine we have uh by default
35:45we'll have just the root the root user
35:47but after that we'll I will show you how
35:49to create other users and here we type
35:53at and we paste the IP address. Now hit
35:58enter and you should be able to login.
36:02So because this is the first time we are
36:05trying to login, we get this warning.
36:08And here you can just type yes. And as
36:11you can see, I'm not able to log in uh
36:14because I have added a passphrase. I
36:17need to specify it before logging in. So
36:21now I will add my passphrase. And here
36:24on Ubuntu there, they we have this
36:27option that will automatically unlock
36:28the key. Um so if you want to login
36:31quickly, that's fine. and just click on
36:33unlock and as you can see I am in uh
36:36let's verify so this is the memory usage
36:39it's has 40% we don't have any swap
36:42memory this we are using 22% of the disk
36:46space and yeah it seems to be working
36:50you can see that we have some updates
36:52that we can perform and that's great so
36:56as you can see we are logged in as the
36:58root user and this is the name that we
37:01have specif ified when we created the
37:03droplet. So everything seems to be
37:06working. Okay. So since we are logged in
37:09as user, we can type in sudo app update.
37:14So this is a an abuntu.
37:17So this is a machine running Ubuntu and
37:19this is how we update it. Okay, good. So
37:22now we will type sudo upgrade.
37:26Okay, the type yes and hit enter to
37:29start installing the packages. If you
37:32see a configuration screen during the
37:35update, select keep the local version
37:39currently installed. Don't don't select
37:42any other option because we want to
37:44preserve the working SSH configuration
37:47that Digital Ocean set up for you. In
37:50the Digital Ocean page, we added a SSH
37:55key manually. So that updates the SSH
37:58configuration and we want to keep that
38:00but also add the new modifications that
38:02this update process is trying to
38:05perform. Keep the differences and yeah
38:08let the the updates continue. Great. So
38:12the update has finished. Now we need to
38:16reboot the machine. So just type the
38:18reboot command. This will complete the
38:22upgrade process and start using the new
38:24kernel and packages. So you can see that
38:28the connection is closed. So I was uh
38:31ejected from the server because it needs
38:33to reboot. But don't worry, I can run
38:36the SSH command again to login back to
38:40the machine. So let's wait a little bit
38:43or let's go back to Digital Ocean just
38:45to check uh if it is ready. So I'll
38:49refresh the page. Okay, it says it says
38:51active. Let's hit enter. And voila, I am
38:56back. And as you can see, it says zero
38:59updates can be applied immediately. This
39:02confirms that the update process works
39:06perfectly. Now we are going to create a
39:09nonroot user and configure SSH access.
39:14Why do we create new users instead of
39:17using the root user? Running everything
39:20as root is dangerous because sometimes
39:23you can make mistakes or if you don't
39:27know Linux a lot and you might be using
39:30LMS without knowing the commands that
39:32they give you, you might do something
39:34that will maybe corrupt your file system
39:37or do something irreversible and that's
39:40not good. And if a hacker was able to
39:44log into your machine, he can do every
39:46anything. We want to limit the things
39:49that we can do inside the VM by creating
39:53a nonroot user. And if you want to
39:56perform something that requires pseudo
40:00privileges, we can give the user that
40:03ability momentarily. So the command to
40:06create a user is add user. I will call
40:11that user Emad. So that's my name. Uh
40:15make sure to change that. And as you can
40:17see now we need to provide a password.
40:20This means that if I try to to run the
40:24same command sudo opt update the Linux
40:28machine will ask me to enter my
40:29password. So this is really great
40:32because let's say that someone wants to
40:35login with this specific user that I
40:37created, they will not be able to do
40:40commands that require sudo privileges
40:42because they don't know the password. So
40:45create the password uh retype it again.
40:49And here you can add additional
40:50information like the full name. So let
40:53me just add that uh room number. I don't
40:56need that. For the things that you don't
40:59want to add anything, uh, just hit
41:03enter. And finally, let's type yes to
41:06create the user. So, as I said, we want
41:09this user to be able to run pseudo
41:11commands. So, we will use the user mod
41:14command. And here we will type the the
41:18user. And now we can see that we are
41:21still logged in as the root user, but we
41:23want to log in as this user. As you can
41:26see that I'm inside the the droplet. In
41:30order to exit, I can type exit and hit
41:32enter or I can just hit control and D.
41:37Good. So the connection to this server
41:40is closed. Now I can go back and instead
41:44of uh login logging in as root I will
41:48use my new user. Hit enter. And as you
41:52can see we were we weren't able to
41:54connect. Why is that? And here it says
41:56too many authentication failures. In
41:59your case, you might see just permission
42:01denied public key. In my case, I see
42:04this because I have a lot of public
42:05keys. So the the server was or my laptop
42:09was trying to log in with multiple keys
42:13and it wasn't able to do that. So why
42:16aren't we able to connect as this new
42:20user? This happens because the SSH key
42:24that we added in digital ocean lives in
42:27the root user. This means that we need
42:30to take that SSH configuration and also
42:34give it to my new user. So let's log
42:38back as the root user and we can verify
42:41this. We can go inside the root user and
42:44we have again the SSH folder and here we
42:48have authorized keys and authorized keys
42:51is a file. It's not a directory and we
42:55can verify this by typing ls-la
42:59and you can see that here we have a dash
43:01that means it's a file. If it says d
43:04that's a directory. So authorized keys
43:06contains all the public keys that we
43:08want to use on this droplet. In our
43:11case, it contains just one because uh
43:14when we created the droplets, we added
43:16one key. But we can add more if we
43:18wanted to. But my new user doesn't have
43:22this directory. So if I go to home and
43:25this user new user, you can see that
43:28there isn't an SSH directory. This means
43:32that we we won't be able to login as
43:34this new user. But luckily, we can solve
43:39this issue by just copying this SSH
43:43directory from the root user to this new
43:46user. And the command to do that is cp
43:50preserve mode. And here we want to copy
43:54this directory to this new user. And
43:58preserve mode is an option that we can
44:00add in order to preserve the file
44:03permissions. So you can see that here.
44:06Uh so let me just run this command. And
44:08instead of running ch mode 600 700 etc
44:13on this new directory. Preserve mode
44:16will do that for us. So in just one
44:18command we preserve the the file
44:20permissions. Who can access this file?
44:23Who can read it? Who can execute it etc.
44:25And if now I can look at this. So let's
44:30use ls and let's see what we have inside
44:34the sh folder. You can see that we have
44:38the authorized keys. That's good. But
44:40the problem is that if I look at who
44:43owns that file, you will see that the
44:46root user is owning this file. So we
44:49want to change the ownership and give
44:51that to this new user that you have
44:53created. And in order to do that, we use
44:56the ch own. And here I will also add
45:01capital R. And I want this new user to
45:06own that file. And here I'll go to home
45:09and SSH folder. And now if I go back to
45:13ls-la, you can see that we have that
45:17this user is owning this file. Now we
45:21can exit. Uh again hit Ctrl D go back to
45:26the previous command and here type in
45:29the new user not root and now hit enter
45:33and as you can see we are logged in and
45:36the prompt shows my user and not root
45:41and if I want to let's look at root dosh
45:47I'm not I'm not able to do that because
45:49that's not my directory but Because I
45:53have the pseudo privilege, I can type in
45:55pseudo and enter my password. And as you
45:57can see, I'm still able to look at
46:00things that I'm that I'm not supposed to
46:02look at or per do maintenance uh stuff
46:06like that. But if someone hacks your
46:08machine and is logged in as this user,
46:12new user, they won't be able to do this
46:14because they need to know the password.
46:17Awesome. Uh just one note, if this
46:20didn't work for you, if uh for example,
46:23you weren't able to uh look inside the
46:26SSH directory, that means that you did
46:29something wrong. Just rewind and make
46:32sure that this step works because in the
46:34next step, we are going to disable root
46:37login. We will not allow even us, we
46:40won't be able to login as root. We will
46:42only login as this new user. This will
46:46uh help increase security a lot because
46:49no one will be able to login as roots
46:51because as you saw root doesn't require
46:53providing a password and that's a big
46:56security problem. If someone is able to
47:01login as that super user let me show you
47:05how to stop this from happening. We will
47:08use pseudo again. I'm going to use nano
47:11as matx editor and I'm going to go
47:14inside the shd config.
47:18We want to change uh the following I
47:21will say parameter and in nano in order
47:24to search you can click on or press ctrl
47:28w paste the thing that you are looking
47:31for and hit enter. You can see that by
47:33default this value is set to yes which
47:36means that we can login as root but we
47:39want to change that to no. We also want
47:42to disable password authentication even
47:45though we have specified in the creation
47:49of the droplets that we want to use an
47:50SSH key but it's it's also a great idea
47:54to set it here and not rely on the cloud
47:58provider to do that for you. So these
48:00are the two things that you need to
48:02change. You need to change permit root
48:04login to no and password authentication
48:07to no. Now in order to save this you can
48:11click on Ctrl S or if you are on an
48:14older machine you can hit Ctrl O just
48:18hit enter and after that Ctrl X. You can
48:22verify that again if this works. Just
48:24enter. And you can see that the the
48:27values are now set to no.
48:30So now we need to restart the SSH
48:34service. And now let's test this. So I
48:37will type exit. And now I will try to
48:40login as root. Okay. Let's see what will
48:43happen. Uh so as you can see um my
48:45request was rejected because we have
48:48blocked root access to or we blocked
48:52logging in as root. But if I go back and
48:56try to login as my new user I am able to
49:00do that. So that's awesome. So now we
49:03have lots of layers of security. We have
49:06disabled password login. We are using
49:09SSH keys. And even if we use an SSH key,
49:13we need to provide a passphrase. And
49:15within the machine, we created a new
49:18user in order to prevent us from doing
49:21things that are unreoverable.
49:24And if we want to do things that require
49:26sudo privileges, we need to provide a
49:29password. So this means that if someone
49:32was able to breach a layer, they won't
49:36be able to uh do something harmful
49:39because if even if they have the SSH
49:42key, they need to provide a passphrase
49:44or even if they logged in, they need to
49:46provide uh the password. I want to
49:49explain to you how the SSH configuration
49:53works. We have this main configuration
49:57file that we have changed. Remember we
50:01have disabled root login by setting
50:05permit root login to no and we also
50:08changed password authentication to no.
50:12So this is the main configuration file.
50:15But if I go up you will see that we have
50:18this line include and here we include
50:22all these configuration files that are
50:25inside this directory. We call this drop
50:29in configuration files. Now let's try to
50:32look at this folder to understand what
50:35we have there. As you can see I am using
50:38tmox. This is why I have two window
50:40panes so that you can see everything uh
50:43in one place. Here I have copied that
50:46path and let me enter my password. And
50:49here I need to list all the
50:52configuration files. You can see that I
50:54have two of them. And if I use this
50:58command, as you can see, let me zoom in
51:01a little bit. So here I am searching for
51:03this property that we have changed to
51:07no. And you can see that it we already
51:09have it in these two files. What does
51:12this mean? When we created our droplet,
51:16we selected SSH authentication
51:20and digital ocean created this file for
51:23us because on Linux we treat these
51:28drop-in configuration files before the
51:31main configuration. Let me explain. You
51:34can see that here we include all these
51:37files before changing all these fields
51:40manually. This means that even if I come
51:43here and set let me see so if I set this
51:47to yes this will not take effect because
51:51if I go for example inside this file so
51:55let's look at it here we have this
51:58command and here I will just try to
52:01print what we have inside that file and
52:04here I need to type 50 yes so as you can
52:07see this file contains just this
52:10configur this property and it is set to
52:13no. So this means that this will
52:16override any value that we have in the
52:19main configuration file and you can see
52:21that we have also another file which is
52:24called 60 cloud image and here we also
52:28have this property set to no. So as you
52:31can see we have two files that set this
52:34property to no. Which one should we
52:37take? Because this is the SSH service,
52:40we are going to order the files
52:43alphabetically.
52:44So here in this example, 50 cloud in it
52:47will come before 60 cloud image. We are
52:51going to read the properties inside this
52:54file and we are not going to look at the
52:57other ones. So here if we find that
52:59password authentication is set to no, we
53:02are going to take it and we will drop
53:04the other files and the main
53:06configuration file. This is why I said
53:08that the values that we have in these
53:11drop in file configuration are going to
53:14override what we have in the main SSH
53:18configuration file. So you might be
53:21asking why did we edit the main file if
53:25Digital Ocean has already created this
53:28for us. That's a good question. We edit
53:32the main configuration file as a safety
53:34net. If you ever want to do this in the
53:37future without using a cloud provider,
53:40in that case you will not have these
53:42files configured for you. So you will
53:45need to either create one uh manually
53:48and put it inside this directory or you
53:52come inside the configuration
53:55file the the main one and you change the
53:57values manually. This way you guarantee
54:00that your server is secure regardless of
54:02the hosting provider. So now I want to
54:06simplify the way we access the machine.
54:09So if I hit again Ctrl V, I need to
54:12always specify the SSH key and then the
54:16user and uh the IP address. That's a lot
54:20and there is a way to simplify this and
54:23I'll show you how to do that. On your
54:25local machine, you can see that I'm no
54:27longer connected to the VM. I have
54:30closed the connection. I want you to
54:32type this command nano uh SSH config.
54:36You can see that here I already have
54:38some entries but I want you to go to the
54:41end of the file and I want you to copy
54:45and paste the following. So here we have
54:48placeholders that we need to change
54:50here. Uh the host. So this is just a
54:53name. Uh this is a shortcut that we will
54:56be using in the SSH command. So here
54:58I'll call this test website. Here I need
55:02to provide the droplet ID. Uh sorry the
55:06droplet IP address. I'll copy it from
55:10digital ocean. Uh okay. I will replace
55:13this placeholder with that IP address.
55:15And here I will replace this with my new
55:18user. Here the identity file is going to
55:22be the private key. I think I've named
55:24it test ssh key. Uh but I'm not certain
55:29but I I can check this. So save the file
55:32again. You can do it by hitting Ctrl S
55:36or Ctrl O and then enter. And after that
55:39hit Ctrl X. So let me check. Yeah, it's
55:43test SSH key. Okay. And let's go back to
55:46the configuration. And yeah, it's uh
55:49it's correct. Awesome. So you can see
55:51that now after adding this to the config
55:55file we will be able to log to the
55:57website just by typing SSH and the
56:01shortcut name that we have created. So
56:04now I can just type this and as you can
56:07see I am in and this is easier to
56:09remember instead of uh remembering which
56:13key you use and what is the IP address
56:16of that droplet. And now I can just type
56:19this and I'm in. You have successfully
56:22configured a clean Ubuntu server and
56:26secured it with SSH keys. However, your
56:30server is still exposed to the open
56:33internet. In the next subchapter, I will
56:37show you how to work with the firewall
56:41to lock down the network. You will learn
56:44how to configure UFW to block unwanted
56:49traffic, set up fail toban to stop brute
56:53force attacks, and use the recovery
56:57console if you ever get logged out.
57:01Welcome back. In the previous video, we
57:04launched our server and set up secure
57:07SSH access. However, right now our
57:12server is like a house with the front
57:14door locked, but every window is wide
57:19open. The server is sitting on the
57:22public internet and by default it will
57:26accept connections on any port. In this
57:29sub chapter, the firewall strategy, you
57:34are going to close those windows and
57:37lock the parameter. Here's our plan for
57:40this session. First, you will address
57:43the problem. A fresh Linux server is too
57:47open. It accepts traffic from anywhere.
57:52To fix this, you will configure UFW,
57:56which means the uncomplicated firewall.
57:59You will set it to block all incoming
58:02traffic by default and only open these
58:06specific ports you trust. Next, you will
58:10install fail to ban. This tool adds a
58:14layer of intelligence to your firewall.
58:17It automatically bans bots that try to
58:21brute force your SSH keys. Finally, I
58:25will show you the digital ocean recovery
58:28console. This is your safety net. If you
58:31ever accidentally configure the firewall
58:34wrong and block yourself out, this tool
58:38will save you. Now, setting up a
58:41firewall is great, but it doesn't stop
58:44people from trying to get in. If you
58:47look at your server logs, you will see
58:50bots trying to guess your password.
58:53thousands of times a day. We need a way
58:56to stop them automatically. This is
58:58where fail to ban comes in. I will try
59:02to explain this using this diagram that
59:05I created. We have our server and we
59:09have an attacker. So this is a user who
59:14wants to get access to our server. You
59:17can see inside the server we have the
59:19uncomplicated firewall. We have fail to
59:22ban. We have the SSH service that runs
59:26on this port. And we have this log file.
59:30So the first thing that that this
59:32attacker will try to do is to connect to
59:35this server using the SSH service. This
59:39is going to fail because they don't know
59:42the our SSH key and maybe they don't
59:46know the user. In the previous video, we
59:49have blocked access to this server using
59:53the root user. So if they try to SSH as
59:56the root user, they will be blocked. So
59:59if that happens, the service will write
1:00:02to the log something like this invalid
1:00:06user Ubuntu from this IP address using
1:00:10this port. So yubuntu is the user that
1:00:13the attacker tried to use in order to
1:00:16connect to this machine and this is uh
1:00:20his IP. So if we see this we know that
1:00:23there are some bad actors who are trying
1:00:26to log to our server. So fail to ban is
1:00:30a service that we need to install. I
1:00:34will show you how to do that. And it's
1:00:37it's going to monitor this log file. And
1:00:41if this attacker is making multiple
1:00:44requests, we can detect that. And if
1:00:47fail to ban sees that uh we have too
1:00:50many errors coming from this attacker,
1:00:53it's going to tell the firewall to block
1:00:55this IP. And now if this attacker tries
1:00:59to login again, we are going to block
1:01:02that connection. It's not going to enter
1:01:04the server. And this way we limit
1:01:08attacks like these just by reading
1:01:11what's happening inside the log file.
1:01:14That's it for the theory. Now let's open
1:01:17the terminal connect to the machine and
1:01:19let me show you how to set up UFW fail
1:01:23to ban and other things. Before I start
1:01:26the practice session, I would like to
1:01:28mention that I updated the first chapter
1:01:32and now it contains the second
1:01:35subchapter and it's called the firewall
1:01:37strategy and you have all the steps
1:01:41written here and we are going to follow
1:01:43them in this session and this GitHub
1:01:47repository is going to be shared and the
1:01:50link is going to be in the video
1:01:52description. Now let's open the
1:01:54terminal. Last time we configured the
1:01:58SSH configuration to allow us to login
1:02:01directly without specifying the user and
1:02:04without specifying the IP address. So
1:02:07now I can run this command and I am in.
1:02:11Let's start by looking at the authentic
1:02:14the.log file. To do that I will use
1:02:18sudo. I will use the cat command to
1:02:21print the content of that file. I need
1:02:25to go to var log and art.log.
1:02:30It's going to ask me for my password. I
1:02:32will provide it. And as you can see, we
1:02:35have a lots of stuff. And we see that
1:02:39invalid message. Let me try to filter
1:02:44the output to get only uh the rows that
1:02:48mention this word invalid user. And as
1:02:51you can see, we have a lot. Remember
1:02:54that's my user is IMAD. It's not Oracle,
1:02:58it's not Solana or other stuff. And as
1:03:02you can see, we have plenty of IP
1:03:05addresses that are repeating. Let's take
1:03:07this one as an example. You can see that
1:03:10this attacker is trying to connect as a
1:03:14test user after that. Yeah, it's it's
1:03:17trying to connect uh again. And if you
1:03:21look at the time, so yeah, they are
1:03:23making hundreds or maybe even thousands
1:03:27of requests a minute or a day. And
1:03:30that's not good because they are here
1:03:33the the log file is going to be to get
1:03:35bigger and it's going to consume uh
1:03:38storage and we don't want that. It's
1:03:40going to be great if we can block these
1:03:43to not fill this log file. So you can
1:03:47see that I have just created this uh
1:03:50specific droplet for this course and as
1:03:53you can see I'm getting bombarded by uh
1:03:57attackers who are trying to log the
1:04:00machine but luckily we have services
1:04:04that will automatically ban them. So
1:04:07let's get started. First, I would like
1:04:10you to look at the status of the UFW or
1:04:15the uncomplicated firewall. To do that,
1:04:18we can use pseudo UFW
1:04:22status. And as you can see, it's
1:04:24inactive. This is the problem that we
1:04:27have. The firewall is is not active. We
1:04:30need to activate it in order to block
1:04:34the open ports and allow only the ones
1:04:37that we care about because the web
1:04:40application that we are going to deploy
1:04:42is going to be running on one one or two
1:04:46ports. We don't need to open every
1:04:49single port to the public internet. But
1:04:52before activating the firewall, we need
1:04:56to allow SSH connections. If you don't
1:05:00do this step, you will lock yourself out
1:05:02of the server immediately. So run this
1:05:05command sudo ufw allow open ssh. And
1:05:10this is going to allow port 22 to be
1:05:15accessible. Otherwise, if we enable the
1:05:19firewall without allowing SSH, we will
1:05:22not be able to run SSH uh the SSH
1:05:26command to connect to the server. and
1:05:28that's not going to be uh good. In that
1:05:31case, we need to go to digital ocean,
1:05:33open the recovery console and fix this
1:05:36problem. But if you do this, then
1:05:40everything is going to be fine. Now I
1:05:43can go back and run enable to start the
1:05:49uh firewall. So as you can see, we get
1:05:52this message. It says command may
1:05:54disrupt existing SSH connections. You
1:05:56can uh let's say yes. Okay. So it says
1:05:59firewall is active and enabled on system
1:06:03startup. Let's validate this. We have a
1:06:07command that we can run. It's called
1:06:08verbose. And this is the output that we
1:06:12are getting. We have default deny
1:06:15incoming, allow outgoing and disable
1:06:18disabled routed. And the status is
1:06:21active. That's what we want to see. And
1:06:23we have uh we have this table. So by
1:06:27default you can see that we deny
1:06:30everything. This is the most important
1:06:33rule. It means that every single port on
1:06:37your server is blocked unless you
1:06:40explicitly allow it. The the exception
1:06:42list is allow in. So here we allow open
1:06:47we allow SSH on port 22. This is the
1:06:50exception. But the other ports are all
1:06:53closed. We cannot use them. This means
1:06:56that port 80 for HTTP and port 443 for
1:07:01HTTPS are blocked. But in a future
1:07:04episode, we are going to open them. But
1:07:06for now, we only care about SSH. Great.
1:07:10So now we have the firewall. We can test
1:07:13it. I will exit the machine like this.
1:07:17And now I will try to SSH again by using
1:07:21the SSH command. And as you can see I am
1:07:24in. I hope that you followed the steps
1:07:26in order. And if somehow you are unable
1:07:30to reach the server, don't worry. I will
1:07:33show you how to use the digital ocean
1:07:35recovery console to fix the issue. But
1:07:38for now uh we are going to install fail
1:07:41to ban to deal with this problem of
1:07:44attackers trying to log to the machine
1:07:48and b yeah yeah filling in the disk
1:07:51space because the SSH service is writing
1:07:55a lot of stuff uh into the log file
1:07:59because here we are using
1:08:02luckily fail toban is available in its
1:08:05default repository and can be installed
1:08:08told by using the following command sudo
1:08:12up install fail to ban dashy means yes
1:08:15confirm everything. Let me enter my
1:08:18password. Awesome. So we can see that
1:08:22the installation
1:08:24was successful and now we can verify if
1:08:28fail to ban is running or not. As you
1:08:31can see, I run this command pseudo
1:08:34service and the service that we in that
1:08:36we want to inspect is fail toban. And
1:08:38they want to look at the status. And as
1:08:41you can see here in the active row, it
1:08:44says running. Awesome. That's what we
1:08:46want to see. Now, hit Q. By default,
1:08:51fail to ban only bans attackers for 10
1:08:55minutes. This is too short. and bots
1:08:58that are hitting the web automatically
1:09:01are going to come back later. So let's
1:09:04increase this default value value to one
1:09:07day. Let's create a local configuration
1:09:11and edit this default value. So if we
1:09:15try to run sudo nano etc fail to ban, we
1:09:21can look at the jail.coniguration
1:09:25file. Awesome. And as you can see uh we
1:09:28have some default values and here they
1:09:30are telling telling us how to override
1:09:33this. So we can see that they say
1:09:34provide customizations in a jail. file.
1:09:39So this means that we should not updates
1:09:40the jail.config because if the system
1:09:43updates the values are going to be uh
1:09:46overridden and we don't want that to
1:09:48happen but the local file is uh not
1:09:51going to be changed. And if we scroll a
1:09:54little bit, yeah, we can see that we
1:09:56have plenty of values that we can change
1:09:58and the band time is set to 10 minutes.
1:10:01Uh I will explain the other values, but
1:10:04let's start by creating the local file.
1:10:07Here is the command and I want you to
1:10:09copy and paste the following text. Let
1:10:13me explain to you these parameters. So
1:10:15enabled means that we want to turn on
1:10:18protection for SSH band time. As I said,
1:10:22it's going to ban attackers for a
1:10:25specific duration. Here it is set to one
1:10:28day. Max retry allows a user to fail
1:10:32five times before being banned. This is
1:10:35important. Don't set this to one or a
1:10:38low value because you are you can also
1:10:42get banned. Let's say that I forgot to
1:10:45pass the SSH key and I did that let's
1:10:49say three times. If max retry was set to
1:10:52three, I will be banned from the server.
1:10:54I will not be able to log in. Keep it at
1:10:57around three or five. Don't set it to
1:11:00one. If you do once one mistake will be
1:11:03logged out. And if that happens, you
1:11:05need to go to recovery console, open the
1:11:08terminal, and unban yourself. But we
1:11:11don't want to do that. That's a lot of
1:11:13work. Find time is set to 10 minutes.
1:11:16And this is the window of time in which
1:11:18those five failures must occur to
1:11:21trigger a pan. So let's say that t is
1:11:24equal to zero. Time is equal to zero. Uh
1:11:27so the interval is 10 minutes. If we get
1:11:30an attacker who failed to authenticate
1:11:33five times during that duration, we are
1:11:35going to block it. But if that attacker
1:11:38failed only four times, we are not we
1:11:40are not going to block it because uh the
1:11:42minimum value is set to five. uh if we
1:11:46get five failures during that 10
1:11:48minutes, we can ban them otherwise we
1:11:50will wait for the next interval and
1:11:53start the count from zero again. Now
1:11:56save the file and exit. I also want you
1:12:00to enable the failtobend service to
1:12:04start automatically when the server
1:12:06reboots. So do this and start it using
1:12:11the systemctl start command. Now we can
1:12:15verify that the ban time is equal to 1
1:12:18day by running this command failtoban
1:12:22client get shd ban time and it gives us
1:12:27600.
1:12:28600 is basically 60 seconds time stamp
1:12:32which is 10 minutes. Let me retry
1:12:34running the enable and start commands
1:12:37again. Let's go back to get SHD band
1:12:40time and it still gives us 600 seconds.
1:12:44The problem here is that I forgot to
1:12:48restart the service and in order to do
1:12:51that we need to run the restart command.
1:12:54So this will allow fail to ban to read
1:12:57the new configuration that we created.
1:13:00Now if I go back and as you can see we
1:13:02get 86,400
1:13:05seconds which is 1 day in seconds. Now
1:13:09let's see if fail to ban is working. We
1:13:12have this command status SHD. As you can
1:13:16see we get this output. We have the
1:13:18status for the jail SHD. Great.
1:13:21Currently failed total failed. We have
1:13:24the and currently banned. So we can see
1:13:26that fail to ban has banned these two
1:13:29IPs and here we have total banned
1:13:32currently banned. Now these two
1:13:35attackers are not going to be able to
1:13:38reach the SSS SSH service because the
1:13:41firewall is going to block their
1:13:44requests. Awesome. So this means fade
1:13:47toban is working as we expected. Now
1:13:50I'll keep this running. I'll now
1:13:53transition to showing you how to access
1:13:56the digital ocean recovery console and
1:13:59I'll come back and see if we catched
1:14:02some bad actors. You might be wondering
1:14:06why do we need to use the recovery
1:14:09console. I'll give you an example. If
1:14:11you accidentally run UFW enable before
1:14:17allowing open SSH or if you configure
1:14:20SSH keys incorrectly and lose access to
1:14:24your private key. In that case, you not
1:14:27be able to log in to the machine and the
1:14:30recovery console is provided to us by
1:14:33digital ocean to give us direct access
1:14:36to the machine through the browser and
1:14:40this bypasses SSH entirely. To access
1:14:44the recovery console, click on the
1:14:47droplets, click on the access tab,
1:14:50scroll down, and here you have the
1:14:53recovery console. Let's launch it. Okay.
1:14:56So, here it tells us that we need to log
1:14:59in. First, I want you to start typing
1:15:02randomly just to verify that the keys
1:15:07that are pressed matches what you are
1:15:10seeing because sometimes if you have a
1:15:13different keyboard layout, yeah, this
1:15:15might not work well. So in that case you
1:15:18need to basically start typing your
1:15:22login username and password and after
1:15:26that delete everything and start typing
1:15:29your user. So here I want to login as a
1:15:33and the password is going to be this
1:15:36one. As you can see I am logged in and
1:15:40now I can type in the same command that
1:15:44I've showed you earlier. sudo ufw allow
1:15:47open SSH it's already allowed so nothing
1:15:51will happen um because I am using pseudo
1:15:53I need to provide this and as you can
1:15:56see it says skipping adding existing
1:15:58rule but in your case it's not going to
1:16:00say this it's going to add it it's going
1:16:03to add this rule for SSH now we are fine
1:16:07we can exit and let's close this tab and
1:16:12now you can run the SSH command and you
1:16:15can login without any problem. Let's go
1:16:18back to Digital Ocean again. If for some
1:16:21reason you want to reset the p the root
1:16:24password, you can do that. But we don't
1:16:27have that, so there is no need to. And
1:16:30you can open the droplet console by
1:16:34specifying the user and you can launch
1:16:37it. So if you don't want to use your
1:16:39terminal, you can use everything through
1:16:42the web page. And as you can see I am
1:16:44connected as this user. Let's try to
1:16:48connect as root. Awesome. So it failed
1:16:52because in the previous session we
1:16:54blocked accessing this machine as root.
1:16:58Awesome. So everything is working as
1:17:01expected. What is next? You might be
1:17:04wondering. That's a good question. So
1:17:06far you have a solid foundation. You
1:17:10have a secure server that blocks
1:17:12unauthorized traffic and automatically
1:17:16bans attackers. As you can see, I have I
1:17:20rerun the status shd command. As as you
1:17:23can see, now we blocked four people.
1:17:26This is the new IP address that we
1:17:29blocked. So, this means that fail to ban
1:17:31is working. And as you can see, it's
1:17:33automatic. We don't need to do anything
1:17:35manually. And that's awesome. And you
1:17:37might be wondering why this IP was not
1:17:41added to the previous two. So these two
1:17:44got unbanned because the band duration
1:17:47is set to 10 minutes before I changed it
1:17:51to 1 day. And we can verify this by
1:17:54running this. As you can see it says
1:17:57unban. So [clears throat] if I look at
1:17:59the fail toban logs I can filter by this
1:18:02word unban. And as you can see these all
1:18:06these IPs were uh were unbanned. And you
1:18:09can also look at this specific new ID.
1:18:12And as you can see it was banned. So
1:18:15that's that's confirms it. And here I
1:18:18just want to explain this. If you will
1:18:21see that these two got unbanned because
1:18:23before changing the configuration to set
1:18:27the band time to one day, these two were
1:18:29already banned. That's why they they
1:18:31were unbanned in 10 minutes. But this
1:18:33one will stay here and will be unbanned
1:18:36the next day. Let's just verify that it
1:18:39is still there. Yeah, it is still here.
1:18:42And four, this basically keeps a total
1:18:45count of how many IPs were banned. It
1:18:48doesn't mean that we have four IPs that
1:18:50are currently banned. The currently
1:18:53banned is what you need to look at and
1:18:55it shows us that we have just one IP
1:18:57that that is banned from accessing or
1:19:00from trying to communicate with the
1:19:01server. In the next module, we will
1:19:04shift focus from security to software.
1:19:07You will learn how to prepare the
1:19:09environment by installing Python and
1:19:11NodeJS. Setting up swap memory to handle
1:19:15builds on a low RAM server like ours and
1:19:20configuring J unicorn and supervisor to
1:19:23keep your application running reliably
1:19:26in the background. Hi everyone. In this
1:19:30video, you are going to put your server
1:19:32to work by turning it into a web server.
1:19:37You will learn a lot from this video.
1:19:39I'll show you how to move files from
1:19:42your local machine to the virtual
1:19:44machine, use SSH tunneling, find and
1:19:48kill zombie processes, and more. In this
1:19:52video, you will be deploying a full
1:19:54stack application that uses Vue.js JS in
1:19:58the front end and pass API for the back
1:20:01end. The source code is available on
1:20:04GitHub and I will make sure to put the
1:20:06link in the video description. I am
1:20:09using my personal website in order to
1:20:12show you the exact same steps I took to
1:20:16deploy it. The goal here is not to
1:20:18deploy my specific website on your
1:20:21server, but use this project as a
1:20:24reference. Especially if you don't have
1:20:26your own project. If you have your own
1:20:29app, but you don't use Vue.js and pass
1:20:32API, it's fine. You can adjust a few
1:20:36commands in this tutorial to make it
1:20:38work with your specific setup. Now,
1:20:40let's go back to the slides because I
1:20:41want to show you the road map for this
1:20:44session. We will transfer the project to
1:20:48the server using the SCP command and I
1:20:52will show you how to organize the file
1:20:55permissions securely. Next, you will
1:20:58install the necessary runtimes. We will
1:21:01need Python for the back end and NodeJS
1:21:04for the front end. I will show you how
1:21:07to set up swap memory. This is an
1:21:10important step for smaller servers in
1:21:13order to prevent out of memory crashes
1:21:16during the build process. We will use
1:21:19SSH tunneling to securely preview the
1:21:22application on your local browser before
1:21:25opening it to the public. And finally,
1:21:29we will configure J Unicorn and
1:21:31Supervisor to make the backend robust
1:21:34and production ready. Now let's go to
1:21:37GitHub because we need to copy the
1:21:39source code. Click on the code button
1:21:42and copy this link. Now open your
1:21:45terminal. Since this is just a test, uh
1:21:48for me, I will put this in the download
1:21:51folder, but feel free to clone this
1:21:53repository in any directory. Run the get
1:21:57cloned command and here paste that link
1:22:00that we copied and hit enter. As you can
1:22:04see, we are cloning the repository and
1:22:06the command finished. Great. Now we need
1:22:09to move inside this folder. So I will
1:22:12use the cd command and I will type in my
1:22:16this name. And now I am inside that
1:22:20repository. And as you can see we are on
1:22:22the master branch. Now let me type ls. I
1:22:26want to see what we have inside. As you
1:22:28can see we have the backend folder and
1:22:30front end folder. These are the two
1:22:32folders that are that contain the s the
1:22:36code for the the website. The other
1:22:39stuff are not that important for now.
1:22:41These folders are inside my laptop. I
1:22:45need to move them to the server that we
1:22:48created on digital ocean. We could move
1:22:51them one by one, but that is slow and
1:22:55repetitive. Instead, we are going to
1:22:57compress these files and folders into
1:23:01one compressed file. In order to do
1:23:04that, we will use the tar command. So
1:23:07here I am going to call this compressed
1:23:10file source code and we need to exclude
1:23:14some folders that are not important and
1:23:17that will take a lot of space. For
1:23:18example, the node modules contains the
1:23:21dependencies that the front end needs.
1:23:23So we don't need this. we can generate
1:23:25it directly on the server. Same for
1:23:28Python, we have the virtual environment.
1:23:31We need to exclude that. We also have pi
1:23:34cache and the getit directory. So we
1:23:37don't need this these folders. That's
1:23:39why we exclude them. And here we have a
1:23:43dot and this means that we want to
1:23:46compress everything inside this
1:23:49directory. So if you wanted to compress
1:23:52just the back end folder, we you need to
1:23:55change this dot and put back end or if
1:23:58or something else. But here we want to
1:24:00take everything and put put it on the
1:24:02server. Now hit enter. And as you can
1:24:06see, we were able to compress everything
1:24:08inside this compressed file. Now we need
1:24:11to copy this from our laptop to the
1:24:15server. In order to do that, we are
1:24:17going to use the SCP command, which
1:24:20stands for secure copy. And here,
1:24:23because we configured SSH alias in the
1:24:27previous chapter, we don't need to type
1:24:29the IP address of the server, we can
1:24:32just type in the name. So here I will
1:24:35take the source code. So I'm going to
1:24:38copy this and I'll put it on the
1:24:42website. And here we don't need to type
1:24:45root uh the user at and then the IP
1:24:48address. We can just uh use the alias
1:24:51the alias that we created before. And
1:24:54here I will put it inside the temp
1:24:57directory. Now hit enter. And as you can
1:25:00see we have a progress bar that shows us
1:25:02that we are uploading this to the
1:25:04server. And we it's finished. So now I
1:25:08can go SSH to that website or to that
1:25:13server using this command. As you can
1:25:15see I am in let's verify that we have
1:25:18that zipped file and as you can see we
1:25:21have it inside the temp directory. Now
1:25:24let's organize the project files. We
1:25:27should move this compressed file
1:25:29somewhere. But where exactly? The best
1:25:32practice for web applications is to use
1:25:35a neutral location like /var/www
1:25:40or a dedicated folder like web app. This
1:25:44approach keeps your application logic
1:25:47separate from your personal user files
1:25:50like SSH or bash history and prevents
1:25:55permission issues. So let's create this
1:25:58folder with the make directory command.
1:26:02So I will use a make directory command
1:26:05and I'll put everything inside web app.
1:26:08Now we need to move the compressed file
1:26:10into this new folder. For that we are
1:26:13going to use the move command. I will
1:26:16take the path to that compressed file
1:26:19and I will move it to web app. Great.
1:26:22Now we can um yeah yeah yeah let's
1:26:25change directory let's move to web app
1:26:28uh let's type ls and as you can see we
1:26:30have that compressed file now we need to
1:26:33decompress it using the tar command
1:26:35again and here is how we do that hit
1:26:38enter and let's see yes we have
1:26:42everything now we can delete this
1:26:44compressed file because we no longer
1:26:46need it let's type its okay great so we
1:26:48have everything here that's great now
1:26:51Before uh we continue, let's type ls-la
1:26:55and hit enter. Here you need to see that
1:26:58your user is owning these files. As you
1:27:00and as you can see, my user is owning
1:27:02them. If you see root is owning these
1:27:05files and folders, make sure to run this
1:27:08command sudo chown. So change ownership
1:27:12dash r. We want to do this recursively.
1:27:14And here you put in you put the user and
1:27:17the group. in in in my case it's going
1:27:20to be a MAD and here you specify the
1:27:24folder but since we are inside web app
1:27:27um we need to yeah change directory but
1:27:30or we can just use dos okay and
1:27:34hopefully when you when you run ls-la
1:27:37you you'll be able to uh see that the
1:27:40ownership has changed and your user is
1:27:42owning these files. This is very
1:27:44important so make sure to do that.
1:27:47Awesome. We are now ready to install
1:27:49dependencies. Let's start with the back
1:27:52end. First we need to install Python
1:27:54tools because those are required for the
1:27:57back end. Here we are working with the
1:27:59with an Ubuntu machine. So this is how
1:28:01we can install them. So we are going to
1:28:03install pip the virtual environment and
1:28:06dev. And here dashy will just accept
1:28:09everything and we don't need to yeah
1:28:11confirm this operation. Awesome. We
1:28:14installed the those tools. Now I'll
1:28:17clear the screen. Let's go inside the
1:28:20backend folder. And now let's create an
1:28:24virtual environment. So we will use
1:28:26Python tree dash M. We will use the VNV
1:28:30module and we will create a virtual
1:28:32environment called VNV. Hit enter. Okay,
1:28:36great. Let's activate the environment.
1:28:39So we will use source bend/bin/activate.
1:28:45If you see the name of the virtual
1:28:47environment here before your name is the
1:28:50user's name that means that you
1:28:52activated the virtual environment
1:28:54successfully. Now if I let's type ls
1:28:58because I want to see yeah we have the
1:29:00requirements.txt file. Those are all the
1:29:03the dependencies that we need to launch
1:29:06the backend server. In order to install
1:29:08them, we will use pip install-r
1:29:13and we will provide the requirements.txt
1:29:16file. Let's wait for this to finish. And
1:29:19here make sure that before you install
1:29:21the dependencies that your virtual
1:29:24environment is activated cuz we want
1:29:26everything to be contained inside this
1:29:28project. Awesome. We were able to
1:29:31install everything. That's great. Let me
1:29:34uh clear the screen. And now we can
1:29:37start the back end to yeah make sure
1:29:40that everything runs without any issue.
1:29:42And here we are working with an with a
1:29:44fast API application which means that we
1:29:47need to use uh u core. Uh this is this
1:29:51command or this library is installed
1:29:53will be installed after running pip
1:29:55install-r
1:29:56requirements.txt.
1:29:58And here we have uh so the entry is the
1:30:03entry point is inside this file and the
1:30:06yeah the app is called app. We are going
1:30:09to provide 0000 to the host because we
1:30:12want we want to make this the server
1:30:16accessible outside the virtual machine.
1:30:19And finally we want port and let's use
1:30:248,000 because this is what the front end
1:30:27expects. So now hit enter
1:30:30and yeah it seems to be working. If you
1:30:33see application startup complete then
1:30:35that means that you are in a in a good
1:30:37shape and you can see that u is running
1:30:39on this IP 0000
1:30:43and yeah if you try to click on this
1:30:46link nothing will happen because uh
1:30:48let's see let me show you as you can see
1:30:51nothing happens because this server is
1:30:53running inside the virtual machine it's
1:30:55not running locally on our computer uh
1:30:59close the page go back to the VM but if
1:31:02We can't access this the server outside
1:31:05of the VM. How can we test it? Well, in
1:31:09order to test it, we need to to open
1:31:11another terminal. I will use again the
1:31:14sh command and I will go inside the VM.
1:31:19But before I test it, I want you to run
1:31:22the uh UFW
1:31:25status command. uh provide your password
1:31:29and I want you to make sure that the
1:31:33firewall is active and that you have
1:31:36just SSH. So these are the ports that
1:31:39are allowed. But we don't have port
1:31:418,000 or other ports. That's why we were
1:31:44not able to access this server outside
1:31:48of the the VM. But we can at least test
1:31:51using the curl command. So we are going
1:31:53to send a request to this to the server
1:31:57and here 127.001
1:32:00means local host basically inside the VM
1:32:03inside the server and 8,000 is the port
1:32:07and this is going to make a get request
1:32:10to the server running on this port. So
1:32:13yeah, let's see what we'll get. So it
1:32:15says detail not found. Uh maybe because
1:32:20we need to do health. Uh let me go
1:32:22inside the web application back end and
1:32:27let's look at the main file. Let's use
1:32:31uh or let's use maybe nano main.py.
1:32:36Let me make the screen smaller. I want
1:32:39to see the end points that we have here.
1:32:42So yeah, we need to use / API/hel
1:32:46in order to get status is okay. Okay,
1:32:49let's zoom in again and here we need to
1:32:54before help we need to add API and as
1:32:58you can see we get we are we get status
1:33:00is okay which means that the backend
1:33:02server is responding and if you look at
1:33:04the logs you see that we were making get
1:33:07requests this we we were getting 404
1:33:10because the endpoints didn't didn't
1:33:13exist so the root endpoints or the /hel
1:33:17endpoints don't don't exist That's what
1:33:19we got 404. But when we try to talk to
1:33:24this endpoint API/halth, we got 200.
1:33:27Okay, because it exists in the source
1:33:30code. Great. So this confirms that the
1:33:33backend server is running and that's the
1:33:35we installed all the dependencies that
1:33:38we need. Hit Ctrl C because we want to
1:33:40stop the server. We no longer need it.
1:33:42Now we will move to the front end. We we
1:33:45are going to install the dependencies
1:33:47there and do the same thing. We need to
1:33:49verify that it works and after that we
1:33:51will combine them together. Uh you can
1:33:54see that we are inside the backend
1:33:55folder. Let's move uh one layer up.
1:34:00We have the virtual environment is still
1:34:03active. We can leave it like that or if
1:34:06you want you can um write type
1:34:08deactivate like so. And now we need to
1:34:12go inside the front end folder. I am
1:34:15using Vue.js in the front end. This
1:34:17means that we need to install NodeJS.
1:34:21And the best way to install NodeJS is
1:34:24with NVM. Uh if you don't know, NVM is a
1:34:28Node package manager. So just type nvm
1:34:31GitHub. There it is. And the
1:34:34installation the installation is easy.
1:34:36You just have you just run this curl
1:34:39command. So copy it, go back to the
1:34:42server and paste it here. So this is the
1:34:45latest version of NVM. Now in order to
1:34:48install it, as I said, we just hit enter
1:34:51and voila, it's done. Now the the
1:34:54problem is that when you install NVM,
1:34:56you need to create a new terminal to
1:34:59basically get the command. Because now
1:35:01if I run NVM, you can see that it's not
1:35:04recognized. But we have a solution. we
1:35:06can uh use source uh and here I am using
1:35:11bash so I will target bash rc
1:35:16and this so if when you run this it's
1:35:18going to basically load in nvm so now if
1:35:22I type nvm you can see that uh it works
1:35:24and let's verify that the version is
1:35:27correct yes it's 040.4 4. Awesome. Now
1:35:30we can we need to install the late the
1:35:34latest long-term support version of
1:35:37node. And in order to do that, we run
1:35:40nvm install and we we uh yeah we provide
1:35:44LCS because we that's what we want to
1:35:46install. And this is the latest LTS
1:35:50version from node. And if you want to
1:35:53yeah verify if that's correct, you can
1:35:56go to nodejs.org. or uh click on get
1:36:00NodeJS and as you can see so the latest
1:36:02LTS version is 2413.1
1:36:07let's verify that and yeah that's
1:36:09correct so 2413.1
1:36:11we also need to install PNPM this is the
1:36:15command that I use to install the
1:36:16front-end dependencies I don't use npm
1:36:20and this is how you install that make
1:36:22sure to install the latest version of
1:36:25PNPM version 10 and not prior prior
1:36:28ones. Hit enter. And here we want to
1:36:31install it globally. Okay. So, it's it's
1:36:35working. Uh it says that there is a new
1:36:38minor version of npm available. That's
1:36:40not important. So, let's click that. And
1:36:43now let's verify we are inside the front
1:36:46end directory. Type pnpm install and hit
1:36:50enter. And as you can see, yeah, PNPM is
1:36:53very fast and it's uh I really like to
1:36:56work with it. And voila, we have
1:36:58everything installed. And now in the
1:37:01front um so when working with Vue.js or
1:37:04any front end framework, uh you need to
1:37:07run the the build command to ensure that
1:37:10everything builds. So let's do that. And
1:37:12here we are going to hit a out of memory
1:37:15problem. So this is why we need to
1:37:17create a swap memory. But let's verify
1:37:19if that's what will happen or not. Okay.
1:37:22So as you can see the terminal is stuck.
1:37:25Uh but let's wait a minute. If you want
1:37:28you can also go back to the other
1:37:30terminal and maybe run hub. But as you
1:37:34can see the yeah the the VM is
1:37:37struggling. So the I can't run things
1:37:40smoothly and that means that I'm running
1:37:42out of memory. And yeah, this this this
1:37:46command will fail at any second. So
1:37:49let's just wait. Yeah, so this one is
1:37:52stuck. Let's go back to here.
1:37:55And let's wait. Uh I am back to the
1:37:58other terminal. And as you can see, the
1:38:00memory is full. Uh and this is not
1:38:03updating because it's uh struggling. And
1:38:06as I told you, the command failed. And
1:38:09that's because we don't have enough
1:38:11memory to build uh the front end. Is
1:38:14that the end? Uh we have a solution. So
1:38:17when uh on Linux we have something
1:38:19called swap memory. Basically we take
1:38:23some portion from the SSD and we use
1:38:26that as additional RAM. It's going to be
1:38:30slower but it's going to help us uh deal
1:38:33with these situations where we run out
1:38:35of memory. Let's run the df command.
1:38:39Going to show you uh so all available
1:38:43and yeah let's type it h to make it
1:38:46human readable. As you can see here we
1:38:49have 8 GB of disk storage and we have 5
1:38:54GB available. I'm going to take two GB
1:38:57from this partition and I'm going to
1:39:00create a swap file because we don't need
1:39:03all this storage uh to run the front end
1:39:06and back end. And this is how we create
1:39:08a swap file. So we use the f allocate
1:39:12command. We specify the size and this is
1:39:17the the file that we are going to
1:39:18create. And here so let's just run the
1:39:21command first and let me provide my
1:39:23password. It worked and we didn't get
1:39:25any error. That's great. And I want to
1:39:28explain something this. So when you see
1:39:31/ swap file, this doesn't mean that we
1:39:34this is a folder. No, it's a single file
1:39:38acting as a virtual RAM. So this means
1:39:42that you can just you can't do this. You
1:39:45can do this. Um just type in in / LA.
1:39:49And as you can see, it's a file. If it's
1:39:51that D that means that it's a directory
1:39:54but yeah it is a file and it's owned by
1:39:57root. Why did we create a file instead
1:40:00of a dedicated disk partition? Because
1:40:04we can easily resize or delete this file
1:40:08if we wanted to if we don't need it
1:40:10anymore. This is the flexibility of
1:40:12working with files. But we need to set
1:40:15up uh some permissions. Uh again let me
1:40:17type in ls-la. You can see that these
1:40:20are the permissions. We need to change
1:40:22them and uh to do that we are going to
1:40:25use sudo ch mode 600 and swap file. I
1:40:30will explain what the 600 means. Now if
1:40:33I type ls-la you can see that some of
1:40:36these characters have changed. So w
1:40:39means write, r means read and x means
1:40:43execute. But here we don't have x. And
1:40:46the first three let's say letters are
1:40:50related relative to your to the root
1:40:53user. The second three are relative to
1:40:57uh where is the yeah the second tree are
1:40:59here are uh for your user. So in my case
1:41:02it's emad and the last ones are for
1:41:05other users. So here uh so we did 600 we
1:41:10which means that root will be able to
1:41:13read and uh to read from swap file and
1:41:16write to it but the other users are not
1:41:20able to touch this at all. So we created
1:41:23the swap file. Now we need to make this
1:41:26file as a swap space and to do that we
1:41:29are going to use the make swap command
1:41:32and we are going to provide that file.
1:41:35Hit enter. As you can see, it says
1:41:37setting up swap space and here is the
1:41:39size and it gives us an ID uh which is
1:41:43not important. Now let's enable it. So
1:41:46we created the swap space. Now we will
1:41:48use swap on and um swap file. Hit enter.
1:41:53How do we how do we verify this? We are
1:41:56going to use the swap on command. And
1:41:58let's type show. And yeah, we have it
1:42:02here. So this is the file name and this
1:42:05is the size. It says that the file the
1:42:07type is file. That's good. And here we
1:42:10have priority. The priority level is set
1:42:13to minus2. In our case it doesn't matter
1:42:16because this this is um used only when
1:42:20we have multiple swap files. So if um we
1:42:24had another swap file that had a lower
1:42:27priority level, then Linux will
1:42:29prioritize this one. And when it's
1:42:31filled, it's going to move to the other
1:42:32one. But in our case, it's not relevant
1:42:35because we have just one swap file. By
1:42:38default, this swap configuration will be
1:42:41lost when the server reboots. To ensure
1:42:44that the swap file loads automatically
1:42:47every time the system starts, you must
1:42:49add it to the FS tab file. Uh let me
1:42:53show you the FS tab file that I'm
1:42:55talking about. It's this one. Now let's
1:42:58try to uh let's see the content. Okay,
1:43:00so this is what we have. Uh and we are
1:43:03going to yeah put uh this string inside
1:43:08there and we are going to use one
1:43:09command to do that to make things
1:43:11quicker. Here we have echo. It's going
1:43:13to spit this string. We use the pipe
1:43:16command to take the output from this
1:43:18command and give it as input to the t
1:43:21command. Let me let's hit enter. Okay,
1:43:24great. So it worked. Let's go back and
1:43:27as you can see we appended this string
1:43:29to the end of the fsst tab command. Now
1:43:32that we have added extra memory let's
1:43:35try to build the project again. So let's
1:43:37run pmppm run build. Uh I will also yeah
1:43:42let's uh open another terminal and let's
1:43:44go ssh to the server and here I want to
1:43:49run htop. Okay. And as you can see htop
1:43:52also detects the swap memory. So
1:43:54currently it is empty but look what will
1:43:57happen when we start the build process.
1:44:00Let's see what will happen. As you can
1:44:02see we are already filling the swap
1:44:04space and look how the the physical RAM
1:44:08is quickly filling up. Nothing is
1:44:11freezing. That means that it we it
1:44:13works. So if you look at the total uh
1:44:17space consumed it's yeah greater than
1:44:19the total memory that we have inside
1:44:23this uh machine and it seems that it
1:44:26worked. Yeah it built. So yeah that's
1:44:28that's awesome. Before making this
1:44:31tutorial sometimes node was not using a
1:44:35lot of memory so the build failed due to
1:44:39an error called heap limit error. If
1:44:42that somehow happens to you, then before
1:44:46uh running pmpp run build, we need to
1:44:49pass this variable. So here we are going
1:44:52to set max all space size to 2 GB. Uh we
1:44:56do that before uh starting pmppm. Um in
1:45:00my case it worked but if for some reason
1:45:03again uh you get that heap limit error,
1:45:06you can solve that by using this
1:45:08command. Awesome. So we built the
1:45:11project and as you can see we have the
1:45:13dis folder. It contains everything we
1:45:15have in the front end. Uh let's move
1:45:18inside that directory cd dist. Uh let's
1:45:22see we should have yeah we have our
1:45:24assets there. We have the index html
1:45:28some images sitemap uh and stuff like
1:45:31that. Now we can create a temporary
1:45:33Python web server on port 8000. Uh let
1:45:37me use Python and we want to use this
1:45:40module https
1:45:43server and the port is going to be at80.
1:45:46Uh I need to use python tree. Okay
1:45:50great. So we are serving the built files
1:45:54uh on this port. Again we cannot access
1:45:57this. So if you try to open this on your
1:46:00local machine, it's not going to work
1:46:02because the firewall is active and we
1:46:05are not uh letting traffic through this
1:46:09port. But is are we done? Um maybe yeah,
1:46:13we can use the cur command if we want.
1:46:15So we can do something like this. But
1:46:19yeah, we are we have port 8080 this
1:46:22time. And as you can see, we got uh some
1:46:24HTML back. Uh so that's that means that
1:46:27it's it's working right but we want to
1:46:30see this we want to see the UI because
1:46:33the back end uh it's it's fine if we run
1:46:35cur commands on the back end but when it
1:46:38comes to the front end we need to see it
1:46:39with our eyes to validates that it is
1:46:42working but you might say well we cannot
1:46:45do that because the firewall is active
1:46:48but I have a solution for you and it's
1:46:50called SSH tunneling. Let's go back to
1:46:53the slides because I want you to learn
1:46:56about this amazing feature. Okay, here
1:46:59we are and let's look at SSH tunneling.
1:47:02I have this diagram that I made in order
1:47:05to explain this concept for you. We have
1:47:08our laptop is sitting here. It can be
1:47:12running maybe Linux, Mac or Windows. It
1:47:15doesn't matter. We have the SSH key
1:47:18pairs that we created. Uh here we have
1:47:21the private key and we will be uh
1:47:24accessing the the website through a
1:47:28browser and here on the other side we
1:47:30have the server. It's can be running on
1:47:32digital ocean, Azure, GCP, AWS doesn't
1:47:36matter. We also we will have our front
1:47:39end running here. Uh in our case it's
1:47:42Vue.js JS but it can also be react or
1:47:45any other framework or just pure HTML
1:47:48CSS and JavaScript and we have our
1:47:50public key in the middle we have a
1:47:53tunnel that we call SSH tunnel how do we
1:47:56see the app without opening uh let's say
1:48:00port 8080 the we use SSH tunneling for
1:48:03that think of this as a secure private
1:48:06subway line that runs inside your
1:48:09existing SSH connection we don't need to
1:48:12change the firewall because we are
1:48:14already allowed to access the server via
1:48:16SSH to create a tunnel. We use the -l
1:48:21flag. And this syntax tells SSH to bind
1:48:26port 8080 on your laptop. It's this one
1:48:30colored in pink to port 8080 on the
1:48:34server. After running this command, a
1:48:37tunnel will be created. Now if I open my
1:48:40browser and type localhost and specify
1:48:44port 8080 in my browser the request is
1:48:48not going to stay on my computer. It's
1:48:51going to travel through the tunnel. It's
1:48:54going to reach the server the this uh
1:48:57and here it's going to reach the server
1:48:59on this port 8080 because this is what
1:49:01we specified here. It's going to see
1:49:03that we have a server running. This is
1:49:06why we have uh yeah we run that Python
1:49:09command to create a temporary server.
1:49:11Okay. So it's going to forward that to
1:49:14the front end. The front end will
1:49:16respond with that HTML. It's going to go
1:49:18back through the tunnel. It's going to
1:49:20hit the browser and the browser uh sees
1:49:24HTML. It's going to load it. So this is
1:49:26the concept uh of SSH tunneling and it's
1:49:29really great. And as you can see, we
1:49:31didn't update the firewall configuration
1:49:34to allow port at it enabled directly
1:49:38because we are already using SSH. But if
1:49:41we stop this command, the tunnel will go
1:49:44yeah will go away and these ports are
1:49:47going to be are not going to be
1:49:48connected. Awesome. So yeah, let's do
1:49:51this. Uh uh let's create. So here I'm
1:49:54already inside the machine. I need to
1:49:57exit by using Ctrl D or exit. And now
1:50:00yeah, let's run let's write that
1:50:03command. So it's SSH-L.
1:50:06Uh let's type the local port on my
1:50:09machine. Here we need to specify
1:50:11localhost AT80 and let's use my alias my
1:50:16website. As you can see, we are inside
1:50:19and now we have a tunnel. Let's go back.
1:50:23You can see that we have the server
1:50:24running. Now let's go. Yeah, let's open
1:50:28Firefox. And now we can go to localhost
1:50:32at80. And voila. As you can see, it
1:50:35works. You saw that we we got some
1:50:37errors because we are making uh requests
1:50:40to the back end, but we don't have a
1:50:42back end running. Uh yeah, so you can
1:50:45see that if I go to the blogs page,
1:50:47yeah, nothing will show up because we
1:50:49don't have the back end running yet. But
1:50:51that's not a problem. What we want to
1:50:53see is uh basically the UI. Does it look
1:50:57right? Um did we load the images? Um is
1:51:01the font loaded? Do we have animations
1:51:04running? Stuff like that. This is what
1:51:06we wanted to verify. But connecting the
1:51:08front end to the back end or to other
1:51:11services uh should be uh easily done
1:51:15when we connect it to the back end.
1:51:18Awesome. And yeah, let's go back. Uh now
1:51:20what will happen if I do this? So let's
1:51:23close this. Now we have no SSH tunnel.
1:51:27Let's go back and let's refresh. Can see
1:51:29that it doesn't work because now the the
1:51:32two machines are not connected. Great.
1:51:35So everything is working perfectly.
1:51:38Nice. We installed the Python the back
1:51:41end and front end dependencies. We made
1:51:43sure that both servers run correctly. Uh
1:51:47everything worked really well.
1:51:50Now we need to move to installing J
1:51:54unicorn and supervisor. But why do we
1:51:57need them? Let's go back to the slides
1:51:59again to explain this. Here is the
1:52:02diagram that I created for this slide in
1:52:05order to explain this. Uh as you can
1:52:07see, we have three components. We have
1:52:09supervisor, junior and uvicorn. We have
1:52:14used the uvicorn command to run the
1:52:17python server. After that, we used curl
1:52:20to ensure that the the back end responds
1:52:24correctly. So that's done. But we did
1:52:26this manually. What will happen if the
1:52:28server crashes? We need to again SSH to
1:52:32the server and run the UV command again.
1:52:35Is this scalable? No. What will happen
1:52:38if your server crashes at 4:00 a.m. or
1:52:41something? Are you going to uh wait
1:52:44until 8:00 a.m. when you wake up and
1:52:47launch it? No. We want the system, we
1:52:49want the server to do this for us
1:52:51automatically. And this is why we need
1:52:53these three components or this
1:52:55architecture. At the bottom, we have our
1:52:58workers. The worker he's is is Uicorn.
1:53:02The the worker is going to run your fast
1:53:05API code. So if someone visits the
1:53:08website and wants to get blogs, the
1:53:11worker will get that request. It's going
1:53:13to talk to me search and it's going to
1:53:16get the response and send it back to the
1:53:19user. Or let's say that someone wanted
1:53:22to maybe do a search query or maybe uh
1:53:26like a blog. U yeah that that worker
1:53:30will also do that. And let's say that we
1:53:32had a huge traffic um let's say that we
1:53:36we have maybe 100 people trying to
1:53:39access the server at the same time. Then
1:53:41the load is going to be balanced through
1:53:44these workers here. Let's say that we
1:53:46have three workers. So this one is going
1:53:48to get maybe 33 requests. The other one
1:53:50is going to get 40 and the other one
1:53:52gets 30. So yeah and J Unicorn will do
1:53:56that. It's going to balance the load
1:53:57through all these workers. Now we know
1:54:00the role of Uicorn but why do we need J
1:54:03Unicorn? J Unicorn acts as the manager.
1:54:07It spawns multiple Uicorn workers and
1:54:11this is can be configured. So if you
1:54:12want to you have just one URON instance
1:54:15or worker you can do that. If you want
1:54:17to have three or 10 or whatever yeah you
1:54:21can do that. Um you can just change the
1:54:23configuration. We will see this after
1:54:26explaining this slide. Let's say that
1:54:28this worker crashed for some reason.
1:54:31Jorn will detect that. It's going to
1:54:34delete it and replace it with a new with
1:54:37a new uh worker. And this is going to be
1:54:41done automatically for you. You don't
1:54:43need to enter to SSH to the server and
1:54:46start the worker manually. That's not
1:54:49scalable. But this this is above that we
1:54:53have the supervisor or we can also call
1:54:56it watchdog. Uh in engineering this term
1:54:59means a hardware component that
1:55:02basically will hit the reset button if
1:55:05the system freezes for some reason.
1:55:07Let's say that something bad is
1:55:09happening. The unicorn is not doing
1:55:12good. The unicorn curs are also
1:55:16crashing. Supervisor will detect that
1:55:18and uh hit the resize button which means
1:55:21that uh we are going to recreate
1:55:22everything from uh scratch just to
1:55:25ensure that yeah the the our application
1:55:28is healthy again and is ready to receive
1:55:31requests from users. So yeah, this is
1:55:34the this is why we need J unicorn and
1:55:37unicorn and also supervisor and you saw
1:55:41we tested manually. Always do that. Try
1:55:44to test uh your application manually by
1:55:47running the commands like what we did.
1:55:50So we run uh uicorn to start the backend
1:55:54server. After that we moved to the front
1:55:57end server. After validating that we can
1:56:01automate this process. Let's go back to
1:56:03the server and let's do this. Okay. Uh
1:56:07now let's stop the front the temporary
1:56:10front end server because we don't need
1:56:12it. Uh let's also go back to back end.
1:56:18Activate the environment and let's
1:56:21install J Unicorn. I'm not sure if I
1:56:24already have it in the requirements
1:56:26file. Let's just double check. And yes,
1:56:28we have it. So yeah, that's great. So we
1:56:30don't need to install it separately. Uh
1:56:33let's make sure that it is there. Yeah,
1:56:35it is there. So we it just didn't
1:56:38receive any application. So that's fine.
1:56:40So we have J Unicorn and we also have UV
1:56:43core. Now let's try Yeah, we need to
1:56:45create a J Unicorn start script that
1:56:49will tell J Unicorn how to run uh your
1:56:52application. Let's call it J Unicorn
1:56:55start. And let's use nano. I will paste
1:56:59all this directly here. But don't worry,
1:57:03you you should be able to find this on
1:57:06my blog. Yeah, I will make sure to
1:57:09deploy it uh in the future. So if you go
1:57:11to blogs, you will be able to see the
1:57:14individual articles here. So if you want
1:57:17to copy paste them uh quickly, you can
1:57:20do that. Don't worry about it. Or you
1:57:22can go to GitHub. I have this repository
1:57:25where I yeah basically documents these
1:57:28uh videos in a in text format. So we
1:57:32have foundation. So now we are in
1:57:35chapter 2. So you can click here and
1:57:37you'll should yeah will find everything
1:57:39here. So can just copy the commands and
1:57:42uh paste them uh in your VM. So okay
1:57:46let's continue. We forgot to add the
1:57:49shebang. So let's add it. Uh we are
1:57:53going to use bash. We have some
1:57:55placeholders that we need to change.
1:57:57Let's start with the user. So my user is
1:58:01IMAD and the group is also IMAD. So
1:58:05let's provide that. We have the number
1:58:08of workers. Let's set it to one because
1:58:10we don't have a lot of traffic and we
1:58:13are just testing. So as you can see I
1:58:16have this comment. increase this if you
1:58:17have more traffic or or a more complex
1:58:20application. But when you increase the
1:58:22number of workers, you will consume more
1:58:24RAM. So keep that in mind. The
1:58:26application directory is web app back
1:58:28end. That's correct. And we have this
1:58:31stocket file. We'll come back to this
1:58:33later. And here we need to give the
1:58:35project a name. Let's call this my
1:58:38website or something or a mad website.
1:58:41And here the worker is going to be
1:58:43Uicorn. That's fine. here forward allow
1:58:47ips. Yeah. So here here we just want to
1:58:50pass the IP of the user directly uh
1:58:53because yeah we will talk about
1:58:54engineext and unique sockets later so
1:58:57don't worry about this. Here we are just
1:58:59showing that we are starting this
1:59:01website. We move to the directory. We
1:59:05activate the environment. Here we create
1:59:07the socket file. Uh let's go back to
1:59:10sock file. We need to create this. Uh so
1:59:12this is important because it's going to
1:59:14communicate with engineext but we will
1:59:17talk about this later don't worry about
1:59:18it and here start junicorn so here we
1:59:22have the execute command j unicorn is
1:59:25inside the virtual environment we could
1:59:27have just uh yeah since we activated the
1:59:29environment we could just type j unicorn
1:59:32that's fine the application is inside
1:59:34the main file and we pass these
1:59:37arguments so the name of the project the
1:59:40number of workers
1:59:42the worker class which is uicorn. Here
1:59:45it is. Uh time out the user and the
1:59:48group. We bind this to the socket file.
1:59:52Uh we forward the IPs and we have some
1:59:55login. Great. So now save this. Uh I'll
1:59:59hit Ctrl S and then Ctrl X. And now we
2:00:02need to make this executable. So let's
2:00:05do sud ch mode plus x or before that I
2:00:09want I want you to understand the ch
2:00:12mode command. So before that let's run
2:00:14ls-la j unicorn start. You can see it's
2:00:18not executable. But if I run ch mode
2:00:22plus x j uh unicorn start and run the
2:00:28ls-la you can see now we have x and x is
2:00:32is available in the three users. So ruse
2:00:35can execute this file my user ID can
2:00:39execute this file and other users can do
2:00:42the same. Great and you can see that I
2:00:45am the owner of this file. So we
2:00:47configured the unicorn. Now we need to
2:00:50configure supervisor. We could again we
2:00:54could run this uh script manually but if
2:00:58it if for some reason J unicorn crashes
2:01:01we need to come back and uh rerun it
2:01:04again. We want we don't want to do that.
2:01:06We want supervisor to do this for us.
2:01:09Supervisor is not installed by default.
2:01:13Yeah, it's not recognized but we can
2:01:16install it using the op command. So
2:01:18let's do that. Supervisor.
2:01:21Let me provide my password. And it is
2:01:24installing. Great. Let's wait a little
2:01:26bit. Uh let's also create the logs
2:01:29folder because we we want supervisor to
2:01:32save logs um there so that we can uh if
2:01:37something hap bad happens we can uh see
2:01:40what happened there. So that's important
2:01:42and again verify that you are owning
2:01:45that uh directory. So run this and yeah
2:01:49we are running that since we don't use
2:01:52the sudo command we will be uh owning
2:01:54everything but just in case uh it's a
2:01:56good practice to keep checking the
2:01:58ownership. In order to configure
2:02:00supervisor we need to create this
2:02:03configuration file. So supervisor lives
2:02:06inside the etc folder and here we want
2:02:09to create our own configuration and let
2:02:13me paste this. Uh okay so we have so as
2:02:17you can see we want supervisor to run
2:02:19the J unicorn start start command. Uh so
2:02:23the only thing we need to change here is
2:02:25the user. It's going to be emad and here
2:02:29we need to change this placeholder and
2:02:32let's call it a mad website. Okay. And
2:02:36the log file we are going to store to
2:02:38yeah to store logs inside this. This is
2:02:40why we created this directory. And yeah
2:02:43um save this. Hit ctrl x. Great. Now we
2:02:47can start this service. Let's use pseudo
2:02:51supervisor
2:02:53ctl. And here we are going to reread the
2:02:56new configuration. It says available. It
2:02:59detected this uh new config that we
2:03:02created. And now we need to let's run
2:03:05this. Great. So we added this to the
2:03:08process group. And now let's check the
2:03:11status. It should be running. So if I
2:03:13type status uh and here you need to type
2:03:16that project name. Great. It says run
2:03:19in. It didn't crash. That's great. Let's
2:03:23see. So, we have logs. So, let's look at
2:03:26logs. Supervisor log. The file exists.
2:03:29Awesome. Let's do cat or maybe tail.
2:03:33Awesome. So, it's working. You can see
2:03:34that. Uh, let me let me look at the full
2:03:38output. So, these are just this is just
2:03:40configuration. Um, yeah, great. And here
2:03:45the application started. So we are
2:03:47starting J unicorn. Uh it's listening at
2:03:50this socket and here we have just one
2:03:52worker and the application startup
2:03:55complete. So this means that it's
2:03:56working. We can try and uh yeah let's do
2:04:00some curl commands. Oops, we don't have
2:04:02it here. Uh that's fine. Okay. Uh this
2:04:06is the curl command that we used
2:04:08earlier. Uh let's try to run it. And it
2:04:11fail. It says fail to connect to local
2:04:14host. Why this command failed? We have
2:04:17tested this before and it worked fine.
2:04:20Let me show you why this failed.
2:04:23Remember we have uh yeah let's look at
2:04:26the G unicorns tart uh file. You can see
2:04:29that we are using this socket file. So
2:04:34in order to communicate with the server,
2:04:36we need to use the socket file that was
2:04:39created here in back end. And let's
2:04:41verify that it is there. Yeah, as you
2:04:44can see, we have JU Unicorn stock and if
2:04:47I look at it, yeah, it's it's it's a
2:04:50different type of files in Linux. You
2:04:53can see that it says S, which means
2:04:55socket and this is used um it's like a
2:04:58pipe. It's used to uh by different
2:05:00applications to communicate with each
2:05:02other. In our case, we want uh the
2:05:04backend server to communicate with
2:05:06EngineX. Uh but we don't have EngineX
2:05:10yet. We are just preparing for that
2:05:12step. uh but uh to fix that we can do
2:05:15this. So with curl we can specify the
2:05:19Unix socket file and here here is the
2:05:22the socket file that we want to use and
2:05:25same here we are going to use this
2:05:27endpoint and curl will use this socket
2:05:30as a way to talk to the backend server
2:05:34and the backend server will send the
2:05:36response through the socket to curl and
2:05:39curl would display that. As you can see,
2:05:41I I run that and I got status. Okay,
2:05:45awesome. Now we can uh run tail again
2:05:49just to verify and yeah, we got this. So
2:05:51we uh so this is the get requests. We
2:05:55got 200. Okay. And yeah, it took less
2:05:58than 2 milliseconds. Awesome. So yeah,
2:06:01everything is working. Now if we run
2:06:04htop or I want you to install this. So
2:06:08we have a replacement um we have many uh
2:06:12applications like htop with good UI. One
2:06:16of them is called Btop and I really like
2:06:19Btop. Uh it looks like yeah this is a
2:06:22Btop and it's available on a lot of
2:06:25Linux distros. It's also available in
2:06:27Auntu. So that's great. And to install
2:06:30it we can as you can see it looks really
2:06:33really nice. So we are going to use this
2:06:34in order to monitor the processes,
2:06:36memory usage, etc. And to install it, we
2:06:40can run the APC command uh sudo up to
2:06:43install btop. Awesome. Now run btop and
2:06:46as you can see uh it looks really nice.
2:06:48So here you have the CPU usage, you have
2:06:51memory usage. Uh let me decrease the
2:06:54yeah the the font size. Okay. So we have
2:06:57uh so we have 8 GB of storage. We are
2:07:00using 65%.
2:07:03This is the swap um space that we
2:07:05created. It's not used. So we are just
2:07:08using 6%. This is the available memory.
2:07:11It's uh yeah almost 500 megabytes. We
2:07:14are using 35% of the memory. Uh and yeah
2:07:18we have also the network. Uh so we have
2:07:20the download and upload speeds. Um and
2:07:24here we have the processes. This is what
2:07:26we care about. And here you have filter.
2:07:29With the arrow keys, you can uh let's go
2:07:32to memory because that's what we want.
2:07:34And this is going to sort everything by
2:07:36memory. If you want, you can hit R to
2:07:39reverse this. But we want to yeah, we
2:07:41want this sorted in a descending order.
2:07:44And here type filter and run uh and type
2:07:48J unicorn. Why? We want to see the
2:07:51processes that are running. Here we have
2:07:53just one worker but you you can see that
2:07:56we have two processes. Why is that? So
2:08:01one J unicorn creates two services. So
2:08:05one will manage the workers. So one of
2:08:08them is the worker and the other one is
2:08:12uh the one controlling those workers and
2:08:15as you can see they are using uh roughly
2:08:1824 megabytes. That's not a lot because
2:08:20the application is not doing anything
2:08:22right now. But this is why I mentioned
2:08:26that if you increase the number of
2:08:27workers,
2:08:29you are going to consume more memory.
2:08:32Why I why I installed bet? Because I
2:08:34want you to I want you to learn about
2:08:38zombie processes. Um yeah, this is
2:08:41something very important because
2:08:42sometimes uh this happened to me. I
2:08:46didn't pay attention and I my memory
2:08:48usage was rising up quickly because I
2:08:51was I was not restarting J Unicorn
2:08:54correctly uh because sometimes when I do
2:08:57when I work on a new feature uh I push
2:09:00that to the server and uh restart the
2:09:03back end that created zombie processes.
2:09:06Basically some uh G unicorn processes
2:09:09were left here. they weren't doing work
2:09:11but they were consuming memory because
2:09:13they were from the old deployments. So
2:09:15let's see uh we don't have this problem
2:09:17right now. Uh we we have just created
2:09:20this uh but I will try to do uh to
2:09:23basically create zombie processes just
2:09:26to show you how this looks like and how
2:09:28to deal with this problem. Uh but first
2:09:30we need to go and SSH through the
2:09:32website. Great. So if you don't want to
2:09:36look at this, we can run um this command
2:09:39ps and we are going to grab and we are
2:09:43going to use I think aad websites. We
2:09:47have two of them. So yeah, this one and
2:09:50the other one. Uh this looks ugly
2:09:53because we have a lot of text but no
2:09:55problem. It confirms that we have two
2:09:57processes running. Now let me manually
2:10:00let's first go to web app back end and I
2:10:04want to run this manually just to create
2:10:07more processes. Uh let's do this. So J
2:10:10unicorn now hit enter. This is going to
2:10:12create uh yeah this is going to run
2:10:15again the the back end and it's going to
2:10:17create more processes. Uh let's wait a
2:10:20little bit. You can see that the memory
2:10:21usage is getting higher and higher every
2:10:24single time. Okay, good. And now if we
2:10:27want to we can also go inside uh that
2:10:30file
2:10:32G unicorn start. And um yeah let's
2:10:36change this from 1 2 3 save exit. And
2:10:40now we need to restart. We will run this
2:10:42command because we changed the
2:10:44configuration. We need to read that and
2:10:47restart again. Let's do this. So restart
2:10:50and let's uh use my project name. Uh I
2:10:53need to pass my password. Okay, it says
2:10:56stopped. Let's look at this. Okay, so uh
2:10:59the old processes were gone and now we
2:11:01have three new ones. Uh we should have
2:11:04just the manager is going to be
2:11:06controlling the the workers and we
2:11:09should have three workers because this
2:11:10is the new configuration that we
2:11:12created. But we also have how many? So
2:11:15one we have 1, two, three, four. But we
2:11:17have three orphaned processes or we can
2:11:21also call them zombie processes that are
2:11:23look consuming memory but not doing any
2:11:26useful work. And if you get this somehow
2:11:31uh this is how you solve this issue. You
2:11:33can't just come here and hit okay to
2:11:36kill the process because it's not clear
2:11:40which one is uh doing work. I mean you
2:11:44can sort of uh get that information by
2:11:47looking at the memory because the the
2:11:50workers that are busy are going to be
2:11:52using more memory but yeah the you need
2:11:55to confirm this and I'll show you how to
2:11:57do that. Uh let's run uh ps.
2:12:01As you can see we have uh yeah we have
2:12:04plenty of processes. Great. Now I have
2:12:08this uh command that we can run. uh it
2:12:11it's basically giving us the same info
2:12:13that we have here in Btop. I'm just
2:12:15showing you that running commands like
2:12:17these, it's not that fun, but looking at
2:12:20this uh in a terminal app, it's really
2:12:23uh useful and it's fast. But you can do
2:12:26the same thing with just Linux commands.
2:12:29And here are the processes and how much
2:12:31memory they are using. We can also try
2:12:33to uh sum the memory. So we are using
2:12:37almost 300 mgabytes. But yeah, we have
2:12:40some processes that we need to remove to
2:12:42free some memory. So the quick fix is to
2:12:45kill all J unicorn uh processes. And to
2:12:49do that you can either do you can do it
2:12:52here uh by hitting K and and you say yes
2:12:56I want to kill this but uh if you want
2:12:59to kill everything you can use uh
2:13:01process kill and this is the name of the
2:13:04process. uh but this will cause
2:13:07downtime. Your your application your
2:13:09users won't be able to access your
2:13:12application. Uh the front end will try
2:13:14to make requests to the back end but
2:13:15it's it's not going to respond. So they
2:13:17will get errors. So this is the easiest
2:13:21uh you do this and then supervisor will
2:13:24start everything up because it sees that
2:13:26something bad happened. Where are the J
2:13:28unicorn services? It's going to panic
2:13:30but it's going to uh yeah run them.
2:13:34Let's actually do it. So let's look at
2:13:36this. So we have uh all these J unicorn
2:13:39services. Let's kill them. Now let's go
2:13:41back here. You can see they are gone but
2:13:44supervisor create them created them
2:13:46quickly. You can see that we have four.
2:13:48That's the correct amount. We have the
2:13:50manager. Uh one of them is the manager
2:13:53and the other ones are workers. And you
2:13:56can see this is why we used supervisor
2:13:58in order to uh solve this issue for us.
2:14:01Great. Yeah. This is the the easiest uh
2:14:06solution and the quickest but if you it
2:14:08will cause some downtime but yeah it
2:14:11depends on your application. If you
2:14:12don't have a lot of users just do this
2:14:14one it's fine. Uh but if you have more
2:14:17users then yeah we shouldn't do it. Let
2:14:20me show you the other option. So I will
2:14:23again run the J unicorn start command.
2:14:28Uh okay, we created more or more
2:14:32processes.
2:14:33Let's restart. Okay, so you can see that
2:14:36the restart command is the one that's
2:14:38causing the issue. It's yeah, it re it
2:14:41reads the new configuration,
2:14:43but it doesn't delete the old processes.
2:14:47It just creates new ones. So this is the
2:14:50the big issue that we need to solve. And
2:14:52this is what I was doing uh in my
2:14:55websites running currently on
2:14:56production. I was just pushing new
2:14:59changes and I run restart before
2:15:03removing the old ones and this created a
2:15:06lot of orphaned uh processes. So we can
2:15:09see that now we need to remove uh three
2:15:11more processes. Okay, I show you uh I
2:15:14have showed you the easy method. Now
2:15:17let's try to solve this uh surgically
2:15:21without causing any uh downtime. First
2:15:25we need to ask supervisor for the
2:15:29process ID of the J unicorn process
2:15:32which which we can also call master
2:15:34because it is controlling the other
2:15:36workers and to do that we use sudo
2:15:40supervisor ctl P ID and here uh you type
2:15:43your project name so this is the P ID of
2:15:47the master process this is very
2:15:50important in in your case it's not going
2:15:52to be the same it's going to be a number
2:15:54But just uh remember it. Now let's look
2:15:57at the process hierarchy. We can use
2:16:00this command. Okay. So yeah, we have a
2:16:03lot of text. We can also do the same
2:16:05thing here uh in Btop. You can see that
2:16:07you have this tree column. Just uh hit E
2:16:12and as you can see it groups everything.
2:16:14So this is the master process. You can
2:16:17see that it's uh it is controlling these
2:16:19three processes and these ones they
2:16:22don't have a parent. So they are just
2:16:24here they are orphaned processes that
2:16:27are consuming a lot of memory uh but uh
2:16:31doing no work. So we need to kill them.
2:16:33So now this makes it easy. So I can just
2:16:36hit K and kill them one by one. But we
2:16:39can also see this in the ps-f
2:16:42command. Uh yeah I know this is this is
2:16:45a lot of text. So let me just copy this
2:16:48and let me open a text editor. So I'll
2:16:52open them in VS code. Uh so yeah this
2:16:56this stuff is not needed. So you can see
2:16:58that these are all the commands that we
2:17:01uh are used in order to run the s the
2:17:04yeah the junior core service. We don't
2:17:07want them. So let's just delete this
2:17:08stuff. Uh we don't need this two
2:17:12uh and this. Okay great. Let's zoom in a
2:17:15little bit on this part. Okay great. So
2:17:19let me also go back to the terminal. Let
2:17:22me also copy the master ID. So this is
2:17:26the master process ID. It's this one.
2:17:30You can see that how many we have three
2:17:33of uh we have 1 2 3 4 5 6 7 and eight.
2:17:38Where is the the process the manager?
2:17:40Here is the manager. So uh and here I
2:17:43forgot to add the column names. Uh I
2:17:47think uh let let me go back. I just want
2:17:49to confirm something. Let's go here.
2:17:52This is the process ID. This is the
2:17:55parent process ID. RSS. Yeah, we don't
2:17:58need this one. So let's remove it. We
2:18:00just want these two. Let's see. We have
2:18:04uh yeah, these are the three UVON
2:18:07workers and they are attached to this
2:18:11master process ID. And if we look at
2:18:14this, it's going to point out to this
2:18:16one. And this is that fourth J unicorn
2:18:20service. It's this this is the same as
2:18:23this. We have these. So let me double
2:18:26click on this one. As you can see, here
2:18:28is the process ID. It's 531 402 531 402.
2:18:33And it is attached to the parent. Uh
2:18:36let's look at the parent. Yes, it's this
2:18:38one. 531399.
2:18:41This this is your group. It's exactly
2:18:44it's exactly this. You have one process
2:18:47ID and one master process and their
2:18:51children. But we have these three
2:18:54strugglers, these three orphaned
2:18:56processes. They are these ones. So 274,
2:19:00272 and 273. Here they are. And the PP
2:19:04ID is equal to one. Hm. What does this
2:19:07mean in Linux? P1
2:19:11is the system in it process. This means
2:19:15these processes lost their parent but at
2:19:19least they were adopted by the init
2:19:22process because they they need to be
2:19:24attached to something. This is how you
2:19:27detect orphaned processes. If they are
2:19:30not attached to a process ID uh if they
2:19:34lost their parents they will be attached
2:19:36to in it. And now we can safely delete
2:19:38these. And we have the process ids. So
2:19:41we can do it uh here with the help of
2:19:44the top. Let's do it once for this one.
2:19:47So we hit okay. And yes.
2:19:51And it should Yeah, it's gone. Bye. We
2:19:54no longer have that zombie process, but
2:19:56we have the other two. Uh if you want to
2:19:58do this with the terminal, you can just
2:20:00uh use pseudo kill. And here you provide
2:20:03the IP addresses or the process ID
2:20:06sorry. Uh you pass them like this one
2:20:09after the other. Now hit enter. Go to
2:20:12Btop and they are gone and we freed
2:20:15memory. You can see that uh we we had to
2:20:17do a lot of stuff in order to detect the
2:20:19processes that uh are not used by uh our
2:20:23back end. But this this is useful if you
2:20:27don't want to cause any downtime at all.
2:20:29Your application code is on the server.
2:20:32We built it. We run it and we everything
2:20:37went smooth. The back end is managed by
2:20:39supervisor and listening on a private
2:20:43socket file. However, no one from the
2:20:46outside world can access your
2:20:48application yet. This is a problem that
2:20:50we will fix in the next sub chapter. We
2:20:53are going to talk about engine X
2:20:56security headers and stuff like that.
2:20:59See you in the next episode. Hi
2:21:01everyone. In the previous section, you
2:21:04successfully started your backend and
2:21:07set up a process manager. Your API is
2:21:10alive and listening on a Unix socket
2:21:14file. However, your users cannot access
2:21:17your website. Your front end is also
2:21:20just a folder of static files sitting on
2:21:23the server waiting to be served. In this
2:21:27sub chapter, we are going to talk about
2:21:30reverse proxy and security headers.
2:21:33First, we need to sync the front end and
2:21:36backend URLs. After that, we will
2:21:39install EngineX to act as a reverse
2:21:42proxy. I will explain this once we
2:21:45install Engine X. I'm going to explain
2:21:48to you what does 502 bad gateway means.
2:21:53So this is an error that we might get if
2:21:56we don't set up enginex correctly. And
2:21:59finally we will move to the security
2:22:01part. In this sub chapter we are going
2:22:04to focus only on the HTTP security
2:22:07headers mainly CSP and there are other
2:22:10ones but CSP is the one that that
2:22:13requires a lot of work and we will focus
2:22:15on it at the end of this video. Now
2:22:18let's SSH to the machine because we need
2:22:21to start by syncing the front end and
2:22:23back end and I will come back to the
2:22:25slides in if I want to explain some
2:22:28diagrams. As you can see I am logged in
2:22:31to the machine and before we configure
2:22:34engine X you need to prepare your code.
2:22:36You want your front end and back end to
2:22:39act as if they live on the exact same
2:22:42server both in local development and in
2:22:45production. This avoids complex cross
2:22:48origin issues. Let me show you what I am
2:22:51talking about. As you can see, I need to
2:22:53go to the web app directory because last
2:22:57time we moved the source code here. I
2:22:59will change ls to cd. Now I am inside
2:23:03that directory. Let's start with the
2:23:05front end. Let's go to front end. And
2:23:09here we need to go to source. And we
2:23:12have main.js. So we need to open that
2:23:17file. Let's use nano and let's see. As
2:23:21you can see here we are using an
2:23:23environment variable v a base url. Let
2:23:27me show you the content of that
2:23:29variable. Let's do ls. And here I need
2:23:33to go back to front end because here we
2:23:37have the environment file as you can. So
2:23:40let's do ls-la.
2:23:42Yeah, you can see that we have env.ample
2:23:46and we also from that we create the env
2:23:49file. But let's look at env.ample
2:23:52by using cat. So let's provide the file
2:23:57and as you can see this is what we have
2:24:00here. The back end runs on this port.
2:24:04When you are developing this locally the
2:24:07back end will be running on a different
2:24:08server. And in the front end we use
2:24:10Axios to make HTTP requests and it needs
2:24:15a it needs to know where the the API
2:24:17lives and here it lives in port 8000.
2:24:22This is good but it introduces cross
2:24:26origin issues. If we try to send a
2:24:29request to this port, the browser will
2:24:31block it because it sees that you are
2:24:33trying to communicate with a different
2:24:35origin and even though we are running on
2:24:37local host, it it's not going to be
2:24:40allowed because the front end runs on
2:24:42port 8080
2:24:44but your back end runs on port 8,000. So
2:24:48that's how I started developing this but
2:24:50I will show you a simple way to avoid
2:24:52this problem. We are going to use vit
2:24:55and with vit we have something called
2:24:57proxy and it will make this very simple.
2:25:00So this example we no longer need it.
2:25:04That's why I wanted I wanted you to open
2:25:07the main file and change it. So let's do
2:25:10that quickly. Now you can see that here
2:25:12I am using axios and the base URL is we
2:25:16set it here. I I will remove all of
2:25:18this. I don't need it. And here instead
2:25:21I will just put slash. By doing this the
2:25:25browser will automatically send requests
2:25:27to the current host. So if I am running
2:25:29on local host this will will tell the
2:25:32browser to send it to local host. If I
2:25:34am running on my domain it's going to be
2:25:38that domain. That's the first change
2:25:40that we need to make. Now save the file.
2:25:43Exit uh with Ctrl X Y and enter. Great.
2:25:48Now we need to configure vit. So let's
2:25:50look at vit. It's not here. Uh it's it
2:25:53must be here. Yes. So we have vit
2:25:55config. Let's open it with nano vit
2:25:58config. Now I want you to scroll down
2:26:01here. You can see that we have server.
2:26:03And this is uh where we set the port. By
2:26:06default vit starts on on a different
2:26:10port. I think 4,000 something. But we uh
2:26:13I usually change it to 8,080.
2:26:17And here we can add another field to the
2:26:20server block and we we call it proxy.
2:26:23Here is the the field that we need to
2:26:25add to server. You can see that it's
2:26:27called proxy. And here the backend API
2:26:31is yeah it has this prefix / API and we
2:26:36tell V to send that request to this URL
2:26:40or or to this target. This means that if
2:26:43the front end tries to send a request to
2:26:48the API, vit will intercept that and
2:26:52target this URL. This way we can change
2:26:56origin without having to worry about
2:26:59cross origin issues. Let me explain this
2:27:02with another diagram that I created. You
2:27:05can see that here we are on a local
2:27:07machine. we are developing the the web
2:27:09application and here I I am using let's
2:27:13say Chrome and I want to make a an HTTP
2:27:17request. So I I am in the browser. If I
2:27:20want to load static files basically the
2:27:23web page I will call I will be here. So
2:27:27the path will not be API. I will just
2:27:29talk to the front end. The front end
2:27:32will send back the HTML, CSS and
2:27:35JavaScript files. Vit will take that and
2:27:38send it back to the browser and the
2:27:40browser will display the web the website
2:27:43for me. Good. But let's say that I am
2:27:45that I want to search for an article or
2:27:47something that that request will need to
2:27:50talk to the back end. So vit will see
2:27:53the the request that I'm trying to make
2:27:56or the browser is trying to make. If it
2:27:58sees that it has this prefix / ai, it's
2:28:02it's going to send it to the back end.
2:28:04So, VIT sits between the browser and
2:28:08your front end and backend servers and
2:28:11this implementation
2:28:13solves the cross origin issues. Awesome.
2:28:17Now, we need to go to the to the back
2:28:18end. Let's do that quickly. So, we are
2:28:22in the backend folder. Let's see what we
2:28:24have. So, we need to look at main.py.
2:28:26Let's open it with nano. And here we
2:28:29need to remove some things. You can see
2:28:31that here we have a coarse middleware.
2:28:34This is what we needed before using the
2:28:37proxy feature that vit uh gives us. So
2:28:41we need to remove this class. And now
2:28:44let's go down a little bit because here
2:28:47as you can see we used that middleware
2:28:49here because we we allowed the following
2:28:53origins. So this block need we need to
2:28:56remove it. And in order to do that, hit
2:29:00alt a go down and hit ctrl k. That will
2:29:04remove the those multiple those lines.
2:29:08Now we need to yeah save this and go
2:29:12back. Okay. So now the we don't have
2:29:16course issues. everything will the front
2:29:18end will be able to communicate with the
2:29:20back end because as as you as you saw we
2:29:23used uh the proxy feature that v comes
2:29:27with. So after making this change the
2:29:29back end we don't need to to do anything
2:29:31in the back end but the front end must
2:29:33be uh built again. So we need to build
2:29:36the front end. So go back to the front
2:29:38end folder and run pnpm build. Why we
2:29:42need to do that? because the build
2:29:44process would regenerate the the HTML,
2:29:47CSS and JavaScript files and it will
2:29:49bake in the changes in those generated
2:29:52files. But the back end doesn't need
2:29:53that because it doesn't have a uh build
2:29:57step. So just run pmppm build and let's
2:30:00wait for the build process to finish.
2:30:03Awesome. The build finished. Now the
2:30:06first step in this video uh is done. We
2:30:10synced the back end and front end URLs.
2:30:13And now we can move to using EngineX.
2:30:16But why do we need enginex? EngineX, you
2:30:21can think of it as a front door for your
2:30:24server. When a user types your domain
2:30:27name into their browser, the request
2:30:29will hit X first before loading HTML
2:30:33files or before talking to the back end.
2:30:36So engine X is going to intercept that
2:30:38request and it's going to act as a
2:30:41reverse proxy meaning it will decide
2:30:45what to do with that request. Let me
2:30:47show you another diagram that I created
2:30:49to explain this in detail. I am back and
2:30:53let's let's go to the next slide. Here
2:30:55is the diagram that I created in order
2:30:58to explain why we need engine X. As you
2:31:02can see, it's similar to the diagram
2:31:04that I showed you that explains vit
2:31:07proxy. Here we start with the user with
2:31:10a browser. We send a request before
2:31:14talking to the front end or back end. We
2:31:18in EngineX will intercept that request.
2:31:21If it sees that the path is slappi
2:31:26something, engineext will talk to the
2:31:29Unix socket and that Unix socket will
2:31:32send the request to J Unicorn and J
2:31:35Unicorn will send it to the UVORN worker
2:31:39which runs our back end. You can see
2:31:41that this is the part that we talked
2:31:43about in the previous sub chapter. Let's
2:31:46go back here. When we talked about SSH
2:31:50tunneling, we mentioned that the front
2:31:52end is running here and the back end is
2:31:54running on the server and we explained
2:31:57how that works. We have the supervisor,
2:32:01we have J unicorn and our workers and
2:32:04here we mentioned that we created a Unix
2:32:07socket file and I told you that I will
2:32:10explain to you where that will be used
2:32:13and in this video we are using that to
2:32:15communicate with EngineX. Let's go back.
2:32:18You can see that we have the unicorn.
2:32:19Here I simplified the diagram. I am
2:32:22showing just one worker, but you get the
2:32:24idea. You have your Unix socket. It is
2:32:27used in order to allow EngineX and J
2:32:31Unicorn to communicate between each
2:32:32other. You can see that here in the
2:32:34description I I said it's a two-way pipe
2:32:38connecting EngineX and J Unicorn. Uh so
2:32:41let's say that the user is searching for
2:32:44an article. Let's say that the endpoint
2:32:46is / API/ articles. That request will be
2:32:50intercepted by EngineX. It sees that it
2:32:53has the / API prefix. It's going to send
2:32:56that to Unicorn, but it's going to do it
2:32:59through the Unix socket file because
2:33:02here we are inside the VM. And in order
2:33:04for two programs to communicate with
2:33:06each other, we used Unix sockets files.
2:33:09So now the J J unicorn receives a
2:33:13request to the / API/ articles endpoint.
2:33:17It's going to give that to the fast API
2:33:19endpoint. If we have an endpoint, great.
2:33:23The code will run. We'll we'll try to
2:33:26fetch articles. Now we will send that
2:33:28response to engine X. So we will move uh
2:33:32the response will go to G unicorn. We'll
2:33:35will go through the socket and back to
2:33:37EngineX. engineext sees the response.
2:33:40It's going to send it to the browser and
2:33:42yeah, you will see articles. So that's
2:33:44why here we use unique Unix sockets and
2:33:48you can see on the other side we have
2:33:50just the static files and if we if if we
2:33:53have a request to an end points that
2:33:55doesn't start with / API we will go here
2:33:58we will fetch the static files which are
2:34:00HTML, CSS and JavaScript and the browser
2:34:03can display that. So that's why we need
2:34:05enginex and as you can see it works like
2:34:08uh vit but this runs on production. You
2:34:11use vit locally when you develop your
2:34:14application but on the virtual machine
2:34:17or on the digital ocean droplet we use
2:34:20engine x. It's the same concept but one
2:34:23runs on your laptop and the other one
2:34:25runs on the virtual machine. I hope that
2:34:28my explanation was clear. Now let's
2:34:31install enginex. We will do it this way.
2:34:34We are we are on Ubuntu. So we can
2:34:36install this directly from the
2:34:38repository. I will provide my password.
2:34:41And let's wait for a few seconds.
2:34:44Awesome. We installed EngineX. You need
2:34:47to understand that EngineX uses
2:34:49configuration files to know how to route
2:34:52traffic. Let's create a new
2:34:54configuration file for your website in
2:34:58the sites available directory. Now let's
2:35:01do this. pseudo nano etc engine x and we
2:35:06have a directory called sites available
2:35:09and here you put your project name let's
2:35:12call it my website maybe.com uh com we
2:35:18are going to build this configuration
2:35:21block by block so that you can
2:35:22understand everything let's start with
2:35:24the upstream block as you can see I just
2:35:27copy copied this from GitHub and you
2:35:31just I we just need to change this
2:35:34placeholder. I will use the same name
2:35:37that I used for the configuration file.
2:35:40And if you are wondering where to get
2:35:43the code or yeah the the things that I
2:35:46copy paste, you will find them on this
2:35:48GitHub repository. Here it is. So if you
2:35:51go to full stack deployment handbook,
2:35:54you will see that here I am documenting
2:35:56everything. Click on the book folder you
2:36:00have and here you have the chapters. So
2:36:02far we completed chapter one. Now we are
2:36:04in chapter number two. So if you click
2:36:08on the uh the second markdown file let's
2:36:11search for upstream.
2:36:13You can see that here is the upstream
2:36:15block. I you can copy it from here and
2:36:18paste it in nano and uh go from there.
2:36:22So so that you can uh go quickly. So
2:36:24don't worry about the code. Everything
2:36:25will be shared in the video description.
2:36:28First, as I said, start by changing that
2:36:30placeholder here. I use my website com,
2:36:33but you can use any other name. So, this
2:36:37acts as a variable that we can reuse in
2:36:40other blocks. Instead of typing every
2:36:42single time Unix, web app etc., we will
2:36:46just use this variable to make the code
2:36:49clean. Here you can see that we are so
2:36:52here this is a variable. Here we have
2:36:54fail timeout is equal to zero. This is
2:36:58an important setting because it tells
2:37:00engineext to never mark the back end as
2:37:03down even if it fails to respond. So uh
2:37:06if you have a lot of traffic and your
2:37:09back end is starting to respond slowly.
2:37:11Sometimes the engineext will drop the
2:37:14request if it if it doesn't respond
2:37:16quickly. But here we tell engineext
2:37:19don't worry uh even if the back end is
2:37:21down don't fail. Okay, now let's add a
2:37:25different block. It's called the server
2:37:28block. This tells EngineX to listen for
2:37:32incoming web traffic on port 80, which
2:37:35is standard HTTP, and to respond when
2:37:38someone asks for your specific domain.
2:37:41Right now, we haven't created a domain
2:37:44name, but that's not a problem. We will
2:37:48use the droplet IP here in the server
2:37:51name. But later this will be changed
2:37:54from the droplet IP to the domain name.
2:37:58So let's do this. If you don't know how
2:38:00to get your droplet IP, we have
2:38:02different ways of getting this but you
2:38:04can go to digital ocean. Make sure to
2:38:07log in here in the projects. Make sure
2:38:10to select the project where you created
2:38:11the droplet. And here as you can see we
2:38:14have the IP address. So copy it and
2:38:17let's go back to nano and let's paste it
2:38:20here. I will remove this and replace it
2:38:24with the droplets IP. You can see that
2:38:26we also have some log files. And here we
2:38:29need to change this. Uh these are also
2:38:32placeholders.
2:38:34And here I will use again let me copy
2:38:36this my website com and let's place it
2:38:40here. Let's also do it here. This is
2:38:43helpful because if something breaks, we
2:38:45can check the error logs to understand
2:38:48why that happened. Here I have a comment
2:38:52that says we will add the location block
2:38:54some other blocks here next. But as as
2:38:57you can see we are doing this step by
2:38:59step to understand every every line that
2:39:02we are writing. Okay. Now I will remove
2:39:04this comment again. I will use Ctrl K to
2:39:07do that. And let me place the location
2:39:10block. Remember here we have slash which
2:39:14means that we want to serve the front
2:39:16end. If I open the browser and go to
2:39:18this IP address and I add a slash at the
2:39:21end, engineext will catch that and will
2:39:25serve my front end to the user. Now
2:39:30let's try to understand what we have
2:39:31inside here. We have root. This tells
2:39:35engineext where to look for files. In
2:39:37this case, it points to the dist
2:39:40directory because when you run pmppm
2:39:43build, the files that are generated are
2:39:46going to be inside the disc folder. Try
2:39:49files is important because here we have
2:39:52a single page application. It tells
2:39:55engineext to find the exact file the
2:39:59user asked for and that file is going to
2:40:02be here dollar uri. If it is not there,
2:40:05it's going to look for a directory. So
2:40:08if it this is a directory, it's going to
2:40:11look inside it to find the file. If it
2:40:13can't find the file, it's going to use
2:40:16index.html
2:40:17as a fallback. Let me explain this. So
2:40:20here I have my personal website. I can
2:40:23open it. You can see that here we have
2:40:25slash at the end which means that we are
2:40:27serving the front end files. If I want
2:40:30to open something else like maybe my
2:40:34resume, I can do it. So I can do /resume
2:40:38and you can see we get it. But how does
2:40:41this work? If I go back to the machine
2:40:44or to the VM, let's save this file.
2:40:47Let's hit Ctrl X. Let's look at the this
2:40:51folder. What do we have here? We can see
2:40:53that I have a lot of files. So we have
2:40:57some images. We have robot txt. Let's
2:41:00let's also try to get robots txt. robots
2:41:02txt. You can see that I am able to see
2:41:05this file. Why? Because engine x knows
2:41:08where to find those because of this. Uh
2:41:12let's go back to nano because of this
2:41:15location block. So try files will uh URI
2:41:19will be robots txt
2:41:22and uh we know that we should look
2:41:24inside the disc folder. If we find it
2:41:26good, we will show it. But if we can't
2:41:28find it, we will show index.html.
2:41:31Okay. And uh yeah, index html is also
2:41:35should be here. Uh you can see here here
2:41:38it is. So that's that's how that's uh
2:41:41the reason why we added this location
2:41:43block. And this will serve the your
2:41:45front end. We will do another one for
2:41:48the back end. And it's going to look for
2:41:50/ API instead. I will copy it and I'll
2:41:53paste it here. You can see that it is
2:41:56similar. we will use the location block
2:41:58and instead we will look for /
2:42:02okay now we have um a lot of directives
2:42:05the these are called directives so proxy
2:42:08pass is a directive and here we have we
2:42:11need to change this placeholder from
2:42:13your project name to this string so
2:42:17proxy pass will the request over to your
2:42:22J unicorn socket that's what I explained
2:42:25in the slides we are here. So we use the
2:42:28unique socket to allow EngineX to
2:42:32forward the request to J unicorn and
2:42:35this is what we are doing here in this
2:42:37directive. The proxy set header we have
2:42:41two of them are crucial here because
2:42:43they take information about the real
2:42:46user who is trying to access your
2:42:48website because if we don't do this we
2:42:51will all we will send the IP address of
2:42:54your virtual machine and we don't want
2:42:56to do that and finally we have proxy
2:42:59redirect of tells engineext to trust the
2:43:02redirect URLs exactly as uh your backend
2:43:07sends them. So if your back end tells
2:43:09EngineX to redirect, EngineX will trust
2:43:11it. It's not going to do uh something
2:43:13else. Okay, great. So now we have the we
2:43:16we have two two locations, one for the
2:43:18front end and one for the back end. So
2:43:21we can save the file and let's exit. We
2:43:25have two options to test this
2:43:26configuration. We can do it live here on
2:43:29the server. But if you aren't unsure or
2:43:32if you already have a working website
2:43:34and you want to introduce changes to
2:43:36your configuration, if you do if you
2:43:38have a mistake or if the configuration
2:43:41is false, that will cause downtime and
2:43:44we don't want that. There is another
2:43:45option which is to use an a website
2:43:50called EngineX Playground. EngineX
2:43:52playground is a website that allows you
2:43:55to simulate your routing rules safely in
2:43:58your browser without uh having to do
2:44:00this on your production server. You can
2:44:03see that here we have we put the
2:44:04engineext configuration. Uh here we have
2:44:07events HTTP. We as you saw we didn't
2:44:10write these. I'll explain to you why.
2:44:12But we need to have we need to add them
2:44:15for the configuration to work. the the
2:44:17the blocks that we created are these. We
2:44:20created server location. We created two
2:44:23of them. Here we are listening on 80
2:44:25because we we used uh HTTP. Here we are
2:44:29serving the front end and here we are
2:44:30serving the back end. This playground
2:44:33has a built-in testing back end running
2:44:36on port 7777
2:44:38and we can use this as our back end.
2:44:40Here you write your curl commands to
2:44:44yeah send HTTP requests. We'll start
2:44:47with local host. Make sure to use local
2:44:49host. You can't use your own domain
2:44:51because this is a playground and we are
2:44:54testing things internally. To test the
2:44:57cur command you hit run. And as you can
2:44:59see the output says serving view.js JS
2:45:03files which is correct because here we
2:45:05have slash and the this location block
2:45:08will catch that and it will in this case
2:45:11return this string. Uh but let's add uh
2:45:14maybe something else. So let's add API
2:45:18v2. Will this work? Let's see. Let's hit
2:45:22run. And that works because here we have
2:45:25/ API and that catches this. And here
2:45:28you can see that we have u yeah this is
2:45:31a response from the back end. Uh let's
2:45:33do something else maybe let's do API
2:45:39v1 or slash articles. Uh let's remove
2:45:42this. Let's send the request and it
2:45:45works. So this block will catch this.
2:45:47But maybe let's do something else like
2:45:50maybe resume or something. Yeah that
2:45:53will be catched by this block. So
2:45:55everything is working. And if you add
2:45:58complex um I don't know if you work on
2:46:00complex configurations this playground
2:46:03will be very useful. So we know that our
2:46:06configuration works. Now we can close
2:46:08this go back to the VM and I want to
2:46:13tell you why in our configuration we
2:46:16don't have the events block and HTTP.
2:46:20Okay, let's close the file. And here on
2:46:23Ubuntu, we use a modeler system for
2:46:26EngineX. There is a master configuration
2:46:29file located here. Uh let's look at it.
2:46:32Let's do cat etc. EngineX enginex
2:46:38doconfig. Hit enter. So this is uh this
2:46:42is the master configuration file. Let's
2:46:44scroll to the top. You can see that we
2:46:46have events.
2:46:48We it's already here and we have the
2:46:50HTTP block and if we scroll down uh we
2:46:54include some things and what do we
2:46:55include? If you look at this path it's
2:46:57etc engineext sites enabled here we have
2:47:00an asterisk to load every configuration
2:47:03inside that directory and if you
2:47:06remember the configuration that we
2:47:08created is inside this folder. So that
2:47:11means that this master file will load
2:47:14our configuration. That's why we didn't
2:47:17we don't need to add events or HTTP. It
2:47:20is it's already there. And one thing
2:47:22that you need to understand is that the
2:47:24engineext will run as this user. It's
2:47:27not going to run as my user or the user
2:47:30you created. No, it's going it has its
2:47:32own user. It's going to use it. So, it's
2:47:34ww
2:47:36data. Now, we need to enable this site.
2:47:40Um, let's see. So, we have pseudo nano.
2:47:43Okay, good. But I want to use uh ls
2:47:47instead of nano. So let's see what we
2:47:50have inside the engine x. You can see
2:47:52that we have two folders. We have sites
2:47:55available and sites enabled. So sites
2:47:58available uh we define the
2:48:00configuration. So just so that engineext
2:48:03knows that there is a maybe a
2:48:05configuration that's that that it needs
2:48:07to take into consideration. But for it
2:48:11to work we need to put it in enabled.
2:48:13And here we are not going to copy paste
2:48:15it because that's that's not
2:48:18[clears throat] a good bracket because
2:48:19you would need to maintain both. Instead
2:48:21we are going to use a symbolic link. Uh
2:48:24this is uh just a shortcut that will put
2:48:27that file here and if we want we can
2:48:30just change the configuration inside
2:48:33sites available and that change will uh
2:48:35take effect in the other directory. So
2:48:38we we'll have it in two places but we
2:48:40can change it once and it will take
2:48:42effect in uh other places. Here is how
2:48:45we create a symbolic link. We use ln
2:48:49dash s here we need to change this. Uh
2:48:54let's see I need to put my configuration
2:48:56my name the name here. Uh let's go here
2:49:00and let's do it in size available
2:49:02because I can copy it easily. Let's copy
2:49:06this and let's paste it inside sites
2:49:09enabled. So we are creating a symbolic
2:49:12link from here to this directory. Now
2:49:16hit enter. Great. Let's look at let me
2:49:20copy this uh or let's copy the other
2:49:22one. Let's do etc. And here I will add
2:49:26ls-la.
2:49:28And as you can see here it says l which
2:49:30confirms that is a symbolic link. As you
2:49:32can see we have this it is pointing at
2:49:35this file which means that if this file
2:49:37change changes this uh file will also
2:49:40change because it is a basically a
2:49:43shortcut. We are almost ready. So now we
2:49:45need to make sure that the configuration
2:49:47is correct. So run this command and what
2:49:50you should see is this. It should say it
2:49:53should say that the syntax is okay and
2:49:56the the test was successful. If you see
2:49:59an error or something, please fix it
2:50:01because if you use a configuration that
2:50:04is not correct, it's the your websites
2:50:09will not work or people will not be able
2:50:11to access your website. Awesome. So now
2:50:14let's reload X so that it it can read
2:50:17the configuration. And now we cannot
2:50:20access the website. So I can take my IP
2:50:24address. Let's copy it. Uh let's go here
2:50:27and let's paste it. You can see that
2:50:29it's not going to work because we have
2:50:32the firewall. We need to allow traffic
2:50:34through port 80. And in order to do
2:50:37that, we have we can do this. We can
2:50:40allow EngineX. You we use sudo here. We
2:50:43use the uncomplicated firewall and we
2:50:46tell it to allow EngineX. Let's do that.
2:50:49Okay. And now let's verify the status.
2:50:51Sudo UFW
2:50:54status. Uh before as you saw as you can
2:50:57see we we had just open SSH so that we
2:51:00can SSH to the machine but now we also
2:51:03have enginex which means that we can
2:51:05talk to port 80. Now if I go back if I
2:51:08go to the website as you can see it's
2:51:10already worked. Before it it it wasn't
2:51:13uh working. It couldn't find the the
2:51:16machine or the the firewall was blocking
2:51:18it. But now as you can see I can refresh
2:51:21the page and I uh I see the front end.
2:51:24You can see that the website is not
2:51:25secure because we are on HTTP. We will
2:51:28solve this in the upcoming chapters and
2:51:32you as you can see we have some errors
2:51:34because we don't have articles yet in
2:51:37the brows in the application but the
2:51:39front end is running and the back end is
2:51:41also running. So if I click on blogs or
2:51:45courses you can see we cannot we cannot
2:51:47get them because they are not here. And
2:51:50in my web application I am using mill
2:51:53search and we will talk about that in
2:51:55the next chapter. But we also have a
2:51:58problem here. So uh let's see. So if I
2:52:02open the console, let's zoom in a little
2:52:05bit. You can see that I am not able to
2:52:09make a request to the API endpoint.
2:52:13Interesting. So let's go to the VM and
2:52:17let's understand why is this happening.
2:52:20Remember we added log folders. So here
2:52:25let's do this. You can see that here we
2:52:27have error logs for our website. Let's
2:52:30take this and let's put it here and
2:52:33let's look at what we have here. We
2:52:35don't have anything. Okay, that's
2:52:37interesting. And let's look at access.
2:52:39So here access log that's me trying to
2:52:42access the website. So you can see that
2:52:43I made a post request to these
2:52:45endpoints. So here we have tracking and
2:52:48here we have search and yeah and I was
2:52:51using uh Mozilla I'm using Firefox. So
2:52:55yeah that's uh what you see in the
2:52:58access logs but we don't see any errors.
2:53:02Let's look at the supervisor logs. For
2:53:04that we need to go to back end. Let's
2:53:08see. Yes, we have logs. Let's go inside.
2:53:11What do we have here? We have
2:53:12supervisor.log.
2:53:14So let's let's do tail supervisor.log.
2:53:19And as you can see we see the problem.
2:53:21So here it says that uh failed to
2:53:24establish a new connection to me search.
2:53:28So here it is me search communication
2:53:30error. That's what that's why uh what I
2:53:32said but we will fix this in the
2:53:34upcoming chapter. Let's make sure that
2:53:38we can also reach the back end. So here
2:53:41the back end starts at API slash and let
2:53:45me do health because we have an endpoint
2:53:47that tells us if the back end is alive
2:53:49or not. So as you can see we are
2:53:51receiving okay which means that this is
2:53:53working. Uh we can confirm this in uh
2:53:57the main file we have this endpoint
2:54:00API/halth
2:54:01and yeah it means it is working. If you
2:54:05followed the previous sub chapters, you
2:54:09shouldn't get a bad gateway error. But
2:54:11if you do, that might happen because of
2:54:13the engineext user doesn't have the full
2:54:16permissions to to access the Unix socket
2:54:19file. Uh remember the Unix socket is
2:54:22here. You can see that it is called
2:54:24junicorn.sock.
2:54:27And if we try to let's look at the
2:54:29permission. So if we do ls-la
2:54:32juniors
2:54:34sock, here is what we have. Uh I want to
2:54:37correct a mistake that I made in the
2:54:39previous video. So here the first three
2:54:43characters are for the current user. So
2:54:46the current user is IMAD. If you are
2:54:49connected as root then it's going to
2:54:51apply to root. The second tree are for
2:54:54the group and the final three are for
2:54:57other users. So the mod here is 777.
2:55:02Why? Because here uh we have three of
2:55:05them. So every person or every user can
2:55:08read, write or execute this file. This
2:55:12is not good. We need to make sure to
2:55:16restrict this. This is why the we didn't
2:55:19get bad gateway because everything is
2:55:22open. Everyone can uh access read this
2:55:24file. But we need to change this or
2:55:28maybe if you have a different mode then
2:55:31you might get a bad a bad gateway. Let's
2:55:33try to do this. So uh I'll change this
2:55:37intentionally just to show you how to
2:55:39debug this problem. Let's use ch own and
2:55:43let's change the ownership to 700
2:55:48and let's point it to let's do it for j
2:55:50unicorn.stock. What does 700 mean? So
2:55:54this is in binary and let's take these
2:55:58three. The first character is one, the
2:56:00second character is two and the third
2:56:03character is four. If you sum those
2:56:05three numbers, it's going to give you
2:56:06seven. So if all of them are on, if you
2:56:10want to allow read, write and execute,
2:56:12you put seven. If you don't want to
2:56:14allow anything, you put zero. So here it
2:56:17means that for the current user, it can
2:56:20do everything anything with this file.
2:56:23But for the the group they can't and for
2:56:26other users they cannot uh do anything
2:56:29with it. If you want to allow people in
2:56:32in the group to just read the file maybe
2:56:36read and execute in that case we will
2:56:38put five because four + 1 is five. Let's
2:56:42do this. So ch okay so it said uh
2:56:46operation is not permitted. Let's use
2:56:48pseudo. Okay that works this time. Let's
2:56:51go back to ls-la.
2:56:53Sorry, I did ch own. I I should have um
2:56:57changed this to ch mod because these are
2:57:00called mods. Uh so yeah, this time it
2:57:02worked. Uh let's do ls- array. And as
2:57:05you can see it changed. Let's verify
2:57:07that. So the three characters here for
2:57:10other users are not available which
2:57:12means they are set to zero. The second
2:57:15three are read and execute. That's
2:57:17that's uh five. And uh the last three
2:57:21for my user is seven. So I can do
2:57:23whatever I want with this file, but
2:57:25other users cannot. So this is very
2:57:27important. And remember you engine x is
2:57:32not inside this group. It's not my user.
2:57:35It's ww- data. Okay. Now let's try to do
2:57:40this. Uh, as you can see, we get the 502
2:57:44bad gateway error because EngineX is
2:57:47trying its best to send this request
2:57:50that I am trying to make to API/halth
2:57:54to J Unicorn. Let's go back to the
2:57:56slides. It received / API/halth. It
2:58:00tries to send it to J Unicorn, but the
2:58:02Unix sockets, we changed the mode now to
2:58:06not allow EngineX to do anything with
2:58:09it. So that's why it was blocked and uh
2:58:12if it is blocked then it's going to
2:58:14display this bad gateway error to the
2:58:16user. Now how do we solve this? So we
2:58:19can add engine X to my group that will
2:58:22solve the issue. If you want you can set
2:58:24it to 777 but that's uh yeah that's not
2:58:28secure. Any user can do whatever they
2:58:31want with the file. Let's add uh ww data
2:58:34to my uh group. And let's let's also uh
2:58:38let's look at the error logs. Uh I want
2:58:41to see what we have there. Uh as you can
2:58:44see we also get uh something in the
2:58:46error logs. Great. So this confirms as
2:58:49as you can see it says critical
2:58:51permission denied. So it it tried to
2:58:53connect to the socket but it failed
2:58:55because it doesn't have the full
2:58:57permissions. Awesome. So this make uh
2:58:59this confirms that changing the mod will
2:59:02either allow engine X to be able to
2:59:06connect to the socket or not. So let's
2:59:09fix this. We have different ways but I
2:59:11will do it in this way. So I will just
2:59:14use this command sudo user mod- a g and
2:59:20here I have this you put here put your
2:59:23user my user is emad I will put it here
2:59:25and I will add ww- data to my user now
2:59:30we have another command called get end
2:59:33group and here put your user and as you
2:59:36can see you should be able to see ww-
2:59:39data awesome so this confirms that it
2:59:41works Now let's run uh let's do that ls-
2:59:46array and ww data is part of my group so
2:59:50it is a uh let's see so we have this
2:59:53tree so it's able to read and execute
2:59:57okay let's go back to the browser let's
2:59:59refresh uh it is still not working maybe
3:00:03let's try to reload engine x so here hit
3:00:06ctrr and start typing reload it's going
3:00:08to find it in the history
3:00:12Let's reload. Okay, we still we are not
3:00:15able to connect to the socket. I want to
3:00:17go back to uh ls-la
3:00:21and let's do it on J unicorn. And here
3:00:23we have a problem because before I ran
3:00:27ch mode and that changed the yeah the
3:00:30user to 750 and that's not correct. So
3:00:33we need to fix this. So let's do uh ch
3:00:38own. I want to own this file, not 750.
3:00:41750 is not correct. So that was a
3:00:43mistake that I did because instead of
3:00:45typing chod, I typed ch own. And here,
3:00:49let's do it on genicorn.
3:00:52Let's run ls-la. Okay, now that's fixed.
3:00:55Okay, let's go back again to the
3:00:57website. Okay, it's still doesn't work.
3:01:00And maybe because we also need write
3:01:03permissions. So that's how you debug
3:01:04this. Um, let's do it. So let's go back
3:01:08to here and let's change this to 770.
3:01:13Let's look at ls-la. Let's go back to
3:01:15the web application. Okay, now it
3:01:18worked. We need also write permission
3:01:21because we need to write to the Unix
3:01:22socket file. That makes sense. We write
3:01:25data to this file so that G unicorn can
3:01:27read it. And again, G Unicorn will write
3:01:30data to Unix file so that engineext can
3:01:32read it. Okay, great. So that works. And
3:01:35as you can see, the other users will not
3:01:37be able to reach this file. If you
3:01:40followed uh the same steps as I did, you
3:01:43should be uh able to get this to work.
3:01:45Awesome. So, we have finished with the
3:01:49first part of this video. Now, we will
3:01:52talk about security headers. We saw that
3:01:56we were all already secure because we
3:01:58have the firewall. it was blocking port
3:02:0280, but now we allowed port 80 so that
3:02:05we can access the website. But security
3:02:07doesn't stop at the firewall. Browsers
3:02:10have built-in security features, but
3:02:13they only activate them if your server
3:02:15tells them to. And if you want to read
3:02:17more about this, you can go to HTTP
3:02:20security headers, and you will find this
3:02:23documented here on the developer Mozilla
3:02:27website. And as you can see, they have a
3:02:29lot of headers. They will explain them
3:02:31to you in detail, but I will show you
3:02:34because we have a lot of uh headers.
3:02:36I'll just show you the main ones that we
3:02:38need to have. And there are also
3:02:40websites like security headers.com and
3:02:45the http observatory from Mozilla that
3:02:49will test the security headers to see if
3:02:52they are uh on your website or not. If
3:02:55they aren't, you will get a low score.
3:02:58But if they they find them, then they
3:03:00will tell you that your website is
3:03:01secure. Same thing with security
3:03:04headers. You just need to paste here
3:03:06your domain name or uh in our case, we
3:03:09don't have a domain name. So we could
3:03:10try to put uh just the IP address. Uh
3:03:15let's try this. Let's try it here. I'll
3:03:18try it on HTTP Observatory. Ah, so it
3:03:20says that it cannot scan IP addresses.
3:03:23Okay. So that means that we need to do
3:03:25it with a domain name. Okay. But here
3:03:29let me actually take the full the full
3:03:31link. Yeah, it doesn't work. So with
3:03:33HTTP Observer 3, we need the full
3:03:36domain. But I can uh use my own website.
3:03:39Let's try to scan it. And as you can
3:03:42see, I get A+, which is great. It's even
3:03:45higher than the 100. So uh the security
3:03:48here is great. But let's go back to
3:03:51security headers. This is the websites
3:03:53that you are that we are working with.
3:03:55We have created the new configuration,
3:03:57but you can see that it lacks a lot of
3:03:58headers here. The great thing is that
3:04:00they show you the main ones. They have
3:04:02the list of missing headers. They tell
3:04:04you why you need them. And here F. So F
3:04:08is the lowest score that you can get,
3:04:11but you can climb uh from F to A+. And
3:04:16here what I like about HTTP observatory
3:04:18is that if you click on scan history
3:04:21they show you the progression. So in the
3:04:23beginning I got I got the because I
3:04:26wasn't aware of these headers or I
3:04:28didn't add them in the beginning but
3:04:30after that I started to add them
3:04:32gradually to make uh my website secure
3:04:36and now I have the highest score which
3:04:38uh yeah which which gives me the A+
3:04:40grade. We will do the same thing for
3:04:43this website that we created now. Yeah,
3:04:45let's start working with these simple
3:04:46ones because the the complex one is CSP
3:04:50content security policy. That one we
3:04:52will leave it at the end because it
3:04:53requires a lot of iterations to get it
3:04:55right. Now let's go back again. Hit
3:04:58CtrlR. Let's do pseudo nano. Okay, we
3:05:01find this is the configuration. We will
3:05:04start with strict transport security.
3:05:07This one forces the browser to always
3:05:10use HTTPS when talking to your website.
3:05:14And here it will uh force that for the
3:05:17next two years. So this value uh is 2
3:05:21years. You might be asking we don't have
3:05:23HTTPS yet. We are just on HTTP. That's
3:05:26not that's not a problem because this
3:05:28header is ignored by browsers if they
3:05:31see that you are uh accessing the
3:05:34website over regular HTTP. That's fine.
3:05:37But this will be active or it's going to
3:05:39be used by browsers when we switch to
3:05:42HTTPS. So keep it like that. And here we
3:05:45want to also include subdomains. Uh we
3:05:48don't have that but if you plan to add
3:05:50subdomains then uh we also want to
3:05:53include them. Okay. So this is the first
3:05:55one. Now we can yeah let's let's save
3:05:58the file. Uh hit Ctrl X. Remember we
3:06:01need to do sudo engineext-t.
3:06:04Okay. It says successful. Now we can
3:06:07reload. Reload EngineX. Now let's go
3:06:10back to the browser. Let's scan the
3:06:15website again. Okay. So yeah, it didn't
3:06:18change because that header wasn't in
3:06:20this list. That's fine because now we
3:06:23will add X frame options, X content type
3:06:27options, and another one called cross
3:06:30origin cross origin resource policy.
3:06:33I'll copy all of them. I'll go back to
3:06:36nano. Let's go back to nano and I will
3:06:39add them here. So here I have some
3:06:41comments if you want them. Uh I will
3:06:43just remove them from here but they are
3:06:44in the markdown file. Let's start with
3:06:47corp. This one allows sharing. So if you
3:06:50want to allow other browsers to share
3:06:52your resources then you can leave it
3:06:55this way. Otherwise you uh you you might
3:06:59need to block it. And to do that you
3:07:01just need to copy this header. Go back
3:07:04to the browser, search for it. You will
3:07:07find the Mozilla uh the documentation
3:07:09there. And here as you can see you have
3:07:12uh you have an explanation and you also
3:07:14have the directives that you can that
3:07:16you can set. So you have same site, same
3:07:18origin or cross origin. In our case, we
3:07:21are using cross origin because we want
3:07:23to allow other websites to maybe link
3:07:25our articles, images or something like
3:07:27that. But if you want to block them, you
3:07:29put same sites which means that you you
3:07:32allow only yourself to yeah reuse uh
3:07:35some of your resources. Okay, let's see
3:07:38what we have in nano. And as you can
3:07:40see, we allow other origins and origin
3:07:42means other uh domains. Uh I will remove
3:07:45the comment because I don't need it. We
3:07:47have here anti- clickjing. Uh we use
3:07:51Xframe options for that. If you want to
3:07:53read more about clickjing, you can find
3:07:56this Wikipedia page which explains it.
3:07:59Basically, this is uh how the attack
3:08:02works. So, the user is presented with a
3:08:05false interface where their input is
3:08:07applied to something they cannot see.
3:08:10You might visit a website, they will
3:08:12show you this page where you see they
3:08:15are telling you claim your prize. But
3:08:17this page is layered on top of another
3:08:20page. So when you click on claim your
3:08:23prize, you are not clicking on this
3:08:25button. You are clicking on another one
3:08:27which is invisible to you. And here they
3:08:30say confirm bank payment. So they can do
3:08:33uh shady things. They can take your data
3:08:36or something like that. So that's a
3:08:38problem and we don't want to do that. We
3:08:39we want to to prevent this. To do that
3:08:42we add xframe options and we tell same
3:08:46origin. As you can see we are not using
3:08:48cross origin uh or other directive. Uh
3:08:51let's look at the other directories
3:08:53quickly. Let's search for it here. Um
3:08:55Mozilla.
3:08:57Okay. So here this is what we have. So
3:08:59we have either deny sim origin or allow
3:09:02from origin. Awesome. So we have that.
3:09:04We also have MIM sniffing. This one
3:09:08forces the browser to strictly trust the
3:09:10file type declared by the server.
3:09:12Sometimes attackers might upload
3:09:16malicious files maybe like a script but
3:09:19they might tell the browser this is not
3:09:21a script this is an image and this is
3:09:23called MIM sniffing. MIM means uh the
3:09:27the format types. So let's search for
3:09:29MIM. If you are not familiar with MIM
3:09:32let's search for MIM type and yeah it's
3:09:35uh media types. So this uh uh yeah it
3:09:39can mean pawns, images, text, scripts
3:09:43etc. So attackers will trick your
3:09:46browser to think that that script is an
3:09:48image and in that script they might load
3:09:51some uh malicious code or something like
3:09:53that. So we want to deny this. We don't
3:09:55want them to sniff the file types. We
3:09:58added three extra headers. Now save the
3:10:01configuration. Let's do engineext-t.
3:10:04Everything is good. Let's reload. Nice.
3:10:07So, let's go to the browser. Let's close
3:10:10these tabs. We don't need them. Let's
3:10:11scan. Great. Now, we have D. So, you can
3:10:14see that we have two green check marks,
3:10:15but we need to add another ones. Uh,
3:10:18these two are simple. So, let's add
3:10:19them. And let's leave CSP at the end.
3:10:23Okay. And let's add the headers just
3:10:26below them. So, here we have this one is
3:10:29for privacy. When a user clicks a link
3:10:32on your website, uh for example, if I
3:10:35link a maybe a Wikipedia article on my
3:10:39website, if a user clicks it, the
3:10:42browser will send a referral header to
3:10:45the new site. So if you want to protect
3:10:47user privacy by only sending your your
3:10:50root domain name, not the full URL path,
3:10:54you can do this. So this uh then wiki
3:10:57here if they click on that Wikipedia
3:10:59link they will Wikipedia will only see
3:11:02my domain name they will not be able to
3:11:03see the full link. So yeah we can add
3:11:06this and here this one is simple. So if
3:11:09your website doesn't require using
3:11:11camera microphone dual location payment
3:11:15you can disable them because these are
3:11:16things that I don't want and that's
3:11:18great. So if someone tries to maybe hack
3:11:21your website and enable the user's
3:11:24camera that's that will not happen
3:11:26because we are blocking blocking that
3:11:29with the help of the permissions policy
3:11:31header. Now save exit the file verify
3:11:35that the configuration is correct. Now
3:11:37reload let's go to the website and let's
3:11:40refresh. Awesome. So it says C and in
3:11:44order to jump from C to A+ we need to
3:11:47work on CSP which is the final boss.
3:11:50This one is complex but I'll show you
3:11:52how to do it uh step by step. Okay,
3:11:55let's go back to our website. So this is
3:11:58the front end. Everything looks good. Uh
3:12:02let me open another tab. I want you to
3:12:04search for this. So we find it here and
3:12:07we also uh you can also see it here. So
3:12:10yeah, uh if you want to read more about
3:12:12cross sites, that's not what I wanted to
3:12:14search for. I wanted to search for
3:12:16content security policy CSP. Uh yes,
3:12:21here it is. You can see that this
3:12:23article is very long because in order in
3:12:26order to build it, there isn't a one
3:12:30there isn't one CSP that everyone can
3:12:32uses. This depends on what you are using
3:12:35in your website. That's why you need to
3:12:37build it progressively.
3:12:39open your uh website. So I will delete
3:12:43uh the tab that we don't need. Okay, we
3:12:46just want these two. Now we are in
3:12:48Chrome because here we have the ability
3:12:52to change this CSP very quickly. I'll
3:12:56show you how to do this. Now I want you
3:12:59to look at the very top of the network
3:13:02request list. Here we don't have a
3:13:04domain. That's why we have just uh the
3:13:07IP address. Right click, show all
3:13:10overrides. Uh here I will start fresh.
3:13:13Now select let's click on select folder
3:13:15for override. So I will create one here
3:13:20and select it and let's yeah allow this.
3:13:24Now go back to network. Here I want to
3:13:27click on override headers. Yes, that
3:13:30worked. And here let's add content
3:13:34security
3:13:36policy because this is the one that we
3:13:38want to work with. And here in the
3:13:40header value let's start with default
3:13:44source and let's do none. And let's not
3:13:46forget the semicolon. Now hit enter. Now
3:13:50I want you to click on the dot headers
3:13:53pile here. It should say star because we
3:13:57want to apply it to everything. And
3:13:59let's refresh the page. Okay, you can
3:14:02see that we cannot view the website. H
3:14:05and we have a lot of errors. So we can
3:14:07see that here it says refused to load
3:14:09the image, refused to load the
3:14:12stylesheets, refused to load the script.
3:14:14So now that's what we need to do the and
3:14:17also make sure if you want you can open
3:14:19this uh headers file in a text editor so
3:14:22that you can change it here easily. I'm
3:14:25opening it in sublime and yeah I see the
3:14:28same thing. So I'm applying it to
3:14:29everything and I am just playing with
3:14:32this header and default source none will
3:14:35block everything. So this is the
3:14:37strictest CSP and as you can see it
3:14:39doesn't work but we start from here and
3:14:41we start relaxing it in steps. So you
3:14:44can see that there are many things that
3:14:45are blocked. There are images that are
3:14:47blocked some styles. You can see that
3:14:49this image comes from our origin. So
3:14:52this one is fine. Uh let's see we here
3:14:55we are blocking this side sheets because
3:14:57we are it comes from cloud flare so this
3:15:00is not our website it's coming from
3:15:02another place but we know that we need
3:15:04this font it's awesome so we can keep it
3:15:06that's not a problem but this will show
3:15:08you the things you will decide do you
3:15:11want to keep it or block it uh same
3:15:14thing for scripts these are all coming
3:15:16from our assets folder so yeah these are
3:15:18these things can can be kept but uh if
3:15:21you if we see that there are other
3:15:23strange change things we need to decide
3:15:25if we should keep them or not. So let's
3:15:27start with the uh with the images. So I
3:15:30will come here. You can see that here
3:15:32default source I will change it from
3:15:34none to self. I saved the file. Now I'll
3:15:38go to the browser. Let's refresh. So
3:15:40that's solved a lot of problems. Why?
3:15:43Because you see that we don't have
3:15:46assets. The image was loaded because
3:15:48those things were in our disc folder. So
3:15:52when we built the website we already had
3:15:54those images uh those stylesheets
3:15:56everything was there. So when you use
3:15:59self it means our origin everything in
3:16:02our origin uh should be allowed to load.
3:16:07Now we aren't we have uh different
3:16:10errors. So for forget about these ones
3:16:13uh because as we saw before we don't
3:16:15have media search that's why they are
3:16:16failing. But focus on these. Now we
3:16:19cannot load this font. So this is the
3:16:21fonts that we are using in our website.
3:16:23Yeah, let's fix this one. Go back here
3:16:26and we have another directive. We need
3:16:29to come here. Let's add a space and
3:16:32let's put style source. This one will
3:16:34allow us to load styles. Again, we can
3:16:37start with self. And if we want, we can
3:16:40add more things here. But let's save
3:16:42this. Let's go back to the website. Uh
3:16:44or instead, let's let me put it here.
3:16:46And let me load Chrome like this. Yeah,
3:16:49that's better. And I will move this docs
3:16:53this panel uh to the bottom. Now I've
3:16:56saved this. I will refresh the page. Uh
3:16:59we still have some problems with uh
3:17:02Google. So we will copy this link and
3:17:05paste it here. We we we need just the
3:17:08the origin. We don't need the full URL.
3:17:10So we can see that the origin is
3:17:12fonts.googleapis.com.
3:17:14That's what we need. Now save. Refresh
3:17:17the page. And as you can see now we are
3:17:20able to load the font. We don't have any
3:17:22any error. Now we have another
3:17:24stylesheet issue but this time we will
3:17:27just take this domain. Take it copy and
3:17:30let's add it after Google. Sorry let's
3:17:34add it here before the semicolon. Save.
3:17:37Refresh the page. Awesome. So now we are
3:17:40we are able to load font awesome icons.
3:17:43We are able to load the font. And as you
3:17:46can see, we don't have those errors, but
3:17:47we have different errors. So here we are
3:17:50not able to load the font URL because it
3:17:53violates the following default source
3:17:56self. Okay. So you can see that here we
3:17:58don't have uh let me decrease the font
3:18:00size. We have default source. We have
3:18:02style source but we haven't yet added
3:18:05font source. So let's do that. I will
3:18:08come after the semicolon. I will add
3:18:10this and I will add another semicolon.
3:18:13Save this. refresh the page. Okay. So,
3:18:16let's open this and let's see what's
3:18:18happening. So, again, we have the same
3:18:21do origin. Let's copy it. Let's paste it
3:18:24after self. Save. Reload. Okay. So, we
3:18:27have a different domain that's coming
3:18:29from Jstatic. That one is fine. Again,
3:18:33if you see that there is a domain or an
3:18:35origin that is suspicious, then you feel
3:18:38free to block it. But if you think that
3:18:40this is something that you are using and
3:18:44it comes from a reputable website, it's
3:18:46not something suspicious, then add it to
3:18:48the list. So now we solved the problem
3:18:50with fonts. Now let's look at images.
3:18:54Okay. So here we have some X SVG files
3:18:58and in order to keep those we can just
3:19:00add data. I will come where is it? Do I
3:19:04have image source? No, we don't. And
3:19:06here they are telling us. So note that
3:19:08image source was not explicitly set. So
3:19:11again let's do the same thing. Uh I will
3:19:13copy font source. I'll come here and
3:19:16here I'll need to change it to image
3:19:18source. Save. Reload. Okay. That still
3:19:22doesn't work because we need to add this
3:19:25here. We don't have a website but uh we
3:19:28can add where is image. Uh let me make
3:19:31this smaller. Yeah, it's here. I can
3:19:33come after this and add data. Let's
3:19:35refresh the page. And voila. As you can
3:19:38see, we were able to fix uh all every
3:19:42error. And now these are just coming
3:19:44from the back end. But we are not done
3:19:45yet. What we what you need to do is to
3:19:48test your whole website. Make sure to
3:19:50click on every single page. See if the
3:19:53errors are uh if you don't see the goal
3:19:57is that we we should not see errors
3:19:59coming from here. Okay. So as you can
3:20:02see, we have other problems. Okay. So we
3:20:04have inline style issues. Let's go back
3:20:08to sublime text. I want you to find
3:20:12style source. Do we have Yes, we have
3:20:15style source. So we have self font.
3:20:18Google and here. So I will add a space
3:20:20at the end because here they say that we
3:20:23are using so yeah we are using inline
3:20:26styles but that this is blocked by
3:20:28default. So we need to add unsafe
3:20:30inline.
3:20:33Let's save it first and let's reload the
3:20:35page. Okay, we have zero errors. Great.
3:20:38But this is unsafe. Here's the thing.
3:20:40Even though this is unsafe, it's fine
3:20:43here in our case because in our Vue.js
3:20:46application or maybe you uh you might be
3:20:49using React or other frameworks. You you
3:20:52are using inline styles or inline
3:20:54scripts and those are fine because if
3:20:56you don't do those, your application
3:20:58won't work. But yeah, by default they
3:21:00are blocked but we can add them. Let's
3:21:02continue testing the application. Let's
3:21:04go here. Okay, that's fine. So, the
3:21:07other pages are not working because they
3:21:09need to communicate with me search. But
3:21:11at least uh every other page is working
3:21:14uh correctly. Good. Now, after doing
3:21:17this, we can let's let's copy the value.
3:21:21I'll hit Ctrl X. I no longer need this.
3:21:24So, I'll clear this. Uh I'll just keep
3:21:26this. Now, I'll go back to engine X. I
3:21:30need to update the configuration. So
3:21:33let's do that. I will come down and here
3:21:38I'll type add header. It's going to be
3:21:41content security policy. I'll add quotes
3:21:47and I'll add the other ones. As you can
3:21:49see, I have added the CSP. I'll save the
3:21:53file. I'll exit. Let's verify that the
3:21:56syntax is okay. And it isn't. So this is
3:21:59why we need to do this. Let's go back. I
3:22:02forgot to add something at the end.
3:22:04Yeah, I forgot to add always. And why do
3:22:08we do this? Because we want to send this
3:22:09header in all the response codes. So if
3:22:13we get 200, 2010,
3:22:16all of them. So uh this header will be
3:22:19there. And we can also add something
3:22:21here at the end. I will add another
3:22:24semicolon. I will add upgrade insecure
3:22:27requests. This one again forces the
3:22:29browser to change from HTTP to HTTPS.
3:22:33Now we have HTTP. So this doesn't matter
3:22:36but later uh after adding SSL
3:22:39certificates stuff like that we will be
3:22:42able to have HTTPS. Now we are just
3:22:45preparing save the file exit. Verify
3:22:49okay the syntax is okay this time and
3:22:52let's reload X. Okay so this is the
3:22:54moment of truth. I will take this. Let's
3:22:58put it here. We need to make sure that
3:22:59the the website loads correctly. And it
3:23:02is it is loading. Let me move this to
3:23:05the left. Uh I will clear the logs. I
3:23:08also go to the other pages. Yeah,
3:23:09everything seems to be working
3:23:11perfectly. Okay. So now let's scan
3:23:14again. And yes, we got uh a um maybe uh
3:23:19on Firefox might have gotten a plus, but
3:23:22this is the highest score that we can
3:23:23get. And as you can see, everything is
3:23:24green. And the yeah these are the this
3:23:28is the response that uh we received. So
3:23:30as you can see this is what we have
3:23:31added here is upgrading secure requests
3:23:34and yeah they're also telling you if you
3:23:36want to add other requests but this is
3:23:37the bare minimum you need to have these
3:23:40and as you can see most of them are easy
3:23:42but content security policy is the one
3:23:44that takes time because it is unique to
3:23:47each website. Your application is now
3:23:50live. Engine X is serving your view
3:23:52front end and routing API calls to your
3:23:55fast API back end. However, we haven't
3:23:59yet configured search. This is a search
3:24:02engine that I am using to s to allow
3:24:05users to search for articles. It's
3:24:08similar to elastic search uh yeah to
3:24:10elastic search but it doesn't consume a
3:24:12lot of memory. So we are going to
3:24:14configure that in the next chapter. But
3:24:16if you don't care about mail search, you
3:24:19can skip to the to the other chapter,
3:24:22chapter number four where we are going
3:24:24to create our domain name, add SSL
3:24:27certificates and make our websites
3:24:29secure by accessing by making um making
3:24:33users access accesses through HTTPS
3:24:36instead of HTTP. See you in the next
3:24:38chapter. Welcome to module 3. At this
3:24:42point, we have a secure server and we
3:24:45have deployed the front end and back end
3:24:49successfully. There is one missing piece
3:24:52if you are using my project which is to
3:24:55include a search engine. In my case, I
3:24:58am self-hosting M search because it is
3:25:02lightweight, doesn't consume a lot of
3:25:04memory and it is fast. A quick note on
3:25:07this module. If your specific project
3:25:10doesn't need a search feature, you can
3:25:13technically skip this chapter. However,
3:25:16I highly recommend following along
3:25:18anyway because the skills you are about
3:25:21to learn like creating a system user and
3:25:24deploying background services that are
3:25:27completely hidden from the internet are
3:25:30essential server skills you will use
3:25:32anytime you need to host your own
3:25:35database or yeah something else. Here is
3:25:38our road map. First the search engine
3:25:42that we are going to download is me
3:25:44search. There are other options like
3:25:46elastic search but those consume a lot
3:25:49of memory and here we are playing with a
3:25:51small virtual machine and we can't just
3:25:54host elastic search. So we are using me
3:25:57search because it's lightweight it's
3:26:00fast and most importantly it doesn't
3:26:02consume a lot of memory. Next we are
3:26:05going to log this database by creating a
3:26:09dedicated system user. I'll explain the
3:26:13difference between a regular user and a
3:26:16system user in the upcoming slides. Then
3:26:19you will configure systemd to run me
3:26:23search as a background service. This
3:26:26will allow us to perform daily backups.
3:26:30Make sure that me search is running even
3:26:32if it if the service fails and other
3:26:35things. As I said, we are going to
3:26:37schedule automatic snapshots. I'll show
3:26:39you how to do it. You can specify the
3:26:41duration. Do you want to do it daily,
3:26:43weekly? It depends on how much your data
3:26:46changes. Finally, we are going to use
3:26:49SSH tunneling again to access the daily
3:26:53search database uh using a dashboard.
3:26:57Now let's go back to GitHub. I want to
3:27:00show you what we are trying to do. We
3:27:03are now in module 3 data and search. You
3:27:07will you'll find the commands here. So
3:27:10even if you get lost or if you find long
3:27:12commands, don't worry about them. Uh you
3:27:14will find everything inside this
3:27:16markdown document. As I said, if you
3:27:19don't have a project and you rely on my
3:27:22own inside the backend folder, I have
3:27:25everything for you. So even if you don't
3:27:27have data that you want to put in me
3:27:30search, you can just come here in the
3:27:32seed folder and I have created this
3:27:36documents list. As you can see, we have
3:27:39some documents that we can index uh in
3:27:42mill to me search. Okay, so don't worry
3:27:44about this. Everything is inside these
3:27:47two repositories. Great. Now, open your
3:27:50terminal and let's SSH to the machine. I
3:27:54will use
3:27:56my alias. Uh the alias that I'm using is
3:27:59my website. I have successfully logged
3:28:02into the machine. Uh let's run ls. Yeah.
3:28:06Uh let's go to web app. Okay, so we have
3:28:08everything. Great. Let's start by
3:28:11installing me search. If you click on
3:28:15the curl option, I mean you have a lot
3:28:17of options, but we'll go with this one.
3:28:19Copy this command. Paste it here inside
3:28:23the machine. It shouldn't take a lot of
3:28:26time. Yeah. So it worked. Let's see. So
3:28:28we have me search. Uh let's see if it is
3:28:32executable.
3:28:33Yes. If we are the current yeah for my
3:28:36current user it is executable. Now we
3:28:38need to move this binary into user local
3:28:43/bin. Let's do this. So we have mainly
3:28:46search here. Let's move it to user local
3:28:51bin. I need to provide my password.
3:28:54Awesome. Why we do this? Because it
3:28:56allows us to run search from any place.
3:28:59So I can now just type med search d-
3:29:02version and it works. I don't need to go
3:29:05to uh say say I downloaded the binary in
3:29:09the downloads folder. If I don't move it
3:29:12to user local bin, I have to always go
3:29:14to downloads in order to interact with
3:29:16the binary. But now when you put it in
3:29:20uh user local bin, it allows you to run
3:29:23it from any place like we did. Now if I
3:29:26type ls as you can see we don't have the
3:29:28mely search binary but I was able to run
3:29:32the me search uh version command without
3:29:35any error. So now we have me search.
3:29:38Let's create a system user. Before we do
3:29:42that I would like to come to go back to
3:29:43the slides to explain the difference
3:29:45between a regular and a system user. I
3:29:49have created this table to explain to
3:29:52you the difference between these two
3:29:54types of users. Here are the features
3:29:56and here are the types of users. So what
3:29:58is the purpose of a standard user when
3:30:01we started this course? We created a
3:30:04user. We are not using the root user but
3:30:07we created another user that we can use
3:30:09to do maintenance, install software.
3:30:12Yeah, stuff like that. We are using a
3:30:14standard user to administer to do
3:30:17maintenance. So this is the purpose of
3:30:19this user. But a system user is used to
3:30:24run background services only. So if I
3:30:27want to run M search uh in a background
3:30:30service, I need to run to use a system
3:30:32user for that. Can we use the shell in a
3:30:36standard user? Yes. As you saw, I was
3:30:38able to SSH to the machine and uh type
3:30:41commands. But if I try to do the same
3:30:43with a system user, I won't be able to
3:30:45because we deactivate that. This means I
3:30:48can't login as a system user and start
3:30:51typing commands. So this is the same
3:30:53thing that I that I mentioned here. So
3:30:55login is possible with a standard user
3:30:57but it isn't allowed with a system user
3:31:00and generally the standard users have
3:31:03their uh home folder inside/home
3:31:07but system users are yeah put somewhere
3:31:10else. So this is the difference between
3:31:12them and because M search is a software
3:31:15that we want to run as a background
3:31:18service we want this specific system
3:31:21user to be responsible of handling that.
3:31:25Okay. So let's create uh a system user
3:31:28and we can call it med search because it
3:31:30uh its job is to uh uh run this software
3:31:35for us. And in order to do that, we will
3:31:37use the user add command. Here I will uh
3:31:42specify the directory where we want to
3:31:45put this user. As I said, we are not
3:31:48going to put it in home. We will put it
3:31:50here. var lib mail search. We will
3:31:53disable login. So we won't give it shell
3:31:56access. And finally, we will call it
3:32:00search. Mele search. I hope I didn't
3:32:02make a mistake. Me search. Me search.
3:32:05Yes. So now let's hit enter and it
3:32:09worked. Now let's try to look at this
3:32:13directory. As you can see the I don't
3:32:15have the permissions. Why? Because I am
3:32:17using my standard user aad and this
3:32:21folder or yeah this directory is owned
3:32:25by me search the system user we created
3:32:28and we can verify this. And if we do
3:32:30sudo as you can see this user mini
3:32:33search owns this this folder. So we
3:32:37cannot go inside it. But yeah with with
3:32:40root we can do that and we can also
3:32:42verify it here. As you can see the
3:32:45permissions are they are they are
3:32:47constrained. They are they are allowing
3:32:49just this user to write to this
3:32:52directory. Great. So this confirms that
3:32:54everything is working. Now we need to
3:32:56create a folder inside
3:33:00uh that will hold data. We need to
3:33:03create the data folder because me search
3:33:06needs somewhere to save data. So let's
3:33:08do it. Uh let's verify the ownership.
3:33:12This is very important. ls-la
3:33:15as you can see. Uh let me go back. Okay.
3:33:18So data is owned by root and we want it
3:33:22to be owned by med search. So let's do
3:33:24it. We we have used this each own
3:33:28command before. Here we are uh changing
3:33:31the user and group from root to M search
3:33:35and we want this user to own everything
3:33:37inside this directory. That's why we are
3:33:40using dash r. So now if I rerun this
3:33:43command as you can see now the data
3:33:46folder is owned by mini search. Awesome.
3:33:49So now we installed the mini search
3:33:51binary. We created a system user. Now we
3:33:55need to put some data inside. So if you
3:33:59already have your own instance, you you
3:34:02don't need to use let let me go back to
3:34:05GitHub. You don't need to use this JSON
3:34:08file. You can just uh export the data
3:34:12from your local main search instance and
3:34:14use that. If you if you don't have data,
3:34:18then you can use this. I'll show both
3:34:20options. so that uh everyone can uh
3:34:24follow along. Let's start by exporting
3:34:27data from a local me search instance.
3:34:30For that I will create another terminal
3:34:32window. Now I am uh on my local machine.
3:34:36I will start the mearch service and I
3:34:39will show you how to export data from
3:34:42it. I will use T-Max to create two panes
3:34:46here. Okay, I moved where I installed me
3:34:50search and this is how I am using it.
3:34:52Um, okay. So, as you can see it started.
3:34:55Let me show everything and here. Yeah,
3:34:57it is listening on this port. Maybe
3:35:00search by default uses port 7,700.
3:35:03Uh, in this other terminal we can talk
3:35:07to this server. We can use the terminal
3:35:10or if you want you can also use their
3:35:13SDKs. They have one for Python,
3:35:15JavaScript and other programming
3:35:17languages. But for now, to keep things
3:35:19simple, we will just use the yeah the
3:35:23terminal and we'll just run a curl
3:35:25command. And as you can see here, we
3:35:27have something that we call dumps. And
3:35:30here I need to provide my master key.
3:35:32And since we are on a local machine,
3:35:35yeah, I am just using their default
3:35:38master key. Plus in a production
3:35:41environment we will change this into a
3:35:43secure key. But for now it's fine. Uh
3:35:46when what you are working with this
3:35:48locally no one will steal data from you.
3:35:50So it's fine. Keep it like this as the
3:35:52default. Yeah. So now I can't talk to
3:35:55this. But before running the command I
3:35:57just want to explain the difference
3:35:58between dumps and snapshots. If you
3:36:02search for me search dumps snapshots,
3:36:05you will find this page where they
3:36:07explain the difference between these two
3:36:10concepts. A snapshot is an exact copy of
3:36:14your database. You use a snapshot to
3:36:17create backups. So this is what we are
3:36:19going to schedule later. We are going to
3:36:22uh allow me search to save a backup of
3:36:24the database daily. So that's let's say
3:36:27some if uh for some reason something bad
3:36:30happened to me search and you lost data
3:36:34you can recover by using these
3:36:36snapshots. A dump on the other hand is
3:36:40just a set of instructions that are used
3:36:43to recreate the database from scratch.
3:36:45So this is the difference between the
3:36:47two and dumps are the safest way to
3:36:50migrate data across different machines
3:36:53or versions. This is why in uh my local
3:36:57terminal I am using the dumps. I'm going
3:37:00to create a dump with this curl command.
3:37:03So now let's run it. As you can see we
3:37:05get a task uid. Uh in m search tasks are
3:37:09asynchronous which means that they take
3:37:12time to finish but search at least gives
3:37:15us a task u so that we can track the
3:37:18task and the task that we are performing
3:37:21is creating a dump. And if you want to
3:37:24follow a task, what you need to do is to
3:37:27change here dumps to tasks and you
3:37:32provide the U ID and in in our case it's
3:37:354,94
3:37:38and you yeah keep keep the authorization
3:37:41key and hit enter. And here I forgot to
3:37:44change post to get. So let's try to do
3:37:47that. And as you can see the status is
3:37:50succeeded which means that we were able
3:37:52to create a dump. That's great. And we
3:37:55can also if you want you can also verify
3:37:57this in the mini search server. Uh so
3:38:01here we started with 20 workers. And
3:38:04here as you can see we the server
3:38:06received the the task and here it says a
3:38:09batch of task was successfully completed
3:38:12with one successful task which means
3:38:14that everything worked great. Now we
3:38:16just need to find this dump. As you can
3:38:19see, I have my mini search binary inside
3:38:22this folder. If I hit if I try to run
3:38:25ls, you can see that we have a dumps
3:38:28folder. Let's go there and let's see
3:38:31what we have. Let me also remove the
3:38:33first pane. We don't need it. As you can
3:38:34see, I have created a lot of dumps.
3:38:37Let's look at this one. So, we uh as you
3:38:40can see, they label them by date. So we
3:38:42are in 2026 which means this is the one
3:38:44that we created. Today is March 20 which
3:38:47is correct. So this is the one that we
3:38:49created. Great. Now we need to take this
3:38:52dump and move it to the virtual machine.
3:38:56As I mentioned you can if you don't want
3:38:58to use curl you can use one of their
3:39:00SDKs. If if you go to this page I just
3:39:04searched for create dump and I got I got
3:39:07here. If you click on curl, you can get
3:39:10as you can see they have a lot of
3:39:11programming languages. Let's select
3:39:14Python. And as you can see when you
3:39:16create the client, you can just call the
3:39:18create them. Uh the response will be a
3:39:20task u uh same same as what we got in
3:39:25the terminal just by using the curl
3:39:26command and after that you can just wait
3:39:28for it to finish. In the beginning it's
3:39:31going to be incued but after uh a
3:39:33successful execution it's going to
3:39:35return succeeded. Uh so they talk about
3:39:37authorization. So yeah, everything you
3:39:39need to know about uh how this works.
3:39:42Now let's go back to the terminal. We
3:39:44need to move this to the machine. Uh so
3:39:48let's use the SCP command. Let's paste
3:39:51the name of that dump. I will use my
3:39:54alias, my website, and I will put this
3:39:59uh temporarily in the temp directory.
3:40:02Okay. Okay. Great. So that worked. Uh
3:40:05let me copy the name again. I'll go back
3:40:08to the virtual machine and let's find
3:40:11that if if the dump is uh inside the
3:40:13temp deck directory or not. So let's use
3:40:16ls temp and let's paste the name of that
3:40:20file. And as you can see we have it
3:40:23great. Now we need to move this dump. So
3:40:26let's uh go up. I will remove ls and
3:40:30replace it with mv to move this file
3:40:34into var lib mail search. As you can
3:40:38see, I cannot do this because I don't
3:40:42own that directory. So, let's use sudo.
3:40:45Okay, it's it's there. And again, don't
3:40:48forget to use ch own. Uh let's search
3:40:51for it. Hit ctrlr.
3:40:54Start typing ch. We found it. Now hit
3:40:58enter. Let's verify that the permissions
3:41:01are correct. If we copy this, let's use
3:41:06ludo ls-la
3:41:08and paste that directory. Let's see.
3:41:11Yes. So this dump is owned by mail
3:41:14search. We could have created a dumps
3:41:16folder, but for now this is temporary
3:41:19because we just want to transfer the
3:41:20data once and remove this file. We don't
3:41:23we don't need it later. That's why we we
3:41:25didn't create the dumps folder inside
3:41:28the virtual machine. Now let me show you
3:41:31how to run me search as a service
3:41:33because right now if uh it's not it's
3:41:37not running we need to manually do it
3:41:40and if a problem happened that cause
3:41:43many search to stop we need to SSH back
3:41:46to the machine and rerun the command
3:41:48again and this is not good. We want the
3:41:52system to do this for us like when we
3:41:55used supervisor J unicorn to manage the
3:41:59back end uh the fast API back end for us
3:42:02without doing any work. So this is what
3:42:05we are going to do now. We will use the
3:42:08mini search system user to run search as
3:42:12a service. Here we are going to use
3:42:15systemd to create the service. We are
3:42:17not we are not going to use supervisor
3:42:20like we did with J unicorn because uh
3:42:25systemd sits at a lower level. Let me
3:42:29explain this. Uh I have a slide that
3:42:31explains this. So here we are talking
3:42:32about systemd versus supervisor. As you
3:42:36can see we have many layers and here I'm
3:42:38just showing the operating system level
3:42:41and the application level. So operating
3:42:44the operating system level is lower and
3:42:47here we have the services that are
3:42:49running and system D manages these
3:42:52services. We are going to create a
3:42:54service that we will call mini search
3:42:56and the goal here is that we want system
3:42:58D to start this service whenever we
3:43:02reboot whenever a mini search crashes.
3:43:05Uh so yeah systemd will be the manager
3:43:08and we use systemd in uh when we work
3:43:12with things that are low level at the
3:43:15application level for example with our
3:43:18backend API we use supervisor for that.
3:43:21So yeah this is the difference between
3:43:23the systemd and supervisor. Okay now
3:43:26let's go back to the terminal and let's
3:43:29create this service. If you want to
3:43:32learn more about systemd, you can just
3:43:34type systemd and they have here their
3:43:37website where you can learn more about
3:43:39it. It's really awesome and uh it's also
3:43:42easy to use. Let's go back to the
3:43:44terminal. Now we need to create a secure
3:43:47key. Uh if you remember here when
3:43:51locally I am just using this dummy
3:43:54master key that uh yeah the med search
3:43:57team uh shows you uh in their
3:44:00documentation but only do this if you
3:44:03are working locally because no one will
3:44:05steal your data because everything is
3:44:07local in production. We need to create a
3:44:11secure master key and that this is what
3:44:14we are going to do in uh at this stage.
3:44:17Okay, let's go back to the BM. Uh, in
3:44:20etc., we need to create millarch because
3:44:23we don't want to put the master key in
3:44:26the service file. We want to secure this
3:44:29so that we can just call a variable that
3:44:32will contain the master key. And we are
3:44:35going to do that here in this search
3:44:37directory. And inside we will put uh an
3:44:41env file where we are going to create uh
3:44:45two environment variables. I'll put them
3:44:48here but we will change the values. And
3:44:50by default me search will search for any
3:44:54environment variable that that uses this
3:44:56prefix melee.
3:44:59So here the environment is production.
3:45:01This is important. And now we need to
3:45:03change this placeholder to a secure key.
3:45:07You have many options to generate a a
3:45:10secure uh key. Uh one of the options is
3:45:14to use OpenSSL. So when you run it, it
3:45:17gives you this random number that you
3:45:19can use. So yeah, just do this. Copy it
3:45:22and put it there. I will do this and I
3:45:24will come back. Okay, I have created the
3:45:27key using the open SSL command. I have
3:45:33opened the env file and I have pasted
3:45:36that key there. I'm not I am not going
3:45:38to show you the key and you should also
3:45:41do the same. Keep your keys secret. No
3:45:44one should see them. Now we need to make
3:45:46sure that the me search user is owning
3:45:49again these files because we are going
3:45:51to use that user in the service. We know
3:45:54how to do this. We are going to use the
3:45:56chall command and yeah we want to own
3:45:59this file. We also we are also going to
3:46:02change the permissions. Uh I think by
3:46:05now you know what this what 600 means.
3:46:07It means that for the search user we
3:46:10wanted to be able to read and write from
3:46:12this file but we want to block access to
3:46:16any other user. Let's just do ls-la to
3:46:20confirm this ls-la.
3:46:23As you can see everything is prohibited.
3:46:25So we just uh the mail first user will
3:46:27be able to read and write. won't be able
3:46:29to execute. But uh and also for the the
3:46:33group and the other users, they are
3:46:35blocked. They they don't have the
3:46:37permission to even see read this file.
3:46:40This is very important because we want
3:46:42to keep things isolated. We don't want
3:46:45anyone to touch the NV file or to read
3:46:48the secrets from there. Now we are ready
3:46:50to create the service file. And as I
3:46:53said with systemd it's it's really easy
3:46:56and this is how we do this. We are going
3:46:58to create a service file. You can name
3:47:01it whatever you want but at the end you
3:47:03need to add dots service and you put it
3:47:05inside the systemd folder. I will paste
3:47:08this text and I will explain what it
3:47:10does. And again uh we have two options.
3:47:14The first one deals with users who have
3:47:18data and they extracted it from their
3:47:20local uh med search instance like what
3:47:23we are doing now. And the other option
3:47:25is for users who don't have uh data and
3:47:29they just want to use the the JSON file
3:47:32that I provided. I'll come back to that
3:47:35to that second option. But for now, this
3:47:37is what we have in a systemd service. We
3:47:40need to provide a description. So here
3:47:43um provide something that is that
3:47:46explains what this file is doing. So
3:47:49here we are going to run the mail search
3:47:52engine and here what does after mean?
3:47:56This line tells Linux do not attempt to
3:48:00start this service until the server's
3:48:03networking stack is fully active. This
3:48:05sometimes is important for things that
3:48:07need internet access. But if you if you
3:48:11are running something that doesn't
3:48:12require it, you can remove this. This is
3:48:14what it means. So if you want your
3:48:16service to start maybe after some other
3:48:18services you you do that uh here. Now we
3:48:22have the service. So the type here is
3:48:24set to simple and here is where we put
3:48:28the user. As you can see we are allowing
3:48:31only me search to run this service for
3:48:34us. This is why we were creating all
3:48:36these env files. Uh this is why we
3:48:39created this working directory and we
3:48:41made sure that me search is the own is
3:48:45the is the owner of uh these files and
3:48:48folders that we created. Uh let's say
3:48:50that someone hacks your server. They
3:48:52won't be able to look inside for for
3:48:55instance the env file. This allows us to
3:48:58isolate media search as much as as as
3:49:01possible. Okay. Uh so I have this
3:49:03comment here. uh we will load those
3:49:07environment variables. The working
3:49:09directory is here and here we have the
3:49:11execute start. This is the command that
3:49:13will be executed. As you can see we are
3:49:15pointing to the M search binary. We can
3:49:18uh so here I'm putting the the full path
3:49:21but you saw that we can just use M
3:49:23search and it's going to be recognized
3:49:25the database path. So here M search will
3:49:28store all its data in the data uh
3:49:31folder. Here we have dump directory is
3:49:34dumps like uh what I have here locally
3:49:38and these two lines are what we want or
3:49:42this is yeah this is uh what we what you
3:49:45will use if you want to import the data
3:49:48that you exported locally. So here the
3:49:51argument is import dump and here you
3:49:53need to specify the the path to that
3:49:56dump file and this ignore dump if
3:49:59database exists is very important. So if
3:50:02you have data in me search that will
3:50:05raise an exception because this might
3:50:07cause me search to override your data.
3:50:10So just for safety it's going to block
3:50:12it here. What you say is ignore this. So
3:50:14if you have data in me search please
3:50:16don't import this. We don't need it. And
3:50:20finally uh we have this is very
3:50:22important restart on failure. As I said
3:50:25if me search failed for some reason no
3:50:29problem. We don't need to go to the
3:50:31machine and restart it. System D will do
3:50:34it for us thanks to this line. As you
3:50:37can see the the service is easy. I just
3:50:40need to change this placeholder. I need
3:50:42to find the name of the dump and put it
3:50:45here. So here is the name. I'll copy it
3:50:48from here. Let's go back and let's paste
3:50:51it. Let's remove this placeholder. Let's
3:50:54paste this. Save. Exit. Let's run damon
3:50:59reload and enable. So after creating a
3:51:03service, we need to enable it. Uh so it
3:51:06created a system a sim link. That's
3:51:09good. And now we can start it. Okay. Now
3:51:13let's go back and let's do status cuz
3:51:15status will show us if something bad
3:51:17happened. And as you can see, yes,
3:51:19something bad happened. So here it says
3:51:22failed. This is not a problem. This is
3:51:24part of the process. Uh don't worry
3:51:26about it. Okay, let me show you how to
3:51:28debug this. We can use journal CLL the
3:51:32name we put the name of the service and
3:51:34here 10 minutes ago. So yeah, it should
3:51:36be fine. Uh let me remove this. Okay, so
3:51:40it tells me that I am not able to see uh
3:51:44any message. Let's add sudo. Okay, now
3:51:47that works. So this is what we have
3:51:49here. As you can see it tell let's see
3:51:52it says failed to execute
3:51:54user/loc/bin/aily
3:51:57search. Okay so that means it wasn't
3:52:00able to run the binary. Okay let's see
3:52:04why. Let's type- la user local bin and
3:52:10me search. Ah okay. So here the problem
3:52:13is that uh we don't have the uh execute
3:52:17permission. So execute is given just to
3:52:19my user but maybe search is is going to
3:52:23be in the others section. It's not going
3:52:26to be in my group but it's going to be
3:52:28here. And as you can see we don't have
3:52:29X. So we need to provide that. Okay. So
3:52:32this is the issue. Now let's change this
3:52:35and let's use sudo ch mode plus X. Now
3:52:40let's go back. Okay. Now now now it
3:52:43should be working. Now it should work.
3:52:44Uh in order to restart the service we
3:52:47use systemctl restart. Let's look at the
3:52:50status and voila it is running. So it
3:52:54says active running. Awesome. So now
3:52:57let's see what we have here. So here it
3:52:59gives us some info. It says the instance
3:53:01UID. Here it says that a master key has
3:53:04been set but it shouldn't show it. Yeah
3:53:07it's not showing it. So request search
3:53:10won't be authorized unless you provide
3:53:12an authentication key. This is the ex
3:53:14this is exactly what we have done here
3:53:16locally when we when you try to make a
3:53:18post request or something you need to
3:53:21provide uh a key to be able to
3:53:24communicate with the server. So this
3:53:26confirms it uh and yeah they have some
3:53:29other info uh like documentation some
3:53:32stuff like that but what's important is
3:53:34what we have here. So here we have
3:53:35starting one workers uh locally uh it
3:53:39said starting 20 workers. Uh let's
3:53:41verify that. Yeah, it says starting 20
3:53:44workers because I have a lot of cores.
3:53:47If you we look at the memory, let me
3:53:50show you and resources. As you can see,
3:53:51I have 20 cores. So that's why locally
3:53:54it it's going to use all of them. But we
3:53:57are using a poor VM. It doesn't have a
3:54:00lot of CPU cores. That's why it's going
3:54:03to work with what we have, which is one
3:54:06core, and that equals one worker. Uh
3:54:10yeah, it's it's working and it is
3:54:11listening. If we go here, it's listening
3:54:14on port 7,700.
3:54:16Awesome. So that means it's working. So
3:54:19now just hit Q to exit and let's
3:54:23continue. We have talked about option
3:54:25one. But now let's deal with the with
3:54:28other users who don't have their local M
3:54:31search instance. This section is is made
3:54:34just for those people. So for the other
3:54:36ones who have their own local instance,
3:54:38they can skip this section if they want
3:54:40or they can stick they you will learn um
3:54:43you will learn a lot. First let's verify
3:54:46if I have the JSON here. Uh let's look
3:54:49at web app. We need to go to back end
3:54:52and yeah I have the seed folder. Uh
3:54:55let's look at it. And we have
3:54:57documents.json.
3:54:59Awesome. You can read the readme if you
3:55:02want. Let's let me show you the readme.
3:55:04And here let's look at and it's it's
3:55:07going to tell you what you need to run.
3:55:09And here you you are not going to index
3:55:12them manually but I have a uh I have
3:55:15created a script for that. I called it
3:55:17see the mainly search and it is here. It
3:55:20is located inside the scripts folder and
3:55:23this will uh yeah will index the
3:55:26documents for you. Let's go inside back
3:55:29end or yeah what's what was command?
3:55:32Yeah, let's just take it like this. I
3:55:34just need to go inside
3:55:37web app. And now I can run this. Okay,
3:55:41it didn't work because I need to
3:55:43activate the virtual environment. This
3:55:45is the command source van bin activate.
3:55:48No such file or directory because I need
3:55:51to go inside the back end. Uh let's do
3:55:54it now. Okay, as you can see, we have
3:55:56van which means that we were able to
3:55:59activate the environment. Okay, now
3:56:02let's go back to this command and let's
3:56:04remove back end because we are inside
3:56:06that directory and let's run this. Okay,
3:56:09let's see. So here this is just a text
3:56:11see the mainly search with real data.
3:56:14Okay, here is the art the indexes we are
3:56:16going to create. Okay, so here we are
3:56:18trying to connect to mainly search. Uh
3:56:20so here it says index articles already
3:56:23exists because I have imported the dump
3:56:27successfully connected. Uh but here we
3:56:30have an error but it says that we have
3:56:33an invalid API key. That's
3:56:35understandable because I have created a
3:56:38new master key. So let's see what we
3:56:41have inside this file. Let's use nano.
3:56:45Oh sorry uh I need to keep the full
3:56:48path. Let's use nano. Okay. So here we
3:56:52are using I have a settings. I have a
3:56:56configuration and from that
3:56:58configuration I load the API key. So as
3:57:02you can see here we have API key is set
3:57:05to settings dot mainly search master
3:57:08key. This is why. So here I assume that
3:57:11we are loading a default master key. Uh
3:57:15okay that's why it didn't work. Let's
3:57:17see. So let's go inside settings just to
3:57:19show you what what we have there. Let's
3:57:21go inside core. Let's look at config.
3:57:25Yeah, as I told you, we are using the
3:57:27default master key. That's why it didn't
3:57:30work because because we are using a
3:57:32strong master key. And here, let's see.
3:57:36So, we have an environment variable. We
3:57:38have an environment variable that we can
3:57:41open. And there we can put the new
3:57:44master key. Or if you want, you can do
3:57:46it temporarily. You can just change this
3:57:49and the script will work. Let me change
3:57:51it inside the env file. And the env file
3:57:55should be here. Let's go back to back
3:57:57end. Let's use ls-l
3:58:00and let's see do we have env. Okay. So
3:58:03we just have env.example and from that
3:58:06we need to create env. Let's use cp
3:58:11env.ample and we will create av file.
3:58:15Let's open that file. And as you can see
3:58:18this is what we have. The environment is
3:58:20set to development. We can change that
3:58:22to production. Mainly search URL is
3:58:25correct. The index is correct. But I'll
3:58:28create um uh another one. Let's call it
3:58:31articles too because as you saw from the
3:58:33logs, we already have articles. And here
3:58:36I need to change this. And remember use
3:58:38the same key that we have put inside the
3:58:43env uh inside the env file. Uh let me
3:58:46show you. So we have this file where
3:58:50hopefully you have created a strong key
3:58:52and you put it here. I'm not going to
3:58:54show you my key but I will just open the
3:58:56file again. Copy it and paste it inside
3:58:59this env file that we created in the
3:59:03backend folder. I am back and I have
3:59:07changed the master key inside thisv
3:59:11file. Now we can try to run the Python
3:59:15script again. Let's do it. So, Python
3:59:18scripts seed med search. And as you can
3:59:21see this time, we didn't get any error.
3:59:24It says this is the index that we are
3:59:26going to create. It created it. It
3:59:29successfully connected to my search. And
3:59:31here, as you can see, documents were
3:59:33added successfully. Awesome. And I will
3:59:36show you later how to verify this with a
3:59:39dashboard. We can do it with the with
3:59:41the curl commands or the SDKs, but it's
3:59:45going to be very easy with a dashboard.
3:59:48Now, I want to do something. Uh let's go
3:59:51back to the service file here. We have
3:59:56we imported the dump successfully. Now,
3:59:59we don't need it. We can just get rid of
4:00:01these two lines because I as I said, we
4:00:03import the dump only once. We don't
4:00:05import it every single time. We made we
4:00:08made sure that we have the index the
4:00:11documents were there and yeah after uh
4:00:14verifying that you can safely clean this
4:00:16service by removing those two lines. Now
4:00:19save the file exit. We need to tell the
4:00:23dam that we need to reload the damon
4:00:25because we have we changed the service
4:00:28configuration and we also we need to
4:00:30restart the service itself. Awesome.
4:00:33Remember that we can access the web
4:00:37application using our IP. If you don't
4:00:40remember the IP address, you can always
4:00:43come back to Digital Ocean and find it
4:00:47here. I'll copy it. I'll paste it here.
4:00:50Let's go to the front end. So it's going
4:00:53to be
4:00:55sorry I'm not sure why I did that but
4:00:57let's just put the uh the IP address
4:01:00because we are using engineext and
4:01:02engineext will take care of uh routing
4:01:05the traffic but what why I want you to
4:01:08come back here because last time if you
4:01:10remember we weren't able to yeah we
4:01:14weren't able to talk to the to get
4:01:16articles from the back end because me
4:01:19search was not set up Now we have M
4:01:23search but we need to connect it to the
4:01:25back end in order to get articles cuz
4:01:28right now as you can see we we don't
4:01:30have anything. Now we just need to
4:01:32restart supervisor because it doesn't
4:01:35have the new environment variable. This
4:01:37is why it's not it's not able to connect
4:01:39to the me search server. And to do that
4:01:43we are going to use supervisor DTL. If
4:01:47you don't remember the project name that
4:01:49you used, you can always type status and
4:01:52that will give you the pro the yeah the
4:01:54name of that project and currently it is
4:01:57running and it was running for 10 days
4:01:59without any problem. Now we need to
4:02:02restart that. So let's use restart and
4:02:05let's put the name. You can use restart
4:02:07all but if you have many websites on the
4:02:10same server
4:02:11uh it's not a good idea. So it's always
4:02:13great to yeah just restart what you
4:02:16want. Okay. So it said stopped now it
4:02:19says it is saying started. Let's also
4:02:22follow the mini search service logs.
4:02:25This is what we have. It is saying
4:02:27invade the API key because that's when
4:02:30we run supervisor we use an old keys in
4:02:33the NV file. But hopefully now it should
4:02:36it should work. Uh let's verify this. So
4:02:39I am here. I'll refresh the page and
4:02:42voila, as you can see, I am able to get
4:02:44my articles and if we look at the logs,
4:02:48yeah, this is the latest request that we
4:02:51got. It's a post request. It's coming
4:02:54from Python because I am using the
4:02:55Python SDK and the status status code is
4:02:58200 and as you can see media search is
4:03:00fast. Uh so it was able to do this in
4:03:04less than 100 milliseconds and also the
4:03:07benefits of self-hosting misarch is that
4:03:09the communication is local because my
4:03:11back end is in local host and misarch is
4:03:14also uh in local host. If I hosted
4:03:16misarch on another server maybe uh that
4:03:19will add some latency but here
4:03:21everything is inside one box. Awesome.
4:03:24Let's also verify other stuff. Let's go
4:03:26to the homepage because we should see
4:03:28some blogs here. Yeah, that's correct.
4:03:30Let's all let's click on maybe this one.
4:03:34Yeah, let's click on this one. We have
4:03:36the content. We can also add some claps
4:03:41just to verify that it the data is
4:03:43saved. We also get recommendations which
4:03:46is great to see. The number of claps
4:03:48should have changed from 0 to five.
4:03:51Awesome. It is working. This is really
4:03:53great to see. We also have some filters.
4:03:55We can sort by popularity. So this sorts
4:03:59by the number of views. We can change
4:04:01that to ascending. So we have seven 7 11
4:04:0529. Uh we can change that to um number
4:04:09of claps. So these ones these these
4:04:12articles don't have any claps in them.
4:04:15But this one has the most number of
4:04:17claps. It has 15. This one has six five
4:04:20etc. We also can filter by year. And we
4:04:24have yeah we can filter by tags. And
4:04:26here we are using faceted distribution.
4:04:30Now we can let's go back to the machine
4:04:32and let's stop this because we don't
4:04:34need it. And as you can see the the
4:04:36request that we were performing were all
4:04:38successful. They all returned 200.
4:04:41That's all I wanted to share with you in
4:04:42this first subchapter in module 3. You
4:04:47now have a working uh mini search server
4:04:51hosted on your virtual machine. We
4:04:54connected it to the back end. We created
4:04:58a server, a service that will handle
4:05:01managing the mele search service for us.
4:05:05And we tested it in the front end and it
4:05:07worked wonderfully. In the next sub
4:05:10chapter, you will learn how to maintain
4:05:13this database. I mentioned that we can
4:05:16use the Python SDK or just plain curl
4:05:20commands, but it's not enjoyable. I'll
4:05:23show you a better way to maintain your
4:05:26server by uh connecting it to a uh
4:05:30goodlooking dashboard. We are also going
4:05:34to set we are also going to set up M
4:05:37search to automatically create daily
4:05:40backups for us in order to protect the
4:05:43data and we are going to use SSH
4:05:47tunneling again so that we prevent
4:05:50opening firewall ports and this is
4:05:53important because we want we don't want
4:05:55to host the dashboard. You can host the
4:05:57dashboard on your virtual machine but we
4:06:00in our case we don't want to do that. We
4:06:02just want to connect to it. See you in
4:06:04the next sub chapter. Welcome back and
4:06:08today we are going to manage your med
4:06:12search database securely. We are going
4:06:15to use this amazing open-source web
4:06:18dashboard. You can click on this link.
4:06:20It will take you directly to GitHub.
4:06:23Here it is. Uh you can give them a star.
4:06:26uh clone the code, run it locally if you
4:06:29want to, but the project is really
4:06:31awesome and I use it to maintain my
4:06:35database that I am using on my
4:06:36production server. There are other
4:06:38dashboards, but for me, this is the one
4:06:40that I liked. And this is how it looks.
4:06:42In your homepage, you can give your
4:06:45instance a name. It shows you the
4:06:48database size, status, and the version
4:06:51of the mill search binary. You can also
4:06:55create a dump, see the tasks, create
4:06:58keys and view your indices. It's really
4:07:01awesome. These are some some of the
4:07:03features that are present there. You can
4:07:04search documents, upload ones, edit the
4:07:08fields. Uh and all of this is done
4:07:11visually. You don't need to run any
4:07:12command. And again, in order to connect
4:07:15to it, we are going to use the same
4:07:17trick. We are going to use this amazing
4:07:19SSH channel feature so that we don't
4:07:23open open any ports. And this time as
4:07:26you can see we will have we have mill
4:07:28search running on the server. It runs on
4:07:31this port and we want to be able to
4:07:33access or to manage this using this
4:07:36dashboard. The dashboard is going to be
4:07:38running on a browser on our local
4:07:41laptop. And with SSH Simon, we will
4:07:44connect the ports 7,700 from our machine
4:07:48to uh 7,700
4:07:50on the server and they can communicate
4:07:52freely. But if when we stop the SSH
4:07:56connection, not we can't manage the med
4:08:00search instance. We already worked with
4:08:02SSH tunneling. So it should be uh very
4:08:05easy for you to reuse again uh in this
4:08:08sub chapter. Okay. Okay, now let's go
4:08:10back to the virtual machine. I want to
4:08:13start with automatic backups. So, let's
4:08:16open the server or the service. I want
4:08:20you to add two lines here. SK schedule
4:08:24snapshot and snapshot directory. Here
4:08:28the value in schedule snapshot is in
4:08:30seconds. I set it to 86,400
4:08:34and this is exactly 24 hours. uh if you
4:08:37want to save every hour or yeah if you
4:08:40want to change the duration feel free to
4:08:42do that just change this value and we
4:08:45want to save data to the snapshots
4:08:48directory but currently we don't have it
4:08:51so we need to create it and also I
4:08:53believe we should also create the dumped
4:08:55directory uh if you want for to to
4:08:58create some dumps yeah let's do this
4:09:00let's save the file we uh we can create
4:09:04the snapshots directory like this And
4:09:07let's also create dumps if we don't have
4:09:10it. Okay, great. And now let's change
4:09:12the ownership. I will change everything
4:09:16inside my search, not just snapshots
4:09:19because I have also created dumps and I
4:09:21made a typos. Uh let me show you how we
4:09:23can do this. We can use move. Let's use
4:09:26sudo move. Let's take this path and
4:09:30let's rename it. Instead of dumps, it's
4:09:32going to be dumps. Uh let's just verify
4:09:34that it it's worked and let's look at me
4:09:38search and yes we have dumps we have
4:09:41snapshots and data and we can delete
4:09:44this dumps file because we no longer
4:09:46need it. We used this we imported the
4:09:49data now it's it's not useful anymore.
4:09:52So let's use sudo remove and let's put
4:09:55the file there. Sorry. Okay. So let's go
4:09:59back. Ah I wasn't able to delete it.
4:10:01Why? Ah, I did sudo ls. I need to do
4:10:05sudo remove. Now let's go back. Awesome.
4:10:08So everything is there. Let's just
4:10:10verify the ownership. Who owns those
4:10:14folders? M search. Awesome. Now we can
4:10:17continue. Whenever you update the
4:10:20service file, make sure to run damon
4:10:23reload and reload the service itself.
4:10:27Now, Med Search will quietly save a safe
4:10:31copy of your database every single day.
4:10:35And I want to mention that we will keep
4:10:37just the most recent snapshots. We won't
4:10:40have a lot of snapshots because that
4:10:43will uh fill up your hard drive over
4:10:46time. You we just need one and that's uh
4:10:48that's sufficient.
4:10:50Let me exit and let's use this command.
4:10:53SSH-L7,700
4:10:58to 7,700. This will will create a
4:11:01tunnel.
4:11:02Ah, so it says address already in use
4:11:04because I am running Milly Search
4:11:06locally. So I need to stop that server
4:11:09and re retry again. So let's retry.
4:11:13Okay, now it worked. It didn't uh it
4:11:16didn't tell me that the port is
4:11:17occupied. Now let's go back to GitHub
4:11:20because as I mentioned you don't need to
4:11:23host the dashboard locally because this
4:11:27the creators of this project are already
4:11:30hosting it uh on Versel for free and as
4:11:33I said I am using it uh both locally and
4:11:36on my production server and in order to
4:11:40add your own instance you can just click
4:11:43on the plus button give it a name I will
4:11:47call
4:11:48test instance. The host is correct. It
4:11:52should be 7,700.
4:11:54This one is pointing at your local port.
4:11:56And here I need to provide the API key.
4:11:59For that I need to go back here and um
4:12:03copy the code the API key. Again I have
4:12:06copied the key. I have pasted it. Now I
4:12:09click on confirm. Okay. I can save that
4:12:12but doesn't it's not needed. Now just
4:12:15click on your instance. One thing the me
4:12:18search server should be running. So you
4:12:20can always run status if it is if it
4:12:24says running you are good to go. And as
4:12:27you can see I uh here are my indices. So
4:12:29I have articles articles number two and
4:12:33articles test. This was also on my dump.
4:12:36And I have everything. Now I can click
4:12:38on let's say articles. It shows me the
4:12:41distribution. Uh we can click on
4:12:43documents. And here it shows you the
4:12:45fields. You can click on updates. Let's
4:12:48say that I want to update the view count
4:12:50from 14 to something else. I can do it
4:12:53without any problem. So maybe let's just
4:12:56do 999 or something. Submit. It's going
4:12:59to create a task. We can look at that
4:13:02here. So it's it says a few seconds ago
4:13:05and it succeeded. Uh let's go back to
4:13:08here. Let's go up. Let's go to blogs.
4:13:12Let me see. Do we have what was that
4:13:14blog? Let's go back. So, it says elastic
4:13:17search change heap size. Elastic search
4:13:21change heap size. Do I need to refresh?
4:13:24Oh, no. Maybe I used the wrong index.
4:13:27So, let's go back. Okay. Let's click on
4:13:29articles. Yeah, I think I am using
4:13:31articles too instead. Uh, let's look at
4:13:34documents. Okay. Yeah, let's change this
4:13:37one. So, update from 7 to 999.
4:13:41submit. Go back to the website. Refresh.
4:13:45And as you can see, we have 9999. With
4:13:47this dashboard, you can easily manage uh
4:13:50your documents. And here, if you want,
4:13:52you can upload once. You can upload your
4:13:54documents here. Just paste the JSON and
4:13:57hit submit. And that will uh add a new
4:14:01article or product, whatever. Uh for in
4:14:04my case, I am using it to add articles.
4:14:07Uh so that's my content. But if you use
4:14:10it for other purposes, it doesn't
4:14:13matter. You can just just put your JSON
4:14:14here and hit submit or you can drag and
4:14:17drop the file directly. Uh let's let's
4:14:19look at settings here. Uh these are
4:14:21these are the the settings. And this is
4:14:24very important. When you create a dump,
4:14:26the settings needs to be the same. And
4:14:29just like that, we finished module
4:14:31number three. As you can see, using me
4:14:34search is very easy. It's just one
4:14:36binary. It is really fast.
4:14:39uh it's uh we have really awesome
4:14:42dashboards that we can use to maintain
4:14:45this database. We tried to create a
4:14:49system user to isolate the database as
4:14:52much as possible so that we don't leak
4:14:54data, we don't leak keys, stuff like
4:14:57that. And even if someone hacks to your
4:15:00server, they won't be able to extract
4:15:04anything from it. And we used SSH
4:15:06tunneling to access this dashboard.
4:15:09Which means that if I stop the server as
4:15:13you can see I stopped it which means
4:15:15that yeah port forwarding is no uh sorry
4:15:19SSH tunneling is no longer a thing. That
4:15:22means that yes I won't be able to
4:15:24connect to this instance. So if I click
4:15:26on it it says connection failed because
4:15:29we don't we create a temporary tunnel
4:15:32that allows us to manage this. Once we
4:15:34are done, we close it and no one will be
4:15:36able to talk to me research. So even
4:15:39though it is inside the machine, it is
4:15:41isolated. It cannot talk to the outer
4:15:45world. It just talks to the back end
4:15:47locally. In the next module, module
4:15:50number four, we are going to talk about
4:15:53global delivery and app security. This
4:15:57is important because as you can see
4:15:59right now if uh let's say that you
4:16:01deployed this to production your users
4:16:04will have to type your IP address to
4:16:08access your website. This is not good.
4:16:10We need to change this to be something
4:16:13like this. So so that they can just type
4:16:16a name and
4:16:18in behind the scenes this name will be
4:16:20converted to the uh to your IP address.
4:16:24So this is what we are going to fix in
4:16:26the next sub chapter and we are going to
4:16:28talk about DNS records creating custom
4:16:31domains uh adding certificates to uh
4:16:35finally uh have a secure connection
4:16:37because right now it is not secure it is
4:16:40we are just working with HTTP uh in the
4:16:43next module we will finally upgrade to
4:16:46HTTPS and that will add another layer of
4:16:48security to your website.
4:16:52Welcome to module four.
4:16:54Up until now, our application has been
4:16:58running perfectly. But to access it, we
4:17:02need to type the IP address in the
4:17:05browser. This is not user friendly. In
4:17:08this video, we are going to buy our own
4:17:13domain name. I'm going to show you how
4:17:16to transition from HTTP to HTTPS. We are
4:17:20going to use Let's Encrypt for that. We
4:17:24are going to modify the engineext
4:17:26configuration and at the end I'm going
4:17:29to show you how to autorenew your SSL
4:17:32certificate.
4:17:34This is the agenda that we are going to
4:17:36follow. As you can see, we are going to
4:17:38start with purchasing a custom domain.
4:17:42We are going to configure DNS records.
4:17:44We are going to add an A address and a
4:17:47CNAME record. I'm going to explain
4:17:49these. We are going to install three SSL
4:17:52certificates via list encrypt. And
4:17:56finally, we are going to automate the
4:17:58renewal process because by default, the
4:18:03certificates are going to expire after
4:18:0590 days and we don't want to do the
4:18:08renewing manually. We want the machine
4:18:11to do it for us. We are going to use a
4:18:14chrome job for that. Before we start
4:18:17practicing, I would like you to go to
4:18:19GitHub. You'll find the links for this
4:18:22repository in the video description.
4:18:25Click on the book folder and come here
4:18:27to the fourth file. As you can see, I
4:18:30have committed the changes. Everything
4:18:33is here. You will find the steps and the
4:18:36commands that you need to run.
4:18:38Everything is described in this
4:18:40document. So, don't worry. Just follow
4:18:43with me and uh the resources are on this
4:18:46repository. Let's register a domain. In
4:18:50my case, I'm going to use pork bun but
4:18:53if you want you can use namecheep or god
4:18:56daddy or other registars. There are many
4:19:00but in my case I will use pork bun
4:19:03because this is the one I use and their
4:19:06pricing is transparent. They they show
4:19:10you the real price. Sometimes the other
4:19:12ones they show you a very low price but
4:19:15the problem is that when you try to
4:19:18renew in the next upcoming years they
4:19:22the price becomes higher. So that's not
4:19:24good. And here in pork bun as you can
4:19:27see you can type the domain that the
4:19:29name that you want to register and if it
4:19:31is available you can uh you can take it.
4:19:35I will I will sign in because I already
4:19:37have an account and I will create a I
4:19:39will try to register a new domain name.
4:19:42I am back. As you can see, I have logged
4:19:45in with my account. I already have two
4:19:48domain names. This is the one that I
4:19:51showed you in the beginning of this
4:19:53course. If I go there, it takes me to my
4:19:57personal website. And as you can see
4:20:00here, uh, if you look at at the bottom
4:20:03left corner, the link is HTTP. It's
4:20:07insecure. But when I click on it, it's
4:20:10going to use HTTPS because we
4:20:13reconfigured EngineX so that it takes
4:20:17HTTP requests and redirects them to
4:20:19HTTPS. And as you can see, if you click
4:20:22here, it shows you that the connection
4:20:24is secure. And this is what we are going
4:20:26to do with the new domains that I will
4:20:29create now. Okay, I will close this.
4:20:32This is the domain name that I would
4:20:35like to register and I want a com one.
4:20:41Let's search and let's see if it is
4:20:42available. Okay, let me zoom in so that
4:20:46we can see clearly. As you can see, I
4:20:48have typed the name here and look what
4:20:51we found. As you can see, I can I can
4:20:54get the domain for $11 per year. And
4:20:59this is what I was talking about the
4:21:01renew process. As you can see, the renew
4:21:03is the same as the price that I will pay
4:21:06in the first year. But if you if you
4:21:08look at this one, I am going to pay just
4:21:11under $2. But after this year, I will be
4:21:15paying almost $30. And this is why I
4:21:19mentioned in the beginning that the
4:21:20pricing here in pork one is transparent.
4:21:23You can see everything. In my case, I
4:21:25will take the com one because yeah, this
4:21:28is what I want. And I want to also get
4:21:32ma but sadly yeah this one is uh country
4:21:36specific because I live in Morocco. I
4:21:37would like to also take that one. But
4:21:40yeah, you can try your luck. Uh maybe if
4:21:43you live in another country, you can try
4:21:45it. Maybe you will find uh yours here or
4:21:47maybe use a different uh register. Uh I
4:21:51know that I will not find that one. So
4:21:53I'll just skip
4:21:55and in my case yeah I think this one is
4:21:58is fine. So now what we will do we will
4:22:01click on the plus button that will add
4:22:03it to the cart. Now click there and here
4:22:08again pay attention to what they are
4:22:09showing you here. So the registration
4:22:12here is for one year. I want to pay just
4:22:14one year. Uh after that they will notify
4:22:17me to pay for the the second and the
4:22:20upcoming years if I want to stick with
4:22:22them. So the renewal is the same price.
4:22:25That's correct. And here they are giving
4:22:28you other things. They always do this so
4:22:30that you that you pay more. But don't
4:22:33don't add anything. These are the things
4:22:36that they show you here are not are not
4:22:37important. Okay. So this is uh this
4:22:41looks good. Now all we need to do is to
4:22:44scroll a little bit and here click on
4:22:47continue to billing and finish the
4:22:49payment and you can also ver verify the
4:22:52price before doing this. I will click on
4:22:56continue to billing. Here you have many
4:22:58options. You can use PayPal if you have
4:23:00it, crypto or your Visa card. I will
4:23:04select PayPal and I'll I'll finish the
4:23:06process and I'll come back. I am back.
4:23:09As you can see, this is the message that
4:23:11you should see after completing the
4:23:13purchase. They will tell you great
4:23:15success. Congratulations on your brand
4:23:17new domain. That's uh that's good. Now,
4:23:20we can go back. We can click on account
4:23:22domain management. And as you can see, I
4:23:26got my new domain. Uh they say that it
4:23:29will take some time. That's fine. We'll
4:23:30come back to it. Now, we need to
4:23:32configure DNS domain name system. But
4:23:37before I do that, let's go back to the
4:23:39slides because I want to explain DNS how
4:23:42DNS works. I created this diagram to
4:23:45explain this. Let's say that I am this
4:23:48user. I want to check this website. I
4:23:53will type it on this browser that I have
4:23:56here. I will hit enter. What happens
4:23:59after I do that? The browser will talk
4:24:02to the DNS server. In our case, it might
4:24:06be pork bun. It might be cloud player.
4:24:09It depends on what you want to do. For
4:24:12right now, we have created we have just
4:24:15bought a domain and pork bun. They have
4:24:18their own DNS server. Behind the scenes,
4:24:21what will happen is that we need to to
4:24:23give Porkbun a an IP address. And this
4:24:27IP address is the one that we that we
4:24:31can take from Digital Ocean. It's the
4:24:33machine that we created in the first
4:24:35module in this course. And this will
4:24:39will allow pork bun or other DNS servers
4:24:43to create this mapping. They will map
4:24:46the name the domain that you have
4:24:49purchased to that IP address. After
4:24:52typing this domain name, the browser
4:24:56will tell the DNS server where is this
4:25:00address. The DNS server will look in in
4:25:03its databases. It will if it finds it,
4:25:06if it finds that address, it will reply
4:25:09with the IP address and the browser will
4:25:12type that IP address automatically. It's
4:25:15the same. Before we were typing the IP
4:25:18address directly, but now the DNS server
4:25:21will give that information to the
4:25:22browser. Now the browser has the IP
4:25:25address. It will communicate with your
4:25:28Linux server. So it's going to tell it
4:25:30please give me the content and the the
4:25:33server will reply with the content and
4:25:36you will be able to see the the front
4:25:38end that you built. This is how DNS
4:25:41works and this is what we are going to
4:25:43do to create this mapping. We need to
4:25:46configure the domain that we bought in
4:25:48porkbun. We call this mapping the a
4:25:51record. Let's go back to porkbun and
4:25:55let's see if we can change this. I'll
4:25:57refresh the page. Okay, everything is is
4:26:01all right. As you can see, we have DNS
4:26:03here. If you click on it, it allows you
4:26:07to modify it. As you can see, we have uh
4:26:11a few default ones. Yeah, and I think
4:26:14this is a newer one. I will I'll go back
4:26:16to the classic view because this is
4:26:18Yeah, this is the view that I used here
4:26:21uh on GitHub. Let's uh let's see. Yeah,
4:26:24as you can see, before it was like this.
4:26:26And yeah, we get a model. So I I'll
4:26:29stick to the old one. You can see that
4:26:32we have a few records. These are default
4:26:34records that Porkman created for us. We
4:26:37need to remove some of them. I will
4:26:39start by removing the alias and CNAME by
4:26:43clicking on this trash icon. We will
4:26:46also remove the last two, these text
4:26:48ones. That's what we want to do. Now
4:26:51scroll up and here let's add the first
4:26:54one is the address record. Here host
4:26:57leave it blank but here in the answer
4:27:01value we need to put the IP address.
4:27:04Let's go to digital ocean because I
4:27:06forgot the IP address. Okay, here is the
4:27:09the droplet. Here is the IP address. I
4:27:12copied it. I will paste it here. Here we
4:27:15have time to live. Leave it at 600
4:27:18seconds. Um, you can add notes if you
4:27:21want. And now click on add. Okay, let's
4:27:25scroll down. As you can see, we have the
4:27:27new record here. That's great. Now,
4:27:29let's add another one, which is going to
4:27:32be CNAME. This record maps one domain
4:27:35name to another domain name. You use
4:27:38this one to ensure that if someone types
4:27:42www
4:27:44in front of your domain, they can still
4:27:46reach your website. So this is going to
4:27:48map www to uh your domain name. I can
4:27:52show you this with my own website. If I
4:27:54type my domain name, I can go there.
4:27:57That's fine. But I can also do let's
4:28:00remove this part and type ww dot.
4:28:05And as you can see that redirects me to
4:28:08my yeah to my website. And it's it's
4:28:10working. To accomplish that we add we
4:28:13add the CNAME record. Here in host we
4:28:16add www.
4:28:19And here in answer you need to type the
4:28:21domain name and we can just copy it from
4:28:23here and paste it and hit time to live.
4:28:27We shouldn't change that. Now just add
4:28:31this record. Let's scroll down. As you
4:28:34can see we have it here. This is the
4:28:36mapping that I was talking about. If I
4:28:38type this domain name porkbang will
4:28:41return this IP address to the browser.
4:28:44So that so that it can communicate with
4:28:46my droplets and that's it. We configured
4:28:49DNS as I said I used pork bun but what
4:28:52we have learned here applies to the
4:28:55other domain registars the only thing
4:28:57that will change is their user interface
4:28:59but other than that everything I think
4:29:02should be should be the same. We need to
4:29:06SSH into the server. Let's do this. I
4:29:10will use this command to SSH directly.
4:29:14Great. I am now inside the machine. We
4:29:16need to configure SSL certificates. How
4:29:20do we do that? First, we need to install
4:29:22the searchbot client. To do that, we can
4:29:26type this command. As you can see, it
4:29:28installs searchbot and this Python 3
4:29:32searchb engineext. Uh this is a plug-in
4:29:36that allows searchbots to modify the
4:29:39enginex configuration for us. We will
4:29:41see that in action but for now let's
4:29:43just install this. Provide the password
4:29:46and let's wait for a few seconds. Okay.
4:29:49Now let's request certificates for this
4:29:53domain and subdomain. As you can see I
4:29:55have placeholders. I need to replace
4:29:58them with this domain name. I will take
4:30:01it and paste it here. And uh here I will
4:30:04keep the www do and I'll paste the
4:30:08domain name. Here I made a mistake. I
4:30:10need to remove the.com from here and
4:30:13from here. Let's hit enter. The first
4:30:16thing they are asking for your email
4:30:19address. This information is used
4:30:22strictly for urgent security notices and
4:30:26renewal warnings if uh maybe the
4:30:29automation fails. After that, they will
4:30:31ask you if you agree with their terms of
4:30:34service. Type yes. And for this
4:30:37question, they are telling you, do you
4:30:39want to share your email address? Just
4:30:41type no. The account is registered. And
4:30:44now they are requesting a certificate
4:30:46for my domain name. Let's wait for a few
4:30:49seconds. And okay, this is the message
4:30:52that we want to see. Successfully
4:30:54received certificate. And as I mentioned
4:30:58the certificate will expire after 90
4:31:00days. Today is April 4th and 90 days
4:31:05will be yeah it's 3 months. So yeah 4 +
4:31:093 is 7. That's correct. But don't worry
4:31:12we are going to automate the process of
4:31:14renewing the certificates. Let's open
4:31:17the engineext configuration and let's
4:31:20see if searchbot changed the changed it
4:31:24or not. Hit enter and let's scroll down.
4:31:28It seems to me that it's it's the same.
4:31:32Source B did not change it. That's not a
4:31:35problem. I wanted to show this issue
4:31:38because as you can see in the previous
4:31:41chapters, we didn't have a domain.
4:31:43That's why we created just a dummy name
4:31:47here. But the new domain that we created
4:31:50is called this. and searchbot didn't
4:31:54find something uh there. That's why it's
4:31:57it uh yeah it couldn't change it. We can
4:32:00see that search bot because it couldn't
4:32:03find the domain name inside the the
4:32:07configuration. It created the default
4:32:10configuration. Let's remove that because
4:32:12we don't need it. Now open the
4:32:15configuration that we created before.
4:32:17And here we need to update server name.
4:32:20As you can see before because before we
4:32:22didn't have a domain name, we used the
4:32:24IP address. But now we need to change
4:32:27that. Let me show you how we do it. I
4:32:30will come here. Let's hit Ctrl K to
4:32:33remove that line. I will replace it with
4:32:35this one. Server name stays the same,
4:32:38but here we replaced the IP address with
4:32:42the domain name and the subdomain name.
4:32:45Save the file. Exit. Let's ver let's
4:32:49verify that the syntax is correct.
4:32:51Always do that before reloading engine
4:32:53X. And now I will search for reload
4:32:57engine X. And to search you just type
4:33:00something, hit CtrlR and that will show
4:33:03you the previous commands in the
4:33:05history. Hit enter. This is going to
4:33:07reload EngineX because whenever you
4:33:10change the configuration, always check
4:33:12the syntax and after that reload. Now
4:33:14let's run searchbot again with the same
4:33:17command and this time it will find the
4:33:19configuration because we ran this
4:33:22before. Searchbot is asking us if we
4:33:25want to reinstall. If we want if we
4:33:27choose one that will attempt to
4:33:29reinstall and two will just renew. We
4:33:32want to reinstall. So let's type one and
4:33:34hit enter. As you can see this message
4:33:37shows us that third boss was able to
4:33:39find to find the our custom
4:33:42configuration. And now if we open that
4:33:44file, we should be able to see the new
4:33:49stuff that SB added. Let's scroll down.
4:33:52Great. As you can see, it worked. We
4:33:54have some new lines. Let's remove them.
4:33:57Let's go up. Yeah, everything here
4:33:59stayed the same. We have the server
4:34:01name. We We have log files. These are
4:34:04the HTTP headers that we added before.
4:34:08Here we are serving the front end. We
4:34:10serve the API. That stuff didn't change.
4:34:13But here, here is the new stuff that
4:34:16search bots added. And as you can see,
4:34:18this comment proves that we have listen
4:34:21443 SSL. We have some SSL certificate
4:34:25stuff. Let me explain what we have here.
4:34:28Before in the server block, we were
4:34:30listening for port 80. Searchbot
4:34:33migrated that to port 443.
4:34:36And for this port is the standard port
4:34:40for HTTPS. You can search that on the
4:34:42web to verify that information. But port
4:34:4580 is for regular HTTP and it is
4:34:49unencrypted. That's why it is it is not
4:34:52secure. Great. So that's nice. And here
4:34:56um these these new lines just point to
4:34:58the to the keys that search B created
4:35:02for us. We are including this
4:35:05configuration. Yeah, this stuff it's not
4:35:07that important because it is managed by
4:35:09search. The most important thing here is
4:35:12443. It will make our website secure.
4:35:15Now if we scroll down, we have another
4:35:16server block and this one is used for
4:35:19redirecting. Here we have two if blocks.
4:35:22If the host is this subdomain, we are
4:35:25going to redirect to HTTPS
4:35:28and 30. Let's search for it. 301 means
4:35:33moved permanently and it is used for
4:35:36redirection. And this is what we want to
4:35:38do here. If someone visits www do we
4:35:42want to redirect them to https and here
4:35:45we put the domain name and again this is
4:35:48managed by searchbot and here if we type
4:35:51just the domain name we will catch that
4:35:54and we will redirect to this link and as
4:35:57you can see here for this server block
4:35:59we are listening on port uh port 80
4:36:02which is the regular HTTP which is not
4:36:05secure. Yeah, when we catch that
4:36:07traffic, we will redirect it to HTTPS.
4:36:10Awesome. Now save the file and exit. Now
4:36:14let's automate the certificate renewal.
4:36:17Let me clear the screen. We have this
4:36:19command that we can run to test the
4:36:23renewal system. We want to do this to
4:36:25ensure that it works. And we are adding
4:36:28dry run because this is just a test.
4:36:30It's not going to change anything. Let's
4:36:32verify that this is going to work. They
4:36:34found that the account is registered.
4:36:37That's good. And as you can see here,
4:36:39they are saying simulating renewal. So
4:36:41this is just a simulation. And they are
4:36:43saying congratulations all simulation.
4:36:46All simulated renewals succeeded. This
4:36:48is what we want to see before
4:36:50proceeding. Now let's create a new
4:36:53chrome job. We are going to type sudo
4:36:56chrome tab-e.
4:36:58We want to do this in for the root user.
4:37:02This is the first time we are doing
4:37:03this. So the this command is showing us
4:37:06what text editor to use. I like to use
4:37:09nano. I will type one and hit enter. Now
4:37:12let's go down and let's add this line.
4:37:15If you don't understand the chrome tab
4:37:18syntax, you can use this website called
4:37:22chronab.guru.
4:37:24And here you can paste the chrome tab
4:37:27notation. And this is the one that I
4:37:30added in the chrome tab. Why I like this
4:37:33website is because they show you they
4:37:35try to convert this into a sentence that
4:37:37you can understand. And this expression
4:37:41means at exactly midnight every Sunday.
4:37:46And this is the command that we want to
4:37:48run every sun. Every Sunday we want to
4:37:52run the renew command. And we are adding
4:37:55quiet because we don't want to generate
4:37:58a lot of system logs. We want to do this
4:38:00quietly. Now save the file. Exit. Good.
4:38:04Now Linux will run this job every
4:38:08Sunday. And again, as I said, you don't
4:38:10need to do this manually. The system
4:38:12will take care of the renew process. But
4:38:16we have a small problem. Let's say that
4:38:19before the expiration of the
4:38:22certificates, we were able to
4:38:24automatically renew it. Engine X will
4:38:26not be able to see it because we need to
4:38:29reload it in order to load the new
4:38:32certificates. In order to solve this, we
4:38:34will create a post renewal hook. I will
4:38:38explain what that means. Let's create
4:38:40this file. Look where we are creating
4:38:43it. We have etc. Let's encrypt. This is
4:38:46what we are using for the certificates.
4:38:49And here they have a renewal hooks
4:38:51folder. And we want to do this in the
4:38:54post. So after renewing this is what
4:38:56post means after an operation. After
4:38:59that we are going to reload engineext.
4:39:01This is a simple script that will run
4:39:04this command for us whenever the
4:39:06certificates are updated. Save the file
4:39:09and exit. But don't forget to make it
4:39:12executable so that searchbot can execute
4:39:15this. And you can always verify if that
4:39:18works by using ls-la.
4:39:23And what you want to see is X in yeah
4:39:26three positions and okay you are now set
4:39:29up. You can just kick back and enjoy
4:39:31your sleep. I think we have finished the
4:39:35full setup. Now we can test if this
4:39:38works or not. Let's go back. You
4:39:40remember that before we used to go to
4:39:43Digital Ocean, copy the IP address,
4:39:46paste it here in order to uh see the the
4:39:50website. Now this does not work because
4:39:53we have uh updated the engineext
4:39:56configuration to look for the domain
4:39:59name. Let's do that. I will or I can do
4:40:02it from here. I can click on this and
4:40:04let's hope that it works. Yes, it did.
4:40:06As you can see, I have typed the domain
4:40:09name. This is my website. Uh as you can
4:40:13see, it is working. Uh the front end and
4:40:15back end are both connected. Let's load
4:40:18an article. Let's scroll down a little
4:40:21bit. This is a long article here. I can
4:40:24add some claps. Let's go up. Let's
4:40:27ensure that this works. Yes, it says 11.
4:40:29That's the new claps. Yeah, everything
4:40:32is working. And we can also try to force
4:40:36this to go to HTTP. Uh let's use that.
4:40:41And here I will type the domain name.
4:40:43What will happen? This will go this will
4:40:46redirect to HTTPS automatically. Boom.
4:40:48That worked. As you can see, the
4:40:50connection is secure. Awesome. This is
4:40:53great. Uh, that works. And we can also
4:40:56verify www do. Let me remove this part.
4:41:01And let's hit enter. And that also
4:41:04redirects without any problem. Amazing.
4:41:08Now we have uh successfully created
4:41:12purchased a domain. We changed the DNS
4:41:17configuration. We added the A and C name
4:41:20records. We used let's encrypt to make
4:41:24the connection encrypted. But we are not
4:41:28done yet. In the next subchapter, we are
4:41:31going to talk about CDN layer. Why do we
4:41:35need to talk about that? Because the
4:41:37problem is that for me, this droplet
4:41:41that I created is hosted in Germany. I
4:41:44live in Morocco. So if I want to access
4:41:47this website, it's going to load fast.
4:41:49But if let's say that someone is from
4:41:52China or Australia, if they want to to
4:41:55go to my website, the load time is going
4:41:57to be very slow. They need to wait for
4:42:00seconds instead of milliseconds to yeah
4:42:03to load the the content of the website.
4:42:06That's a problem. We are going to fix it
4:42:08in the next sub chapter with the help of
4:42:11Cloudflare. We are going to use it to
4:42:14cache the static files on their servers
4:42:17because they have servers all over the
4:42:20world and that will mitigate this
4:42:22problem. So that person in Australia
4:42:24that before was used to wait in seven or
4:42:288 seconds for the website to load, we
4:42:30now only wait for maybe a second or
4:42:33less. And that's what we are going to
4:42:35focus on in the next uh video. Welcome
4:42:39back. In this video, we are going to
4:42:42talk about the CDN layer and caching.
4:42:47Here is what we are going to work on. In
4:42:50this video, I'm going to show you how to
4:42:53route traffic through a content delivery
4:42:56network. There are many, but we are
4:42:59going to use Cloudflare for this
4:43:01tutorial. We are going to configure
4:43:04strict end toend SSL encryption. I will
4:43:08show you how to unmask the visitor IP
4:43:11address in engine X because now
4:43:13Cloudflare will sit between the user and
4:43:17engine X. This is an important step that
4:43:20we should not forget. And finally, I
4:43:24will show you how to implement caching
4:43:26strategies for single page applications.
4:43:30Let me show you why we need to rely on
4:43:34CDNs, especially for applications that
4:43:38don't require a backend. Last time I
4:43:41bought this domain to demonstrate how to
4:43:45link your domain to your VM. And as you
4:43:50can see, the application loads and uh
4:43:52here is my website. You can use this
4:43:55free utility that allows you to check
4:43:59the load times. Here you can test. Let's
4:44:03test this website. Keep from all
4:44:05checkpoints and click on start test.
4:44:10Here is the results that we got. As you
4:44:13can see, this shows us the result. Shows
4:44:17us a lot of information. But as you can
4:44:19see, it tests the load time of our
4:44:21website in different locations. You can
4:44:23see that here Frankfurt is in Germany
4:44:26because I am hosting this server on a
4:44:30virtual machine in Germany. That's why
4:44:32the load time is basically zero. But if
4:44:36we go um let's see. So here the the
4:44:40worst load response time is almost 5
4:44:44seconds which is in China. It makes
4:44:47sense because the distance between
4:44:48Germany and China is very big. We we
4:44:52will also see here u yeah as you can see
4:44:55we have Japan we have yeah other
4:44:58countries that are very far but the the
4:45:01countries that are countries that are
4:45:03close to Germany uh especially in Europe
4:45:07Africa those will uh see very uh very
4:45:10low uh load times. This is a good tool
4:45:14that you can use to test how fast your
4:45:17your website loads for users. I wanted
4:45:21to start with this because it shows the
4:45:23problem that we currently have. And if
4:45:27you want to solve this, one option is to
4:45:29host your website on many servers. For
4:45:32example, you can host one maybe
4:45:34somewhere in Europe. That will give
4:45:37people in Europe
4:45:39very high load times. Maybe you put
4:45:42another one in um USA, you put another
4:45:45one in Africa, another one in Asia. But
4:45:48that will drive the cost up very
4:45:51quickly. CDNs solve this issue for us.
4:45:54They allow us to serve static files to
4:45:58users all over the world. And here cloud
4:46:01player, this is what we are going to
4:46:04use. They have plenty of servers uh
4:46:07scattered around the world. And let me
4:46:09show you this diagram that I made that
4:46:12explains how this works. We have two
4:46:15situations. The first one is before
4:46:18applying CDN and the other one is after.
4:46:21Let's now look at the first diagram.
4:46:24Let's consider that the server here is
4:46:26hosted maybe here in somewhere in
4:46:28Morocco. And let's say someone here in
4:46:31Asia wants to check my website. The load
4:46:34time will be let's say almost 4 seconds.
4:46:38And this user will see a smaller load
4:46:41time but still it's not it's not very
4:46:43responsive. 1.2 seconds is bad. But if
4:46:46someone is here in Morocco or Algeria,
4:46:49the the website will load instantly for
4:46:52them. This is the problem that we have
4:46:55distance that we need to travel and that
4:46:58takes time. With CDN, we'll solve this
4:47:01by including edge servers. These servers
4:47:05are located close to the end user. This
4:47:10person who lives in Asia will connect to
4:47:14a server
4:47:16the closest server to them. We call them
4:47:19edge. And that edge server will
4:47:21communicate with the main server. And
4:47:24these edge servers are provided or are
4:47:28given to us by Cloudflare. And here they
4:47:31will cache your static files just the
4:47:34front end so that users can load them
4:47:36instantly. And as you can see, with the
4:47:39help of these edge servers, we were able
4:47:42to bring down this 3.9 seconds to almost
4:47:46zero. This is the benefit. And I will
4:47:49show you how to do this in this video.
4:47:52And the great thing about this is that
4:47:54we have the before situation. After
4:47:56applying the test, we will come back
4:47:58here. We will uh start the test again.
4:48:01And hopefully we will see that for
4:48:03example for Beijing it will go from 5
4:48:05seconds to maybe uh 1 second or less
4:48:09hopefully and uh we will see that at the
4:48:11end of this video. Now go to
4:48:13cloudplayer.com.
4:48:15Let me zoom out a little bit so that you
4:48:17can see. And here make sure to create an
4:48:22account. If you don't have it you can
4:48:23loging in for free. They have a free
4:48:25tier which we we are going to use in
4:48:27this video. I will log in and I'll come
4:48:29back. After you log in, you will see a
4:48:33add button. Click on it and choose
4:48:36connect a domain. Here I will type my
4:48:40domain name. I will choose this one.
4:48:44Keep import DNS records automatically
4:48:48checked. This will allow Cloudflare to
4:48:51automatically fetch the A and C name
4:48:55records that we created uh in pork bun
4:48:59or in your preferred domain registar.
4:49:03Here we have a section uh about AI
4:49:06crawlers. If you don't want AI companies
4:49:10to scrape your websites, select block on
4:49:14all pages or you can select the other
4:49:17options. I don't want them to scrape my
4:49:20content. So, I will select this. And
4:49:23here we have this option that will
4:49:25instruct AI bot traffic with robots txt.
4:49:28You can read more about this. Uh it's an
4:49:31it's a setting that Cloudfare will add
4:49:34to robots txt to block AI companies from
4:49:39scraping your website. As you can see,
4:49:41they have a few few options. So, they
4:49:43have AI input, AI train. If you want,
4:49:45you can read read it here in the
4:49:47documentation. I'll keep that enabled
4:49:50and I'll hit continue. As I mentioned,
4:49:52they have a free plan that you can use.
4:49:56They also have other plans. If you want
4:49:58to scale your website, you need to move
4:50:00to maybe the pro plan. But for this
4:50:04tutorial, the free plan will be enough.
4:50:07And here they are showing you the how
4:50:09much you will pay monthly or yearly. We
4:50:11don't care about that because yeah we
4:50:13will be on the free pan anyway. This
4:50:16step is very important. Cloudflare here
4:50:19is listing the DNS records. It's found
4:50:22on pork bun. Here we need to verify the
4:50:26status. Here you can see that we they
4:50:28show you two icons. Uh so if the icon is
4:50:32orange like we see here, it means it's
4:50:34proxied which means traffic will go
4:50:37through the cloudflare network network.
4:50:39And this is what we want. Here we have
4:50:41the A and C name records. They are
4:50:44pointing to the correct values. And here
4:50:47the proxy status is proxied. This is
4:50:49correct. This is what we want to see.
4:50:51Email traffic uh for the MX and TX
4:50:56records. Here you want them to be DNS
4:50:59only. We don't want we don't want
4:51:01Cloudflare to proxy them because
4:51:04Cloudflare does not proxy email traffic.
4:51:08If you do this, your email will stop
4:51:10working. So, please keep it as DNS only.
4:51:14And if you have any NS records, in my
4:51:18case, I don't have them. You need to
4:51:19delete them. You will have this delete
4:51:21button. Just click on it and delete
4:51:23those records. Uh I can show you how
4:51:25this looks in the documentation. Click
4:51:28on the book folder. Come here to uh the
4:51:32fourth module. Search for NS. And yeah,
4:51:37here is the page. You can see that here
4:51:39if you see these NS records, just click
4:51:42on the delete button to remove them.
4:51:45Okay, now we can go back. I'll click on
4:51:47continue to activation. And if you are
4:51:50wondering, there is here a warning that
4:51:53tells you this host name is not covered
4:51:55by a certificate. This is not a problem.
4:51:57We created a certificate in the previous
4:52:00tutorial. We used Let's Encrypt for
4:52:03that. So yeah the and as you can see the
4:52:06website is secure we have um as you can
4:52:08see it's verified by let's encrypt just
4:52:11click on continue to activation and at
4:52:14this step we need to connect to porkban
4:52:18because we need to uh update some stuff
4:52:20I'll I'll log into pork and I will come
4:52:23back I am in the domain management page
4:52:27here as you can see we have uh the my
4:52:30new domain I'll click on ns by default
4:52:33We have a few name servers. We need to
4:52:36replace these by the the ones that
4:52:39Cloudflare has given us. Just click on
4:52:42this link. It's going to copy the name
4:52:46servers. Remove these depot ones and
4:52:50start pasting the name servers that uh
4:52:54Cloudflare has given you. And you can
4:52:56follow the steps. You can read here. So
4:52:58they tell you to log into the DNS
4:53:00provider. In my case, it's Porkban. And
4:53:02after that they tell you to find the
4:53:04name servers section. Here it is. And
4:53:08after that add these and delete the
4:53:09other uh name servers. I have done that.
4:53:12Now I just need to click on save name
4:53:14servers and hit okay. Now return to
4:53:18Cloudflare and click on I updated my
4:53:22name servers. In this page click on
4:53:25check name servers now to see if the
4:53:28name servers have been propagated. Uh if
4:53:31you read the text, they tell you that
4:53:32this uh takes around maybe an hour or
4:53:35two or sometimes even a day. But if you
4:53:38want, you can click on check name
4:53:40servers now. And uh as you can see,
4:53:44Cloudflare is saying that they are
4:53:45checking the name servers for my domain.
4:53:48So they are telling me to wait a few
4:53:50hours. But that's not a problem. We can
4:53:52continue the configuration. We will come
4:53:54back and uh verify that everything is
4:53:57proxied through Cloudflare. Now on this
4:54:01left panel we can click on SSL uh /TLS
4:54:06click on overview and here make sure
4:54:08that your domain is selected here I have
4:54:12two uh I will select the the new one
4:54:14that we added and here we need to set
4:54:16this to full strict uh hit confirm. Why
4:54:21do we need to select this instead of
4:54:24full? Because as I said in the previous
4:54:26tutorial, we used let's encrypt to
4:54:30secure uh our website and uh we saw that
4:54:34when we were add when we were verifying
4:54:36the DNS uh Cloudflare was giving us that
4:54:39warning because they don't know if we
4:54:42have a a certificate and uh by choosing
4:54:45full strict cloud player will know that
4:54:47we have our own certificates and will
4:54:50accept it. Now I want to show you a
4:54:53utility called dig. Uh you can search
4:54:57for it. It's it's called as I said dig
4:55:00and you can read about it uh here on
4:55:03Wikipedia.
4:55:04It's a utility that we can use to verify
4:55:08if the internet is seeing the new name
4:55:12servers that we got from Cloudflare.
4:55:14First make sure that it is installed by
4:55:17typing dig-v. It's going to give you the
4:55:20version. If you don't have the utility,
4:55:23you can install it like this on Ubuntu
4:55:26by installing DNS utils and that will
4:55:29give you access to this utility. To use
4:55:32it, you type big plus short. We want to
4:55:36verify the name servers. And here uh I
4:55:39will put my domain name. As you can see,
4:55:41it is showing me the these four that are
4:55:44coming from Porkbun, which means that
4:55:47the new name servers haven't propagated
4:55:50yet. So, we will have to wait for a few
4:55:53hopefully for a few hours and not a full
4:55:55day. You can also go back to Cloudflare,
4:55:59go to overview. Here you should see that
4:56:03this message that tells you that your
4:56:05domain is now protected by Cloudflare.
4:56:08That this is what we want to see.
4:56:10And if you made any mistake, you can
4:56:13always go back here. For example, if you
4:56:15want to update the DNS records, just
4:56:19click there or you can find it here on
4:56:21the left panel. And here are the
4:56:23records. You can see we have all the
4:56:25records and also the name servers. You
4:56:28can also use another website called
4:56:30what's mydns.net.
4:56:33We can do the same thing here. I can put
4:56:36my name uh I can put my domain name
4:56:39here. Here I will select NS because this
4:56:41is what I want to verify. And if you hit
4:56:44enter, as you can see, it is starting to
4:56:47propagate. There are some locations
4:56:50where the Cloudflare name servers
4:56:53haven't propagated yet. But yeah, the it
4:56:56it will take a few hours for uh every
4:57:00location to see the new name servers.
4:57:02Let's go back to dig to see what we get.
4:57:05We are still getting uh yeah we are
4:57:08still getting the old name servers but
4:57:10yeah it's going to uh it's going to show
4:57:12up uh in a few hours with this website.
4:57:16We were we are able to confirm that uh
4:57:19the new cloud player name servers are
4:57:21starting to propagate. Let's go back to
4:57:25this website. I will take a screenshot
4:57:29for this previous test and uh I will
4:57:33rerun it again and let's see if uh
4:57:36things have changed or not. Okay. Uh we
4:57:40see some improvements. Uh obviously uh
4:57:42yeah uh the DN the name servers haven't
4:57:45propagated fully but we should be able
4:57:48to see some uh differences. I'll take
4:57:51another screenshot and I'll put the two
4:57:54images next to each other. On the right
4:57:56side we have the before
4:57:59and here this is after. As you can see
4:58:02let's start with Beijing. Before it was
4:58:045 seconds now it is 3.4 seconds and this
4:58:07will keep improving. Um maybe the first
4:58:11time you try to load the website that
4:58:13will take maybe uh let's say 3 seconds
4:58:16but after that the files are going to be
4:58:18cached which means that they are going
4:58:19to load very fast and uh since uh the
4:58:22name servers are still propagating
4:58:25I will maybe I'll try to rerun this uh
4:58:28afterwards uh and just before ending
4:58:31this tutorial and we'll see if things
4:58:33will improve or not. But you can see
4:58:34that there are some locations that
4:58:36improved uh drastically. Uh especially
4:58:39Beijing. Uh let's look at maybe Hong
4:58:43Kong. So Hong Kong is now it takes just
4:58:461 second. Soul takes 0.9 seconds. Okay,
4:58:50good. We see some improvements which is
4:58:52a great thing to see. Well, I will come
4:58:54back to this website later. I will leave
4:58:56it here. And now let's go back to the
4:58:59slides. This is uh what I observed on my
4:59:03personal website. You can see that uh I
4:59:06used the same tool and here I I see big
4:59:10changes because I let the name servers
4:59:13propagate for maybe a day before running
4:59:16the test again. For Tokyo, as you can
4:59:19see, we were able to reduce the time
4:59:22from 1.3 seconds to 0.3.
4:59:25So again, same situation. Jakarta, same.
4:59:29and Beijing it went from 4 seconds to
4:59:311.5 seconds. This is the benefit of
4:59:34using a CDN especially if your website
4:59:37is
4:59:38doesn't even if it has a back back end
4:59:41uh attached to it there is no problem
4:59:44your front end will load quickly but
4:59:47because we introduced cloud player we
4:59:50now have a problem I have created this
4:59:54illustration to explain you the issue
4:59:57let's say that we have a user this is
5:00:00their IP address now if let's say that
5:00:03they want to check my homepage, the
5:00:06request will go through Cloudflare with
5:00:09the IP of that user. That's not a
5:00:12problem. But when Cloudflare will try to
5:00:16send that request to my application,
5:00:19it's going to send it to EngineX. But
5:00:21the problem is that we are not sending
5:00:23the user's IP. Instead, we send
5:00:26Cloudflare's IP address and this will be
5:00:30sent to the back end. And uh that's
5:00:32problematic if you want to do I don't
5:00:35know if you want to track uh IP
5:00:38addresses. This is bad because let's say
5:00:40that you you received 100 requests from
5:00:44let's say 50 users your backend will see
5:00:47the same IP address and this is the
5:00:50problem that we will try to fix. And
5:00:52luckily, if you read the Cloudflare
5:00:54documentation, they tell you that you
5:00:57need to add CF connecting IP to your
5:01:00EngineX configuration. And that will
5:01:03allow you to unwrap the IP to get uh
5:01:07yeah to get the user's original IP
5:01:10address. And now your backend receive
5:01:12the correct IP and you can store that in
5:01:15your in a database or uh yeah uh use it
5:01:18in your analytics or something like
5:01:20that. Let me show you how we do that. I
5:01:23have connected to the machine. Let's
5:01:26open this file. Let me type my password.
5:01:30And here I want you to copy this from
5:01:33the GitHub repository and put it here.
5:01:36Let me show you where you can get that.
5:01:39Here is where to get this code block. It
5:01:42is available in uh my GitHub repository.
5:01:45Let me go back here. If I scroll down,
5:01:48you can see that we have this line which
5:01:51uses this header to get the real user
5:01:54IP. And don't worry about this. If you
5:01:56want, you can find the link to the
5:01:59documentation. Just click on this that
5:02:02will take you to this page and you can
5:02:04read about this header in detail. Here
5:02:08we have the Cloudflare IP addresses. And
5:02:12here I added a note that shows you these
5:02:16IP addresses and they you can also see
5:02:19the dates that maybe if they change them
5:02:21or if they added more you can just come
5:02:23to this page, copy them and put them in
5:02:27this configuration file. Good. Now we
5:02:30can save the file. Exit. This file will
5:02:33be loaded automatically. So there is no
5:02:36need to go to our website configuration
5:02:39and uh include this. Now we need to
5:02:41verify the syntax to ensure that
5:02:43everything is correct. Good. And now we
5:02:47need to reload uh engine X. But before I
5:02:50do that, I just want to go to the
5:02:52website because now we are using uh
5:02:55Cloudflare. I want to show you the
5:02:57situation before and after. We already
5:03:00we have the logs. So we can check them
5:03:02to see the IP address that we are
5:03:04getting. Let's check the access logs. I
5:03:08want I will do tail instead of cat and
5:03:12let's see what we are getting. This is
5:03:15the the IP that I was talking about. We
5:03:17can see them here. Uh let me go to the
5:03:21website again just to make sure that we
5:03:23are receiving Cloudflare's IP addresses.
5:03:27Okay, I have visited the web page. I am
5:03:29using tail-fs
5:03:33and let's start refreshing the page. You
5:03:34can see that here we get we are using uh
5:03:37some I uh we are receiving some IP
5:03:40addresses. We confirmed that we are
5:03:42receiving one IP address which is coming
5:03:45from Cloudflare. We might receive
5:03:47different ones but all of them are
5:03:48coming from Cloudflare because here we
5:03:50have uh plenty of uh IP ranges. Now
5:03:55let's stop this and let's reload X.
5:03:58Okay, I've reloaded. Now uh let's let's
5:04:01watch and let's see what we get after uh
5:04:05refreshing the page. Aha. As you can
5:04:07see, I am getting my own IP address now
5:04:10instead of the server's IP. And I can
5:04:14refresh. And uh yeah, as you can see, I
5:04:17am getting my IP address. And I'm I'm
5:04:19also receiving other IP addresses. These
5:04:22are coming from either bots, probably
5:04:25bots because no one knows my website and
5:04:28not real humans. Uh these are just
5:04:30automatic bots that are uh targeting
5:04:33your website because they are trying to
5:04:35see if they can get some credentials
5:04:39from your website. I can show you this
5:04:40if we scroll uh let me use maybe cat or
5:04:44something to see the full uh log. I can
5:04:48also take this to grab and search for
5:04:51env. As you can see, this person or
5:04:55yeah, this boss is trying to see if they
5:04:58can find some env files on my uh web
5:05:02server. Um I mean even if they they try
5:05:05they won't be able to get anything
5:05:06because we are not serving those files
5:05:08when we created our engineext
5:05:10configuration. But you need to be aware
5:05:12that if your server is public, it's
5:05:15going to be hammered by bots. Awesome.
5:05:18Now the thing that we need to do is um
5:05:21to start working on caching. When you
5:05:25develop a front- end application, there
5:05:28are some pages that are not going to
5:05:31change. Those can be cached. But the
5:05:34pages uh that are dynamic, we should not
5:05:38change them. For example, if I change my
5:05:41index.html,
5:05:43this is a single page application. If I
5:05:45add another feature, let's say to one of
5:05:47my blogs, I want to I want users to be
5:05:51able to see it. If I cache index.html,
5:05:55they won't be showing they won't be
5:05:57receiving the new updates because
5:06:00browsers they are the they are designed
5:06:02to cache resources so that they load
5:06:05fast for users. But sometimes if this is
5:06:08not implemented correctly uh they might
5:06:11not receive uh new updates from you and
5:06:15this is what we are going to focus on.
5:06:17Uh now let's open my engineext
5:06:20configuration and here we need to start
5:06:23adding some caching. We have a single
5:06:28page application. I'm going to divide
5:06:30the caching strategy into two parts. The
5:06:34first one we will uh we will never cache
5:06:37index.html
5:06:39because the whole application is
5:06:41injected there and if uh something new
5:06:45was added to the front end it's not
5:06:48going to show up if we cache index.html
5:06:51but assets like images logos those can
5:06:54be cached forever. So let's start uh
5:06:57applying this. Let me show you how we do
5:07:00this. I will come here and I will add
5:07:04another location block. This block will
5:07:08tell the browser to always communicate
5:07:11with the server to see if there is a new
5:07:15version of index.html.
5:07:18If if we have one then the browser will
5:07:22try to load it. And here in the caching
5:07:25strategy we are using no cache. uh so
5:07:29that we tell the browser to not store it
5:07:32and instead try to communicate with the
5:07:36server every time uh this file is
5:07:39requested. And as you can see here we
5:07:41are using equal because we want to focus
5:07:44just on this specific file. We can also
5:07:48add this location block to cache assets
5:07:52forever. Here we are the the assets are
5:07:55located in the b folder. Here the
5:07:59caching strategy is now we are setting
5:08:02it to public. Max age is set to this
5:08:06very big value. Uh it is in seconds. If
5:08:10you convert it to years it's going to
5:08:12give you exactly one year. And here we
5:08:15are using immutable. This is an
5:08:18instruction that tells the browser that
5:08:20this file will never change. Let's save
5:08:24the file. Let's exit. Uh now we need to
5:08:29verify that the syntax is okay. Good.
5:08:32And we can reload. Let's go to the
5:08:36browser. Let's refresh the page. Just
5:08:38ensure that everything is uh working
5:08:41without any issue. Okay, great. Now what
5:08:44I want you to do is to open the network
5:08:48tab. I will zoom in a little bit here.
5:08:50Select all or you can select the static
5:08:53files. Refresh the page and let's see
5:08:56what we have here. I'll select HTML, CSS
5:08:59and JavaScript. If you look at this
5:09:02column transferred, you see that the
5:09:04browser is telling you that most of
5:09:07these static files were cached. That
5:09:10makes sense because we have added the
5:09:12the caching strategy but the root file
5:09:16which is uh the index.html HTML we
5:09:20always uh we are always communicating
5:09:23with the server to get the latest
5:09:26version. And as you can see here in the
5:09:28cache control we uh it is set to no
5:09:31cache which means that the configuration
5:09:34that we added to engineext uh has been
5:09:36loaded uh successfully and in the
5:09:39response we get uh index.thml the HTML
5:09:42and as I mentioned in Vue or other
5:09:45frameworks we inject uh the application
5:09:48inside this div with the id app that's
5:09:52why we need uh it's very important to
5:09:54not cache this file amazing this worked
5:09:57now I want uh I want to mention the pros
5:10:01and cons of using a CDN like Cloudflare
5:10:06or yeah other CDN's uh the benefit here
5:10:10is speed. We saw that before we weren't
5:10:15able I mean for users in China they
5:10:18needed to wait 5 seconds to load the
5:10:22website and my website is not heavy.
5:10:25Imagine if the website was uh very huge
5:10:28or not optimized that 5 seconds uh would
5:10:33have turned to maybe 7 seconds or or
5:10:36more. Um but now thankfully with the
5:10:39help of Cloudflare we were able to bring
5:10:42that down from 5 seconds to 1.5 seconds
5:10:46which is amazing. And as you can see the
5:10:48the numbers have changed because
5:10:50probably the name servers uh have
5:10:54propagated that's why we see this
5:10:56dramatic shift. Uh yeah, as you can see
5:11:00even here uh when trying to run the dick
5:11:03command I no longer see the pork bun
5:11:05name servers instead I am seeing the
5:11:08cloud player name servers amazing this
5:11:11didn't take even an hour and we can see
5:11:14a huge boost in uh performance that this
5:11:17is very amazing another benefit is
5:11:20security even uh even though we are
5:11:23using the free plan we are able to uh
5:11:28get some security. Um, CloudFlare will
5:11:31protect your website from DDoS attacks
5:11:34and other types of attacks. It's going
5:11:36to block AI crawlers from scraping your
5:11:40website uh which is amazing. But the
5:11:42problem is uh this adds complexity. We
5:11:46have introduced another step before
5:11:49reaching uh our application. And if uh
5:11:53let's say that someday a uh a problem
5:11:56happened and your application is not
5:11:58accessible you have to verify everything
5:12:01and um yeah because you don't know if
5:12:03the problem is in cloudflare or is uh in
5:12:07your application and sometimes if cloud
5:12:10if there is an outage in cloudflare your
5:12:12application will not be accessible and
5:12:14this happened many times uh in the
5:12:17recent months uh many people who relied
5:12:21on Cloudflare suddenly couldn't uh or
5:12:25users couldn't re reach their
5:12:27applications. So these are the pros and
5:12:30cons of using this. But I mean I would
5:12:34say the main benefit is that you won't
5:12:36pay a lot of money to host your
5:12:38application in uh many web uh on many
5:12:42web servers scattered around the world.
5:12:45Your website is now globally
5:12:47distributed,
5:12:49passed, and protected by Cloudflare's
5:12:52network. You have successfully finished
5:12:55the performance setup. However, before
5:12:58we move on to automating your
5:13:01deployments, we need to ensure that your
5:13:05server is completely locked down against
5:13:07hackers and automated scanners. As you
5:13:10saw in the engineext logs, we saw that
5:13:12there are many bots who are trying to
5:13:15fetch secure files. So we need to ensure
5:13:19that everything is secure so that we can
5:13:21rest easy and don't worry about
5:13:24something being stolen from your web
5:13:26server. In the next chapter, we are
5:13:28going to talk about vulnerability
5:13:31scanning. I will show you how to test
5:13:34your own defenses. I'm going to show you
5:13:37how to update the engineext
5:13:39configuration to block access to files
5:13:43like envit
5:13:46something like that so that bots
5:13:48automated bots cannot reach them. I will
5:13:51show you how to scan uh your web server
5:13:55for open ports using a tool called end
5:13:57mapap. This will uh with this we can uh
5:14:00verify that only the minimum amount of
5:14:04ports are open. Uh we want ports for
5:14:08HTTP, HTTPS and SSH. These are the only
5:14:11ports that we want to open to the entire
5:14:14internet. But the other ports like uh
5:14:18your applications back end or me search
5:14:21or other services services those needs
5:14:23to be uh closed and if we want to use
5:14:26them we need to use SSH tunneling like
5:14:29we did with the previous tutorials. This
5:14:31is what we are going to focus on uh in
5:14:34the next tutorial. See you next time.
5:14:37Welcome back. In this tutorial we are
5:14:40going to focus on vulnerability
5:14:43scanning. You are going to learn about a
5:14:46concept called soft 404. We are going to
5:14:50fix it because this is present in our
5:14:52application. You are going to configure
5:14:54engine X to block malicious bots that
5:14:57are scanning your website 24/7.
5:15:01And finally, you will use a tool called
5:15:03N MAPAP to scan your firewall from the
5:15:07outside. Let's talk about this trap that
5:15:10we call soft 404s. I have this diagram
5:15:14that explains it. Here we don't have a I
5:15:17mean we could have a regular user but
5:15:19mostly these are bots. They will try to
5:15:22get files that contains um valuable
5:15:26information like your env files. So this
5:15:31request will go through cloudflare and
5:15:33will reach your application. Now,
5:15:36engineext doesn't know about these files
5:15:38because the location blocks that we
5:15:40added are serving just our back end and
5:15:44the built files uh for view. So, env is
5:15:48not known. So, engineext will just fall
5:15:51back to serving index.html
5:15:54and this query or this request will
5:15:57succeed. But this is a false success
5:16:00because the intent of the bot is to get
5:16:04not index.html.
5:16:06Why this is a false success? Because the
5:16:08bot will and will think that it was able
5:16:12to get thev file. So it's going to
5:16:15always come to your web server and it
5:16:18will hammer it day and night. It's going
5:16:20to keep asking it for the same file. But
5:16:22if it gets A44 not found, it's going to
5:16:27skip this. And this is useful for
5:16:30malicious bots and also crawlers like
5:16:33Google's crawlers, for example. If they
5:16:35see that this path doesn't exist on your
5:16:37website, it's going to stop uh accessing
5:16:40it. And this this is b this is good for
5:16:43you and for them because you will uh you
5:16:46will not start wasting bandwidth on
5:16:49requests that are uh going to give 404
5:16:53and uh in this case bot will not will
5:16:57see a signal that will tell them you are
5:16:59not going to get this file anyway. So
5:17:00they are going to back up. This is the
5:17:03trap that we are uh that we have right
5:17:06now in our application and we can test
5:17:08it here. Uh this is uh the new domain. I
5:17:12can come here. So we can see that we
5:17:14have just slash that will go to
5:17:16index.html but I can add dot /.Nv. As
5:17:20you can see uh the request was
5:17:22successful. I was a I am still on the
5:17:25website. I don't see any error and my
5:17:28application uh I configured my
5:17:30application to load a 404 page because
5:17:34yeah this is not known. there is no
5:17:35route that will there is no page that
5:17:37will uh be loaded after accessing this
5:17:40URL. So I have a custom page but this
5:17:44gives 200 as a uh HTTP status code. Uh
5:17:49but we don't want to do this. Now if I
5:17:51go to my production server and type the
5:17:56same query I am I will get 404 not found
5:17:59because I am blocking these uh most of
5:18:02these files that uh automated boss try
5:18:05to check and now the response is 404
5:18:08it's not 200. If I go here and click on
5:18:13this request we get 404. If we go back
5:18:17to this website where we have the soft
5:18:19404 problem, you can see that here we
5:18:22are receiving 200. Okay, let's see how
5:18:26to solve this problem. Again, you can go
5:18:29back to my GitHub repository. You have
5:18:32everything in this file. If you scroll
5:18:34down a little bit, you have the
5:18:36vulnerability scanning section. And
5:18:40here, this is what we need to add. You
5:18:43can see that here we are blocking a lot
5:18:45of files um so that we are blocking
5:18:48mainly anything that starts with dot we
5:18:51have SQL all yeah there are plenty of
5:18:54things that that we don't want to uh
5:18:57serve. I will copy this and I will paste
5:18:59it uh in uh my engineext configuration.
5:19:03Now let's go to my server. I am still
5:19:06logged in. I will use nano and let's add
5:19:10this uh you can add it u here or below
5:19:14it doesn't matter let me explain to you
5:19:17the second location block here we are
5:19:20blocking uh a lot of stuff even though
5:19:22we are not using PHP in this application
5:19:26you if you inspect the logs you will see
5:19:28that they are trying to access some
5:19:30admin panels PHP files stuff like that
5:19:34so I am just I just have this
5:19:36application block that will check all
5:19:38these um file extensions and uh will
5:19:41block them. Here I as you can see I am
5:19:44returning 404 so that I can display that
5:19:47page and access log and log not found
5:19:51are set to off because I don't want to
5:19:53store um yeah these failed requests uh
5:19:57so that I don't consume a lot of disk
5:19:59space because as I mentioned you will be
5:20:02receiving thousands of requests per day
5:20:05just um from bots that are trying to
5:20:08access these files and here we have
5:20:10something called well-known phone. Uh
5:20:12you can go to this page on Wikipedia and
5:20:16read more about them. Uh this is
5:20:18important. We don't need to block this.
5:20:22If you do, search bot will not be able
5:20:25to verify your domain and your SSL
5:20:28certificates. And if that happens, um
5:20:32yeah, uh s will not be able to renew
5:20:35your certificates and that will break
5:20:38after 90 days. we are so here again we
5:20:42are using uh an expression that will
5:20:44capture most files like envit stuff like
5:20:49that but uh we have one exception which
5:20:52is the the wellknown uh directory we
5:20:56don't want uh we don't want to block
5:20:59that we want to keep it for all files we
5:21:02are going to deny that request and again
5:21:04we are going to return 404 and we don't
5:21:06want to add anything or yeah we don't
5:21:09want to add noise to the log files.
5:21:11Okay, now let's save the file. Let's
5:21:14verify that the configuration is
5:21:16correct. Yes, it is. And let's reload.
5:21:20Now, let's go back to the browser.
5:21:23Let's see what we have here. Okay, so
5:21:26remember this is my new domain. This is
5:21:28the response that I re that I got before
5:21:31this update to the configuration. Now,
5:21:35let's refresh the page. And voila, we
5:21:39get 404. Now I cannot get um
5:21:44or what were the other stuff. Uh yeah,
5:21:47let's uh check the logs. So they were
5:21:50getting some PHP stuff. I'm not sure
5:21:53what about uh what they were trying to
5:21:55get. Yeah, you can see that they are
5:21:57trying to go to some admin thing. Uh
5:21:59yeah, I'm not familiar with PHP, but we
5:22:02can uh take this and see if that will
5:22:05get blocked or not.
5:22:07put it here and yeah that worked because
5:22:10we have PHP at the end and um yeah we
5:22:14can also I think we are also capturing
5:22:16some Python stuff let's see mainph
5:22:20that also gets blocked but let's try
5:22:23with Python yeah that works amazing so
5:22:26yeah the list can be can be big uh I
5:22:29just try to you can analyze the the logs
5:22:33to see uh the files that the bots are
5:22:37trying to access or you could search
5:22:39online. There are many blacklists that
5:22:42you can find to add to your engineext
5:22:44configuration. If you want, you can uh
5:22:47use one of these GitHub repositories to
5:22:51uh block uh bad bots. And here uh so
5:22:55yeah, this is a repository. They give
5:22:58you some text files that you can just
5:23:00put on your engineext configuration. Uh
5:23:03so yeah they had a few stuff here that
5:23:06uh that you can just include in your
5:23:08configuration. You can read more about
5:23:10this and you can also uh come here on
5:23:13this other repository and use it. They
5:23:16have plenty of uh files here that you
5:23:18can check. But this is something that
5:23:21you can also configure yourself just uh
5:23:24by reading the yeah by reading the logs.
5:23:27You can also use LLMs. they will be very
5:23:30helpful in this uh step. You can explain
5:23:33to them your stack so that they don't so
5:23:37that they can help you uh block
5:23:39everything but u yeah don't touch some
5:23:43some things that will make your
5:23:45application not load or break. Now we
5:23:48have one more thing that I want you to
5:23:50learn and that will involve this tool N
5:23:55map the network mapper. It's a free
5:23:58utility that we can install uh on Ubuntu
5:24:02and other Linux dros. You just run um
5:24:07sudo app install end mapap and you will
5:24:10get it. Uh I already have it so there is
5:24:12no need to download it. And uh let's
5:24:15verify the version. We have 7.95.
5:24:19That's good. And to use it, we are going
5:24:22to run this command here. I will I'll
5:24:26type my domain name. And let's see what
5:24:29we have. Now end mapap is going to
5:24:32start. The f flag will run a fast scan.
5:24:36And this is going to check just 100
5:24:39ports. And uh one one thing that I want
5:24:43to mention is that end mapap you should
5:24:45run this command on your local computer
5:24:48uh not uh on the on the virtual machine.
5:24:52Uh you can see that I have opened
5:24:54another terminal tab and let's see what
5:24:56we have here. So we have 97 and here we
5:25:00have another three ports. The the sum
5:25:02gives us 100 which confirms the theory
5:25:05um or what I explained about this F
5:25:08flag. Now let's analyze the the output.
5:25:10You can see that here uh we have not
5:25:13shown 97. These are other ports that are
5:25:17not available or that are not accessible
5:25:19from the out from the basically the
5:25:22public internet. So that's that's a good
5:25:25sign. And after that we have three ports
5:25:28which are this is what I explained uh in
5:25:30the beginning. We want to allow just
5:25:33HTTP, HTTPS and SSH. We don't want to
5:25:37open port 8080 or 7,700. These are ports
5:25:41that we are using uh in that I am using
5:25:44in my applications uh to basically
5:25:47access M search fast API and also uh
5:25:52view but those are running uh but they
5:25:55are p uh they are they're not accessible
5:25:58from uh the outside and uh as I showed
5:26:01you in the previous tutorials if you
5:26:04want to access those you don't open a
5:26:07new hole in the firewall instead Use SSH
5:26:11tunneling to open a temporary tunnel
5:26:14between your local setup and the virtual
5:26:17machine. Do your work and then close
5:26:20that uh tunnel. At this point, you were
5:26:24able to deploy your application
5:26:27manually. You have u learned to how to
5:26:31deal with uh bad actors that are trying
5:26:34to access your website. you learn how to
5:26:36optimize the load time and plenty of
5:26:39other skills that are going to be very
5:26:42um beneficial.
5:26:44We saw you saw that we are using digital
5:26:47ocean but you can take the concept that
5:26:49you have learned here and use those with
5:26:52other cloud providers. But this is the
5:26:55first step. Now we are going to enter
5:26:58another phase where we are going to
5:27:00automate a lot of stuff. We are going to
5:27:04create an automation pipeline that will
5:27:07allow you to make changes to your
5:27:10application locally. Maybe develop a new
5:27:13feature uh fix a security problem and
5:27:17instead of uh taking the source code,
5:27:21copying it from your local computer to
5:27:24the virtual machine, do the build
5:27:26process and um install the dependencies
5:27:30and then serve your application. We you
5:27:33are going to automate this process. You
5:27:35are going to learn about CICD which
5:27:39stands for continuous integration and
5:27:42delivery which will allow you to click
5:27:44on a few buttons and those buttons will
5:27:48run jobs uh on your behalf. Those jobs
5:27:52will check for security problems. They
5:27:55will check if uh dep the dependencies
5:27:58that you are relying on are secure. They
5:28:00will build your front end. They will
5:28:03shut your backend application, reload
5:28:06the new changes if there are any and um
5:28:10other other kinds of job and you will
5:28:12just sit there and watch uh your new
5:28:15changes uh live become live on your
5:28:18website and that will take you a few
5:28:21minutes to do instead of hours of manual
5:28:24labor. In the in that next module, we
5:28:27are going to start slow. In the first
5:28:30sub chapter, I will show you how to put
5:28:33this code on GitHub. I assume that you
5:28:36already have one or you can also use
5:28:38other websites like GitLab or uh
5:28:42Codeberg or yeah any any g compatible
5:28:45client. Uh we'll show you how to protect
5:28:49your main branch so that you don't push
5:28:52code accidentally to it. We need to run
5:28:55that pipeline to ensure that everything
5:28:56is secure, everything is correct before
5:28:59merging it to main. And after merging, I
5:29:01will show you how to do the deployment.
5:29:04But that will come uh in the next sub
5:29:06chapters. So here we just want to focus
5:29:08on the code quality. I'll show you how
5:29:10to use the rough library because here we
5:29:13are using Python to uh standardize to
5:29:17run linting and formatting in order to
5:29:19standardize the code that we write. And
5:29:23on the front end side you will use
5:29:25eslint which is the the default one.
5:29:28Okay we will uh focus on formatting the
5:29:31code and catching linting errors in the
5:29:33next uh tutorial. See you next time.
5:29:37Welcome to module five titled the
5:29:40automation pipeline. In the previous
5:29:43modules you were able to deploy the web
5:29:47application manually. We have started by
5:29:50copying the source code from your local
5:29:54machine to the VM. You have been able to
5:29:58install the dependencies, build the
5:30:00front end and other stuff. This is
5:30:02great. It's important to do things
5:30:05manually, but you should not continue
5:30:07like this because if you add features to
5:30:10your application, you want to do the
5:30:12deployment automatically. you you want
5:30:15to focus on writing the code and let the
5:30:18this automation that we are going to
5:30:20build in this module do the work for
5:30:22you. Here is what we are going to talk
5:30:25about in this sub chapter. First I will
5:30:28show you how to protect the main branch
5:30:31using GitHub rule sets. In this tutorial
5:30:35I am using GitHub but you can do the
5:30:37same thing if you are using GitLab or
5:30:40other uh websites. I will show you how
5:30:43to automate local checks using
5:30:46pre-commit. And for Python, we are going
5:30:49to use the rough project to do linting
5:30:52and formatting. And for JavaScript, we
5:30:55will be using ESLint with prettier. If
5:30:58you use other other programming
5:31:00languages, look for the projects that
5:31:03are there and use them. The concepts are
5:31:05the same, but we are using different
5:31:07technologies.
5:31:08This is the repository that I was
5:31:11talking about a lot during the previous
5:31:13modules. Here you can find everything
5:31:15related to the to the book that I am
5:31:17writing and these the slides everything
5:31:20is inside this repository and I want to
5:31:23show you this is my this is my other
5:31:26GitHub repository where I host my where
5:31:29the source code of my personal website
5:31:31and I want to show you what will what we
5:31:34will achieve by the end of this module.
5:31:36You can see that here is the automation
5:31:39that I was talking about. You have
5:31:41certain jobs that you run here on
5:31:44GitHub. They run whenever you push code
5:31:48and they do a lot of things for you. For
5:31:50example, here this job will check my
5:31:53dependencies. It will look at the Python
5:31:55libraries. If it sees that there is some
5:31:59security vulnerabilities, it's going to
5:32:01stop the code that I pushed and it's not
5:32:03going to be merged to the main branch.
5:32:06This will protect you from deploying
5:32:09code that is not secure. Uh here we are
5:32:12performing linting and formatting. In
5:32:15this video we are going to do this
5:32:16locally but we can also do it here on
5:32:19GitHub actions. You can run tests. We
5:32:23have different kind of tests here. Here
5:32:25you can see that we have unit tests,
5:32:27integration tests and end to end test.
5:32:30Finally, if everything goes well, we
5:32:33will build the front end and the
5:32:35pipeline succeeds. And we we can add
5:32:37other actions that will allow us to
5:32:40build the front end and deploy it. For
5:32:43example, if I look at this one here, as
5:32:46you can see, I have uh plenty of jobs or
5:32:50workflows. And here I have this one,
5:32:52manual deploy. after merging a pull
5:32:55request to the to the main branch. I
5:32:58come here and run the manual deploy
5:33:01because here as you can see we just
5:33:03built the front end because we are 100%
5:33:06sure that there is no problem with the
5:33:07code because it uh it passed all those
5:33:10checks. Now I come here I run the job
5:33:13and here it notifies me sends an email
5:33:15telling me hey do you want to deploy? I
5:33:18just click on a button to approve that.
5:33:21And as you can see it says deploy and I
5:33:23can check the website. This is what we
5:33:26want to do. And we don't need to SSH to
5:33:29the machine, run commands manually.
5:33:31Everything is going to be done for us uh
5:33:34in this interface. And what we do is we
5:33:38just create a pull request. We check the
5:33:40code. And here you have the checks. So
5:33:43for example, here it says all 17 checks
5:33:46have passed. If that's the case, we
5:33:48merge. But if a check fails, it's not
5:33:51going to be it's not going to to let me
5:33:53merge until I fix it. But for now, let's
5:33:56just focus on this branch. It's brand
5:33:58new. I haven't set up anything. We are
5:34:01going to do this uh together. The first
5:34:04thing we want to do is to secure the
5:34:06main branch. I also have my code editor.
5:34:09Here it is. And as you can see, if I
5:34:12click on this icon, it shows me the
5:34:15changes that I have here. You can see I
5:34:17I can come here uh add this and push.
5:34:21Nothing is go no one is going to stop
5:34:23me. But what what if I what if I don't
5:34:26want that? Let's say that you are
5:34:27working with a team of people and you
5:34:30want to do a pull request before
5:34:32merging. In that case, we want to
5:34:34protect this so that you don't push
5:34:36domain accidentally. But to fix this, we
5:34:39come here, we click on settings. Let me
5:34:42zoom in a little bit. Here we click on
5:34:45branches and we add a branch rule set.
5:34:49Here we need to give the rule set a
5:34:51name. Let's call it main branch
5:34:54protection. Here it currently it's
5:34:57disabled. We want to enable it. Let's
5:35:00choose active. Let's scroll to the
5:35:03target branches section. Click on add
5:35:06target. Let's select include default
5:35:10branch. Let's go back to the repository.
5:35:13As you can see, my default branch is
5:35:15main, but in some cases it could be
5:35:17master. That's why here it shows default
5:35:20instead of uh main or master. And here
5:35:24we have some rules that we can apply.
5:35:26You can read through them. And here for
5:35:29example, let's scroll down to restrict
5:35:33deletions. This will prevent anyone from
5:35:36deleting the main branch. That's why we
5:35:39want to restrict it. Here we have
5:35:41another rule that we need to keep. It
5:35:43says block force pushes. This will
5:35:46prevent you from writing from using git
5:35:49push-force
5:35:51because when you use force you rewrite
5:35:53history and that can lead to problems
5:35:56here. These two are uh checks by
5:35:58default. We want to add this third one.
5:36:01Require a pull request before merging.
5:36:05If you are working with a team, required
5:36:08approval should be either one or two. It
5:36:10depends on the size of the team. But if
5:36:12you are working as a solo developer,
5:36:15select keep this at zero. I am I am
5:36:18doing it on my other project. Why?
5:36:21Because it builds the habit of uh
5:36:23creating a branch for each feature you
5:36:25work on. And if you here you have to do
5:36:28zero because you are the only one who is
5:36:30working on this project. And it is very
5:36:32helpful. is going to make you think of
5:36:36always working with making small
5:36:38changes, creating a pull request and
5:36:41merge it to main like I do with my other
5:36:44projects. We will come back to this uh
5:36:47in the upcoming sub chapters because we
5:36:50don't want to do everything in this in
5:36:51this first video, but we just want to
5:36:53get up and running. Now, let's click on
5:36:56create. Here it's going to tell you you
5:36:58need to sign in. I will do this and I'll
5:37:00come back. I am back. You can see it
5:37:03says rule set created and here it is and
5:37:06it it says applies to one target which
5:37:09is the main branch. This is what we want
5:37:12to see. Now if I go back to the
5:37:15repository you can see that this is our
5:37:18main branch. If we click on view all
5:37:20branches and here you can see that we
5:37:23have we can also view the rules from
5:37:25this button. And yeah we have three
5:37:28branch rules. That's correct. But let's
5:37:30go back to VS Code and let's see if we
5:37:32can push something to this codebase from
5:37:36main. Here is the repository. Let's make
5:37:39some change to this file. I will come
5:37:41here and let's just add an empty line.
5:37:45You can see that we we have modified the
5:37:48file. I will stage it and let's create a
5:37:51commit. Get commit- m let's call it
5:37:56test. Hit enter. Okay, that worked. But
5:37:59now let's try to push this get push. As
5:38:03you can see I was not able to do that.
5:38:06Let's see what is the error. You can see
5:38:08that here it says repository rule
5:38:10violations found on the main branch.
5:38:14Here it says changes must be made
5:38:16through a pull request. This is what we
5:38:18want to see. So remote rejected our push
5:38:22operation. Amazing. And to undo this
5:38:25commit because it was just test, you can
5:38:27run git reset add. And here we just want
5:38:31to reset the latest commit. Let's do it.
5:38:34And as you can see, uh, it was removed.
5:38:37And it shows me that this is the last
5:38:38modification I made to this file. Great,
5:38:41it's working. So, let me just undo this
5:38:43because I don't want to make that
5:38:45change.
5:38:46You have successfully protected the main
5:38:49branch. But we still need to check the
5:38:52commits that we are pushing. What do I
5:38:55mean when I say we need to check
5:38:56commits? Sometimes you might be working
5:38:59on your project and you might
5:39:01accidentally commit let's say a secret
5:39:03key. That's that's a problem because
5:39:06that will that will sit on the git
5:39:08history and everyone can see it unless
5:39:11you rotate the key. With something
5:39:14called pre-commit, we are able to catch
5:39:16this problem before pushing the commit
5:39:18to the remote repository. We can do
5:39:22other checks. We can check if the
5:39:24linting is correct, if the code is
5:39:26formatted correctly and other stuff.
5:39:29This concept is called get hooks and we
5:39:32are going to target pre-commit hook
5:39:35specifically. So when you type get
5:39:38commit, this hook will intercept uh your
5:39:42request. Basically, it's going to see
5:39:44what you are what you are committing and
5:39:46it's going to run a series of checks and
5:39:49you can add uh as many checks as you
5:39:52want. If everything passes, great. The
5:39:55request will go through and your commit
5:39:57will be deployed to the cloud or yeah to
5:40:01the cloud repository. But if something
5:40:04if a check fails the the hook will going
5:40:08to block you unless you fix the problem.
5:40:11Okay. So this is what we what we are
5:40:12going to do uh now and we are going to
5:40:15use this amazing project called
5:40:17recommit. Let's go back to VS Code to
5:40:20show you how to use this. Here you can
5:40:23see that I have some changes made to
5:40:26this get ignore. I'll try to stash that
5:40:29because I don't need it. Now I need to
5:40:33add this specific file
5:40:35pre-commit-config.yml.
5:40:39Here we are going to add the checks that
5:40:41we want to perform. Let's now start with
5:40:44the back end. You can see that here we
5:40:46don't have any Python code but we will
5:40:48add one to test this. Let's just add
5:40:51main.py and let's create a a function
5:40:56that will print hello world. We can run
5:40:59this and it should work. So if we do
5:41:03main.py,
5:41:05as you can see, we get hello world. We
5:41:07are doing this just to test this hook.
5:41:10We have two kinds of texts that we can
5:41:13use. You have remote repositories and
5:41:16you have local custom hooks that you can
5:41:19create on your own. For example, here if
5:41:22I click on this link, it takes me to
5:41:25GitHub. You can see that people who are
5:41:28working on rough have created this
5:41:31pre-commit that I can just install by
5:41:35just putting the link of the repository
5:41:38here. So this means when I run install,
5:41:41it's going to go to this repository.
5:41:44It's going to bring this version. How do
5:41:47I know if this is the latest version?
5:41:49It's simple. Just go back to the
5:41:52repository. Scroll down and here click
5:41:55on releases and that will show you the
5:41:57latest release. Currently we uh in here
5:42:01in rough the latest version is 0.15.12
5:42:05and this is exactly what I have
5:42:07specified here in this field and here
5:42:10you put your hooks what you want to run
5:42:13here I have two kind of checks I have a
5:42:17linting step and a formatting step.
5:42:20Let's focus on the first hook. You can
5:42:23see that the ID is rough and I give it a
5:42:26name and I specify that I want to
5:42:27perform LinkedIn. And here I have I am
5:42:30passing additional arguments. And here I
5:42:32pass d-fix because sometimes there are
5:42:35some some things that we can fix in the
5:42:38code. And here I am targeting everything
5:42:40inside the backend folder. So let's do
5:42:44that. Let's let's create a backend
5:42:46folder. And let's drag this file and put
5:42:49it there. Otherwise, this hook will not
5:42:52find anything. It's going it's going to
5:42:53succeed. But we want to target only u
5:42:58Python files that are inside the backend
5:43:00folder. In the second hook, we are just
5:43:02performing formatting. Uh it's simple.
5:43:05This file is is good. There is no
5:43:07problem in it. Now, if I try to use this
5:43:11hook, it's not going to fail. Let's see
5:43:13how to install this. You have different
5:43:15options of installing it. You can use
5:43:17Anaconda, you can use poetry, uv or any
5:43:21package manager in Python or you can
5:43:24simply use the ven module. Uh let's try
5:43:28let's try that. Then ven as you can see
5:43:31we have our virtual environment and
5:43:34let's activate it. Source van bin
5:43:38/activate.
5:43:40As you can see here I have van and I am
5:43:42using python 3.12 here we need to add
5:43:46dot getit ignore so that we don't push
5:43:48what we have here and let's put van here
5:43:52okay good now we have just three
5:43:53modifications now let's install precomit
5:43:56to do that make sure that the
5:43:58environment is activated and run pip
5:44:01install precomit
5:44:04great that worked and now we need to run
5:44:06precomit install you can see that it
5:44:09says pre-commit installed at this
5:44:11location. Let's controlclick on this
5:44:13file just to see what we have here. You
5:44:16can see that this file was autogenerated
5:44:18by pre-comit and here we have this
5:44:20variable install python that points to
5:44:23the virtual environment. The problem
5:44:25with this approach is that you can see
5:44:27that we have installed precomit on this
5:44:31project. So if I move if I delete this
5:44:33virtual environment, the pre-commit hook
5:44:36will fail. It's not going to find this
5:44:38path and it's not going to work. But if
5:44:41you stay here, it's going to work
5:44:43without any issue. But we have other
5:44:45options that we can try to fix this
5:44:48problem because ideally we want to
5:44:50install pre-commit globally so that we
5:44:52can use it in any Python pro in any
5:44:55projects not just Python and that way we
5:44:58will fix these issues. So using a
5:45:01virtual environment is perfectly fine.
5:45:03The only issue that you will face is
5:45:06when you decide to delete the virtual
5:45:08environment. But if you don't then you
5:45:11can keep using this or we have another
5:45:14option which is to install pre-commit
5:45:16globally with pep x. This is another
5:45:20python library that we can use and we
5:45:22have plenty of options to download it.
5:45:25Here I am using because I am on iuntu I
5:45:27am using apt to install this from the
5:45:30repository. Uh I have already installed
5:45:33it. So, it's not going to add anything,
5:45:35but I just want you to go to GitHub, you
5:45:38will find the pipex there. As you can
5:45:40see from the about section, it says
5:45:43install and run Python applications in
5:45:47isolated environments. And this is what
5:45:49we want to do. I am not using the
5:45:52regular pip because I don't want to
5:45:54install this globally because it's not
5:45:57it's a bad idea. That's why we use
5:45:59virtual environments. But PEX solved
5:46:03this issue for us. And if you scroll
5:46:05down a little bit, you can see here in
5:46:08the image, it shows you that you can
5:46:10install libraries and run them in
5:46:13isolation. And you have plenty of
5:46:15options to install this here on Linux.
5:46:18You can use pip if you want. And uh I
5:46:21decided to use apt. But I just want you
5:46:24to go here to in to look at your current
5:46:27OS to uh get the perfect command for
5:46:31your system. After installing pipex, you
5:46:34need to run pipex insure path. And
5:46:37that's that's the second command here.
5:46:40Let's do it. As you can see, it says
5:46:42that it already found the path. So
5:46:45that's fine. And if you see this
5:46:47warning, no problem. uh the the message
5:46:50that let tells us that if it is working
5:46:52is this otherwise pipex is ready to go.
5:46:55If you see this, you are good to go.
5:46:57Let's clear the page. And now we run pip
5:47:00x install pcomit. As you can see, I have
5:47:04already installed it before. That's why
5:47:06it didn't work. So it tells me that it
5:47:08seems that this library is already
5:47:11installed. So that's fine. And as you
5:47:13can see here is the location of that
5:47:17specific library. Now all we need to do
5:47:20is to go back to VS Code. You can see
5:47:24that here I have activated this virtual
5:47:26environment. Now I need to deactivate.
5:47:29Let's type deactivate. Good. The virtual
5:47:33environment is is no longer there. And
5:47:35we can also delete it because we uh I
5:47:38want to show you that this is going to
5:47:39work without a virtual environment. You
5:47:43can see we have removed that folder. And
5:47:46I can also remove the changes from git
5:47:49ignore. We need to do the same command
5:47:52pre-commit install. I'll hit enter. As
5:47:55you can see, this worked. And in this
5:47:57file, you can see that the path has
5:47:59changed. Now the path is pointing to the
5:48:03pipex environments. This is what we
5:48:05want. Now, as you can see, I don't have
5:48:07a virtual environment. That's fine
5:48:09because this is installed globally. Now,
5:48:12I can open another project, run
5:48:14pre-commit install, and it's going to
5:48:17use the same virtual environment. That's
5:48:19great. And this solves the issue with
5:48:22local virtual environments. Great. Now
5:48:24we can continue. First, we need to
5:48:27commit this file. This is the first
5:48:29operation that we need to do. So, let me
5:48:32commit this. I'll click on the commit
5:48:34button. Great. It works. The code uh I
5:48:38have I have been able to create the
5:48:39commit. Nothing has stopped me. Now,
5:48:41because we have this specific hook that
5:48:44targets Python files, if I try to commit
5:48:48this, it this these hooks are going to
5:48:51check the file. And I'm going to do this
5:48:53in the in the terminal because it's
5:48:55going to show us what is happening.
5:48:57Okay, let's do get commit dash m and
5:49:01let's say test python hooks and let's
5:49:06hit enter. As you can see the first
5:49:10check passed. So linting was good but
5:49:13formatting was not good. So here it says
5:49:17failed and we as you can see it says one
5:49:20file reformatted and the commit is not
5:49:23going to go. The file has been changed
5:49:26and as you can see here is the change
5:49:27that was made and we are using this to
5:49:30make sure that formatting is consistent.
5:49:33If you work with a team, you want the
5:49:35formatting to stay consistent because
5:49:38otherwise it's going to be very hard to
5:49:40review put requests because they are
5:49:42going to make changes with different
5:49:44formatting and and that's not a pleasant
5:49:47uh situation to be in. This is why we
5:49:49use pre-commit hooks. And as you can
5:49:51see, we were not able to push this. And
5:49:53if everyone is using the same set of
5:49:56rules, it ensures that the code uses the
5:50:00same formatting rules and everything
5:50:02stays the same. The only thing I need to
5:50:04do is to add this again to add the
5:50:08modifications. Let's try to commit. And
5:50:11voila, it worked. You can see that uh
5:50:14both checks have passed and I was able
5:50:17to create the commit without any
5:50:20problem. As you can see, I can click on
5:50:21the button to push these two commits.
5:50:24But this is not going to work because I
5:50:26am on the main branch. We can test it
5:50:29again. It's fun to test stuff like this.
5:50:32So, let's do get push. And it was
5:50:35rejected. Great. So, this confirms that
5:50:37it it's working. Awesome. So, this was
5:50:40just an example. But now let's go back
5:50:42to my other repository in order to show
5:50:45you the pre-commit hook, the pre-commit
5:50:48configuration that I have there. It's
5:50:51totally different from this one because
5:50:53we have JavaScript and Python. Here is
5:50:57my other project. And if you remember,
5:50:59this is exactly the code that I told you
5:51:02to download if you didn't have your own
5:51:05projects to work with. And here here is
5:51:08my configuration. We have plenty of
5:51:10things but you can see that the first
5:51:12part is similar to what I have showed
5:51:15you in the other project. We are using
5:51:18rough and we are we have two hooks one
5:51:22for linting and the other one for
5:51:24formatting. Let me collapse this because
5:51:26we have seen it here. I have another
5:51:29repository and this is used for
5:51:32JavaScript. Here we are using eslint
5:51:35because we want to do linting. We want
5:51:37to verify that the syntax in our
5:51:40JavaScript code is correct and that
5:51:42there are no problems. As you can see
5:51:45here, we have just one hook and uh I've
5:51:48called it yes link front end. Here I
5:51:51have files. It's going to target
5:51:53everything inside the front end folder.
5:51:57This is a regular expression that will
5:52:00target only view and JS files inside the
5:52:04front end directory. If I open source
5:52:07you can see that here view files end
5:52:10with dot view that's why I have it here
5:52:13and I also have js here I am not using
5:52:16typescript if I was using it I will
5:52:18change js to ts so the types that uh I'm
5:52:22looking for files and here I also add
5:52:25arguments so that I can fix problems if
5:52:28I find them and I also point the conf to
5:52:31the configuration file because yes lint
5:52:33can be configured and here is a
5:52:35configuration file. So here if you want
5:52:38to disable some rules, you can uh come
5:52:40here and modify the rules that uh you
5:52:44want your hook to check for uh or to
5:52:48skip in this case. And here I have
5:52:50additional dependencies. These are these
5:52:53will change. It depends on the project
5:52:55that you are working on. If you aren't
5:52:57working with Vue, you won't you won't
5:52:59add this. So yeah, these are just
5:53:01additional dependencies that will be
5:53:02installed in order to run this hook. So
5:53:05this is the front end part and we also
5:53:08have some local repositories or local
5:53:11hooks that we create on ourselves. This
5:53:14is what why I mentioned before that you
5:53:17can use either hooks created by the
5:53:20community or use your own. For example,
5:53:23here I have two local ones and here I
5:53:26will you can see that I I call this one
5:53:28prettier because with prettier I can
5:53:30check many files. I can check JS view,
5:53:34CSS and other file extensions and uh
5:53:37same if I detect that some format
5:53:40something is not formatted well I I add
5:53:43the dash dash write to change the file
5:53:46and uh after that I can just come here
5:53:48and add click on the plus button to
5:53:51accept the change and finally I have
5:53:53this other hook that will automatically
5:53:56generate the site map for my website and
5:53:59this will run whenever I commit and here
5:54:01you can see that I just have a simple
5:54:03bash script. I go inside the front end,
5:54:06run the sitemap. So this is a script
5:54:09that I have and it's going to generate
5:54:11this XML file that I will stage. Using
5:54:15pre-commit is really easy and it helps
5:54:18you catch a lot of problems locally in
5:54:22before deploying or publishing the the
5:54:25code changes. And we can also do a
5:54:28simple test here. Let's let's open app
5:54:31doview view and let me add some spaces
5:54:34some new lines here that will trigger
5:54:37prettier and it's going to block this
5:54:38operation. Let me open my terminal and
5:54:41let's check the status just to confirm
5:54:44that everything is all right. Let's add
5:54:47this. Okay, now we can write the commit
5:54:50message. Let's say uh let's hit enter.
5:54:56As you can see that works here. rough is
5:54:59not failing. It is skipped because we
5:55:01haven't changed anything uh in the back
5:55:03end but uh we have changed something in
5:55:07the front end directory. Yes, lint
5:55:09worked. That's not a problem. But
5:55:11prettier didn't like the changes that we
5:55:13have made. And because we have set
5:55:16rights, you can see that it changed the
5:55:17file. So, it removed those new lines for
5:55:20us. That's great because I can just come
5:55:22here verify the new changes and click on
5:55:25add. And yeah, I can come let me also
5:55:30you can see that sitemap was generated
5:55:32for us automatically. That's good. I can
5:55:36come back and uh run it. And as you can
5:55:38see this time it worked because we fixed
5:55:41all the issues. And that's it for this
5:55:44video. Your local environment is now
5:55:46fully secure. You have protected the
5:55:49main branch and you have created
5:55:52pre-commit hooks to check commits before
5:55:56pushing them. In the next sub chapter,
5:55:59we are going to talk about continuous
5:56:02integration. We are going to perform
5:56:04automation using GitHub actions that
5:56:07will allow us to write workflows that we
5:56:10can that we can run after let's say
5:56:14creating a pull request that will allow
5:56:16you to do something similar to what I
5:56:18have here. You can define your jobs and
5:56:23uh put them in a pipeline that will run
5:56:26uh automatically and if everything
5:56:27succeeds you can merge your pull
5:56:29request. This is going to take us a lot
5:56:32of time to develop. But I'll try to go
5:56:34uh step by step. I'll try to go slowly
5:56:37so that you can understand everything.
5:56:39Uh once you understand this, once you
5:56:41make one or two jobs, then the rest is
5:56:43going to be repetitive. You will uh then
5:56:46you will know how to add uh anything.
5:56:48And you might be asking, do I need to
5:56:51follow a specific rule? You can. No, you
5:56:54can design this however you like. Just
5:56:57make sure that you catch the things that
5:56:59you want to focus on. For example, if
5:57:01you want to do security, you need to add
5:57:03some jobs that will test if your app is
5:57:06secure. If you want to check LinkedIn
5:57:08and formatting, you need to add jobs
5:57:09that will check that. So each job is
5:57:12going to check one thing for you. And by
5:57:14building uh the these sets of jobs, you
5:57:17will ensure that uh the code that you
5:57:20push to production is secure. Uh there
5:57:23are and there are no bugs no bugs in it.
5:57:26See you in the next video. Welcome back.
5:57:29In this video we are going to talk about
5:57:32continuous integration. In the previous
5:57:35video you have seen how to perform
5:57:38checks locally to stop bugs or
5:57:42unformatted code from being synced with
5:57:46the remote repository. Today we are
5:57:49going to do the same thing but on GitHub
5:57:52or you can do the same thing on GitLab
5:57:54or other platforms. The same concept
5:57:57applies but there are some uh things
5:57:59that might change from platform to
5:58:01platform. Here is what we are going to
5:58:03talk about in this video. We are going
5:58:06to learn about continuous integration.
5:58:08You are going to build modular and
5:58:11reusable workflows to automate linting
5:58:15formatting and to perform unit tests.
5:58:19And of course, we want to block bad code
5:58:21from being merged to the main branch by
5:58:25using GitHub status checks. In this
5:58:28session, we are going to build a small
5:58:30pipeline that will do the following. You
5:58:34can see that here. This is specific to
5:58:36GitHub but you can find the same concept
5:58:39on GitLab also or other platforms. Here
5:58:42we have five jobs. You can see that we
5:58:45have two jobs that we run to test the
5:58:49back end and we have the same that but
5:58:52for the front end. The first step is to
5:58:55perform linting and formatting and after
5:58:58that we perform unit tests. And you can
5:59:01see by default GitHub will run jobs in
5:59:05parallel but some jobs can depend on
5:59:08each other. For example, before I run
5:59:11the unit tests in the back end, I need
5:59:14to make sure that there is no syntax
5:59:16error and the code is formatted
5:59:18correctly. If this is not the case, this
5:59:21job will be skipped. It's not going to
5:59:23run. The same the same goes for the
5:59:25front end. Finally, we have this
5:59:27pipeline success which is going to be
5:59:30used to allow us to merge the pull
5:59:35request or to uh block it. You can see
5:59:38that in this case everything passed that
5:59:40mean that's why I get a green check mark
5:59:43which means that I can merge this pull
5:59:46request. If let's say the lint failed
5:59:48this is going to turn red and it's going
5:59:50to block the mer the pull request. We
5:59:52will keep adding more jobs to this
5:59:54pipeline in the upcoming videos. But for
5:59:57now, let's just start with these two.
5:59:59This is what we have performed locally
6:00:01in the previous video. But on top of
6:00:03that, we are going to add unit tests. I
6:00:06am now on GitHub and I I switched to my
6:00:10secondary account. Here I have forked
6:00:13the source code that we used to deploy
6:00:17the website on digital ocean because I
6:00:20wanted to experiment here uh without any
6:00:23problem. So I have forked this and now I
6:00:25will create I've already created a pull
6:00:28request but we will start from scratch
6:00:30and I will show you how this works and
6:00:33how to build a GitHub action. First
6:00:37let's start by creating a branch. Here
6:00:40we are using VS code. I will just click
6:00:43on this button that will allow me to
6:00:47create a new branch from a specific
6:00:50branch. And here I am creating it from
6:00:52master. I will call this test number
6:00:55two. I will hit enter and I will select
6:00:58head. I will switch to that branch
6:01:01because remember the master branch is
6:01:04protected. I cannot push directly to
6:01:08that branch. If you are using GitHub,
6:01:10you start by creating agitub
6:01:14folder. You can see I already have it
6:01:16here and we have plenty of things. But
6:01:19to create a pipeline, we put our
6:01:21workflows inside the workflows folder.
6:01:25Let's open it. You can see that I have a
6:01:28lot of things. Don't worry about these.
6:01:30Don't worry about this. Uh what you are
6:01:32seeing here is the final product. Uh we
6:01:35are going to build these step by step.
6:01:38I'll go through these and remove them
6:01:40and I will keep just the ones that we
6:01:41need for this video. I am back. As you
6:01:44can see, I have removed a lot of files
6:01:47and let's start by working on linting
6:01:49and formatting in the back end. If you
6:01:51want to learn more about this, you can
6:01:54search GitHub actions. As you can see,
6:01:57you can search GitHub actions and you'll
6:01:59find you will find them here. Their
6:02:02documentation is great. So if you want
6:02:05to dive deeper, you can use this or you
6:02:08can just take the documentation and uh
6:02:12use that with your preferred large
6:02:14language model and they will help you uh
6:02:16understand the concept. But I'll try to
6:02:18also do my best to explain to you how
6:02:20this works. Our goal is to build a
6:02:24modular and robust pipeline. Which means
6:02:27that we want each job to be contained in
6:02:30its own file. And we call these reusable
6:02:34workflows. For example, here I am
6:02:37hovering over the backend lint and
6:02:39format. This is going to be one file
6:02:42which is this one. The code that we will
6:02:45put here will run at this step. And
6:02:48after that this file ci will be the
6:02:52orchestrator. It's going to call these
6:02:54uh workflows independently one after the
6:02:57other. And we don't want to put
6:02:58everything inside one giant file because
6:03:02if you get an error, it's going to be
6:03:04very hard to debug. But by putting
6:03:07everything inside a its own file, it's
6:03:11going to make life easier and you will
6:03:13see how the the orchestrator is going to
6:03:16be very minimal. It's going to be very
6:03:18easy to read and debugging is also going
6:03:20to be fun. To create a workflow, you
6:03:23start by giving it a name. Here you can
6:03:26put anything and it's going to be easier
6:03:28for you to locate. For example, if we
6:03:32let me go back to GitHub, you can see
6:03:35that here we have this actions tab. If I
6:03:38click on it, uh you can see that we have
6:03:40plenty of workflows and you can see that
6:03:42they're named uh they are named. Let's
6:03:44look at this one. You can see that it's
6:03:46called backend lint and format. Let's go
6:03:50back to GitHub and we we find it here.
6:03:53back end lint and format. You can click
6:03:56on it, you will see the how many times
6:03:58it we run it, etc. Uh, okay. And
6:04:02sometimes you can even run them manually
6:04:04if you want to. Let's look at the
6:04:05workflow or the orchestrator. You can
6:04:08see that I can come here and run it
6:04:10manually if I want to. We will I will
6:04:12show you how to implement that. But this
6:04:14is why we give them names so that it
6:04:16becomes easier for us to uh find them
6:04:19here in the actions tab. After that we
6:04:22add the onb block and here we specify
6:04:26workflow call. This tells GitHub that
6:04:30this file is a reusable template not a
6:04:34standalone script which means we cannot
6:04:35run it on its own. We need to call it
6:04:38from the orchestrator. After that we
6:04:41create a series of jobs that we want to
6:04:43run inside this specific workflow. And
6:04:47here we want to perform just lint and
6:04:49formatting. That's why we you can see
6:04:52that here we have jobs and here you can
6:04:54specify your jobs and here we are going
6:04:56to run one one specific job. We are not
6:04:59going to run more than that. Inside the
6:05:02lint and format job we can also give it
6:05:06a name and runs on is very important.
6:05:09Here you can choose different operating
6:05:11systems. Here I am using iuntu and
6:05:14latest will take the la the the recent
6:05:18version of Ubuntu. Why are we doing
6:05:20this? Because remember we are going to
6:05:22run this on the cloud. It's not going
6:05:25the these checks are going to be
6:05:27executed on GitHub's own machines.
6:05:31That's why we need to specify a an
6:05:34operating system. After that we add
6:05:37defaults run and we specify the working
6:05:40directory. We are doing this to make our
6:05:42life easier because if we don't add this
6:05:45we have to always type cd back end
6:05:49whenever if we want to perform if you
6:05:51want to perform something inside the
6:05:53backend folder we need to always do that
6:05:56but by putting this GitHub will
6:05:58automatically move us to the backend
6:06:01folder and we can run the commands
6:06:03without always repeating cd space back
6:06:07end and finally we put the steps A job
6:06:11is a series of steps and here you can
6:06:14you see the the template is the same.
6:06:16Everything has a name. You can see that
6:06:18here we have name checkout repository.
6:06:22This is going to be very helpful. If you
6:06:25go to your pipeline if you click on this
6:06:29backend lint and format you see here are
6:06:31the steps and let me split the screen
6:06:34and let's look at let's look at this.
6:06:36You can see that here we have checkout
6:06:39repository. This is exactly the name
6:06:41that I have here. Here we have setup
6:06:43Python. We also find this here. And why
6:06:46is this important? Because let's say
6:06:47that this step has failed. You can
6:06:50quickly go back to your YAML file and
6:06:54find that specific step. It's going to
6:06:56make your it's going to make debugging
6:06:59very easy. And as you can see, each step
6:07:01is going to run. And here they're going
6:07:03to run sequentially one after the other.
6:07:05And if a step failed, the other ones are
6:07:08going to be skipped. So let's look at
6:07:10what we have here in this specific job.
6:07:13First we need to check out the
6:07:14repository. Because we are running on a
6:07:17virtual machine, we need to download the
6:07:21source code. And here we have something
6:07:24that we call actions. You can see that
6:07:26here we have two different keywords that
6:07:29we are using. Here we have users and
6:07:31down below we have run. Basically this
6:07:34keyword is going to use scripts that the
6:07:38community has made. Let me show you what
6:07:41I am talking about. If I copy this name
6:07:44and search for it, let's see what we are
6:07:47going to get. You will find that there
6:07:49is a repository that contains the code
6:07:52for that specific action that we want to
6:07:54run. And uh you can see that here here
6:07:56is here it is actions/checkout.
6:08:00If I go back, it's the same name. and
6:08:03let's verify the version. How do we do
6:08:05that? It's simple. You click on the
6:08:07releases page and here you find the
6:08:10versions. So the latest version is 6.0.2
6:08:14but I am using version 6. And this is
6:08:17important because sometimes these
6:08:19actions change and uh if they introduce
6:08:22breaking changes you can read more about
6:08:25them here on GitHub. But if you use run
6:08:28this means you will provide uh commands
6:08:32that the uh that Ubuntu will run. This
6:08:36is the difference between users and run.
6:08:38Okay. So when I do this, let's look at
6:08:41here on uh this job. Let's open this.
6:08:45You can see that we this action has been
6:08:48executed and now it is syncing the
6:08:50repository. And yeah, you can open this
6:08:53and read more about this. You can see
6:08:55that they are initializing the
6:08:56repository and fetching the repository.
6:08:59This is what we want to see. After this
6:09:02step, the source code is on this Ubuntu
6:09:04machine. And now we can run commands in
6:09:06it. Setup python is going to download
6:09:09Python for us. And uh let's uh let's see
6:09:13here. We install rough. You can see that
6:09:15we use pip to install that library. And
6:09:18it worked. After that, we run the
6:09:21llinter and formatter. We can see that
6:09:24everything is great. Every there is no
6:09:26problem. And here these are jobs that
6:09:30GitHub will run. We can see that we
6:09:32don't have these uh post setup Python.
6:09:34It's not here. But the this these are
6:09:36just steps that are uh run to clean the
6:09:40machine because we are done with it and
6:09:42uh there is no need to keep this there.
6:09:45And you might be wondering how many
6:09:47steps do I need to put here. You can put
6:09:50as many steps as you want. Uh you feel
6:09:52free to customize your jobs however you
6:09:56like and um yeah there is no constraint
6:09:59know the difference between users and
6:10:01run and sometimes it's going it's easier
6:10:03to just use pre-made scripts. Uh so for
6:10:08example here if we wanted to install if
6:10:10we wanted to use rough you can we can
6:10:14search for it and uh use it maybe
6:10:16instead of doing all these steps. Uh
6:10:19let's do this just uh to show you how
6:10:22how to search this. So you type rough
6:10:25GitHub action and you search for it. As
6:10:28you can see there is one. So instead of
6:10:30repeating instead of doing these these
6:10:33steps manually we can reuse this rough
6:10:36action and that will make our life
6:10:38easier. It depends sometimes you might
6:10:41you don't find actions for the specific
6:10:44uh library that you are using. In that
6:10:46case you you use uh this approach and
6:10:50here I wanted to show you both so that
6:10:52you so that you don't get stuck if you
6:10:54don't find a a an action for a specific
6:10:58library that you are using. We have
6:11:00spent a lot of time in this specific
6:11:04file. So now let's go through the other
6:11:06steps quickly and let's see the
6:11:09difference between the back end and
6:11:11front end. Okay. Now let me go back to
6:11:14the CI pipeline. uh let's click on it
6:11:16again and let this time let's focus on
6:11:19uh linting and formatting the front end
6:11:22and here I will also open that file here
6:11:24it is front end lint and format okay
6:11:27let's see what we have here uh we start
6:11:30same as before we define the name and uh
6:11:33here we use workflow call to let GitHub
6:11:36know that this is a reusable template
6:11:39and here we have again just one job here
6:11:42let's see this time the working
6:11:44directory is front end. We are using
6:11:47checkout. And here we use, as you can
6:11:50see, this time we didn't install PMP
6:11:53manually, but we used an an action that
6:11:57we can find on GitHub. And we specify
6:12:00that we want to install PNPM version 10.
6:12:03But same thing with node, we are using
6:12:07an action to install node and we target
6:12:10this specific version. And here I forgot
6:12:13to talk about caching. We are doing that
6:12:16here. And on pi the the python side you
6:12:21can see that here we also cache pip. Why
6:12:23are we using caching? Basically if your
6:12:26dependencies don't change you don't need
6:12:29to download them every single time. So
6:12:31caching will make sure to save uh the
6:12:35dependencies so that uh the next job
6:12:38will just read them instead of
6:12:39downloading them each time. This is
6:12:41going to make your pipeline faster and
6:12:44it's uh yeah it's great uh it's a great
6:12:46thing to add. After that we install the
6:12:49dependencies. We run eslint to lint the
6:12:53frontend files and we use prettier for
6:12:56formatting and uh as you can see we have
6:12:59the these steps are are shown here on
6:13:02GitHub. Here we have check out
6:13:04repository, set up pmppm and nodejs, you
6:13:09install the dependencies and you run the
6:13:12linting and formatting steps. Great. Uh
6:13:15these are the two things that we have
6:13:17done in the previous video. We did them
6:13:20locally um before accepting the commit
6:13:24and now we replicated them here on
6:13:27GitHub using GitHub actions. Now we are
6:13:30going to add one more job. uh we are
6:13:33going to perform unit testing and let me
6:13:36show you how that looks like. Let's open
6:13:38backend unit tests. It's going to be uh
6:13:41similar but uh we will change some steps
6:13:44and I want to emphasize that in this
6:13:47course I am not going to show you how to
6:13:49write unit tests because this is outside
6:13:52our scope. If you want to do this that
6:13:54will that will take a lot of time and it
6:13:57depends on what programming language we
6:13:59are using. So we are going to consider
6:14:02that as a blackbox. And if you are using
6:14:05let's say go or another backend language
6:14:09you can swap the commands that I have
6:14:11here with the ones that are appropriate
6:14:14for the specific stack that you are
6:14:16using. Okay. So let's get started. Here
6:14:20the name is backend unit tests. And here
6:14:24we have again one job which I called
6:14:27tests. Let's go inside. First, we
6:14:30download the source code. We set up
6:14:32Python. We install the dependencies. And
6:14:35here we run the unit tests. Okay. And
6:14:39let's look. You can see that here we are
6:14:41using piest. This is the library that
6:14:45that we are using to test our backend
6:14:47code. And after that, this is not
6:14:49important. Basically, here we are using
6:14:51code coverage. Why? So that we can see
6:14:54the code coverage. You can see that here
6:14:57in the readme I showed that okay uh the
6:15:00tests cover 95% of the back end and 85%
6:15:04of the front end and uh yeah you if you
6:15:07want you can add that and for that we
6:15:09use the code coverage package or yeah
6:15:13you can search for it code cover this is
6:15:16the the service that we are using they
6:15:18have a free plan that uh that you can
6:15:21benefit from okay and so this step is
6:15:24not important but what we have here is
6:15:27is what we need. Yeah, it's it's the
6:15:29same. So, if I open this and look at run
6:15:32unit tests, you can see that here are
6:15:35the tests that we have. The everything
6:15:38went well. So, uh so we have 9 59 tests
6:15:42and they all pass without any problem.
6:15:45But if let's if one of them failed, this
6:15:47step is going to fail which means that
6:15:50this pipeline will not succeed. And we
6:15:52are doing the same thing uh in the front
6:15:55end. Let's open that front end unit
6:15:58tests. Let's go down. So here after
6:16:00installing the dependencies we have this
6:16:04test script and on in the front end we
6:16:08put those in the in your package.json
6:16:11file. So here in this scripts object you
6:16:16let's find test. Here it is. We are
6:16:18using v test for that and we have other
6:16:21commands. So if you want to run end to
6:16:23end tests, we are going to look at this
6:16:25in the upcoming tutorials. But for now,
6:16:28we just run unit tests. And again, we
6:16:31use code coverage to get the code
6:16:34coverage for the front end. Great. So
6:16:36now we created these four individual
6:16:39components. Now let's see how to put
6:16:41them in CI.YAML
6:16:44so that they run so that it orchestrates
6:16:47all of them. Here is CI.YML. I have
6:16:50removed uh some stuff so that um yeah so
6:16:53that you don't get overwhelmed. First
6:16:55the name is CI pipeline and here we have
6:16:59something different. So here we put on
6:17:02push on pull request and on workflow
6:17:05dispatch. Let's start with this one. So
6:17:08on push will run this pipeline whenever
6:17:11we push something to the master branch.
6:17:13We will also run it if we create a pull
6:17:16request that targets the master branch.
6:17:19This is what I have here. If I click on
6:17:22pull requests. Let's look at this one.
6:17:24It is targeting the master branch which
6:17:27means it's going to trigger this
6:17:29workflow to run. If I scroll down, uh
6:17:32you can see that I have the workflow
6:17:34here. Here I have my four steps and you
6:17:38can see I cannot merge because we have
6:17:41some failures. Some jobs have failed.
6:17:43And finally we have workflow dispatch.
6:17:46and workflow dispatch allows us to look
6:17:50at the workflows. Let's look at CI
6:17:52pipeline. And I can come here and run it
6:17:55from this page. This is useful
6:17:59especially if you want to uh if
6:18:01something failed and you want to rerun
6:18:04the job manually, you can do it from
6:18:07this uh page. Let's start with the front
6:18:10end link format check. First we give it
6:18:13a name and here users. This time we
6:18:16already have a workflow. We are going to
6:18:19reuse it here. And as you can see now it
6:18:21looks clean. Imagine taking all all
6:18:24everything inside this file and putting
6:18:26it here. And then do the same thing for
6:18:29these three. This file will be very
6:18:31long. Debugging. If you get a problem
6:18:34here uh in your workflow, it's going to
6:18:37be challenging to read the whole file,
6:18:41find the bug, and solve it. But now it
6:18:44looks clean. I just use this keyword to
6:18:47let GitHub know that the implementation
6:18:49is inside this specific file. And I do
6:18:54the same thing for backend lind, front
6:18:56end unit tests, and backend unit tests.
6:18:59The only thing that that we changed here
6:19:02is that we added the needs keyword. Why
6:19:05did we do that? Let's go back again to
6:19:08the pipeline. Let's collapse these two
6:19:11and let's look at just uh let's look at
6:19:14just these two jobs. By default, GitHub
6:19:18will run your jobs in parallel. Here we
6:19:21are performing linting and formatting in
6:19:23the front end and back end. And because
6:19:25these jobs are independent, we allow
6:19:28GitHub to run them in parallel. But if I
6:19:31want to run unit tests, I need to ensure
6:19:33that the code is clean. There is no
6:19:35syntax error. There is no problem in the
6:19:37code. If that's the case, I can run
6:19:39this. So if you have a dependency, you
6:19:42add the needs keyword. So this job needs
6:19:46front end lint format check to succeed.
6:19:50And this creates this dependency which
6:19:52will force GitHub to run them in
6:19:54sequence. We do the same thing for
6:19:56backend. It's going to wait for backend
6:19:59lint format check. If backend lint
6:20:02format fails, this job will not run.
6:20:05GitHub will skip it. And finally, we
6:20:07have pipeline success. For the pipeline
6:20:10to succeed, we want every job to
6:20:13succeed. Okay, that's why we put
6:20:15everything here. again runs on Abuntu
6:20:18latest and if always is important
6:20:21because if something failed we want to
6:20:24run this anyways we don't want to skip
6:20:26it because this is what's going to give
6:20:28us that status at the end the status is
6:20:32going to be either uh success or failure
6:20:35let's look at steps because this is
6:20:37where we have that logic let me expand
6:20:40this page so that we can see this better
6:20:43and let me also decrease the font size
6:20:45First here we have two steps. The
6:20:48pipeline can either succeed or fail. How
6:20:52do we know if that's the case? We use
6:20:55this function contains and here needs so
6:21:00this variable result is going to give us
6:21:02the results for all these jobs. This
6:21:06condition will check if we don't have
6:21:08any failures and if no job has been
6:21:11cancelled we will show check all checks
6:21:14passed successfully. But if we find a
6:21:18failure or something has been cancelled
6:21:21we will show some checks failed or were
6:21:24cancelled and we exit with number one.
6:21:28When we exit with one, we tell the
6:21:30runner that there was an issue and
6:21:33GitHub will display a uh let me show
6:21:36you. It's going to tell you that yeah
6:21:38that the test has failed. We can compare
6:21:41both of these tests to uh understand
6:21:44what's happening here. So the first test
6:21:47you can see it shows us it is showing us
6:21:50a failure. Why? because that specific uh
6:21:55condition this one contains failure is
6:21:59true. Why? Because we have one failure
6:22:01here. It is and this one was skipped
6:22:03that this will make pipeline success
6:22:06exit with one and the pipeline has
6:22:10failed. So it's which means that I won't
6:22:12be able to merge. But in this other
6:22:15case, you can see that it succeeded
6:22:17because um there is no failure and
6:22:20nothing has been skipped. Great. And now
6:22:22if I save this, you can see that we have
6:22:25we are in this branch and uh push the
6:22:28code, we will we will see that this will
6:22:31run. Okay. Now let me remove the
6:22:34splitting to give us some room to play
6:22:36with. Let's push all of this. Uh let's
6:22:39just type test because we are we are not
6:22:42doing something serious now. And but
6:22:44remember to type uh good com commit
6:22:47messages. I'll commit and push. Let's go
6:22:51back to GitHub. You can see that we have
6:22:53this new branch. And uh let's click on
6:22:56new pull requests. And here I don't want
6:22:59to merge with the upstream repository. I
6:23:02want to merge with my fork. And uh let's
6:23:06see. So I want to merge with the master.
6:23:08But here I need to select test number
6:23:11two. Okay. Okay, it shows me these are
6:23:13the changes that I have made. Click on
6:23:15create pull request. Let's keep
6:23:17everything like that and hit create pull
6:23:20request. Okay, let's scroll down and as
6:23:23you can see here are the checks. But
6:23:25notice that we have a problem. I can
6:23:27this button uh it's it's not grayed out
6:23:30like it is still active that which means
6:23:33I can still click on it. That's a
6:23:36problem and I am going to show you why.
6:23:39Because we are here we are not waiting
6:23:42for the final stage to finish which is
6:23:46pipeline success. But the solution is
6:23:50easy. Let's go to settings and we need
6:23:53to update the rule sets. This is the
6:23:56rule that we created before. Open it.
6:23:59Scroll down and here uh make sure to
6:24:02activate require status checks to pass.
6:24:06here. Let's click on add checks and
6:24:08start typing pipeline success and it
6:24:11should appear. Yeah, here it is. Select
6:24:14it and just like that save the changes.
6:24:18Now let's go back to that pull request
6:24:21and let's see what will happen. Uh as
6:24:23you can see I cannot now click on it.
6:24:25And if I hover the message says merging
6:24:28is blocked due to pending merge
6:24:31requests. And here is the pending uh
6:24:34job. It's pipeline success. And this one
6:24:37is going to wait for the other ones to
6:24:39finish before running. And let's see.
6:24:42It's going to run. Let's click on it
6:24:45just to see what's happening. So,
6:24:46pipeline succeeded. This is the message
6:24:49that we have uh that we wrote here. Echo
6:24:53all checks passed successfully. This is
6:24:55what we see here. Great. So, that
6:24:57worked. And now I can merge because
6:25:00there is no failure and nothing has been
6:25:03skipped. Amazing. But let's uh let's try
6:25:07to break this just to show you that we
6:25:10won't be able to merge broken code.
6:25:13Okay, let's open let's uh let's say
6:25:16main.js.
6:25:19Yeah, let's let's remove import and
6:25:21let's just put Axios from Axios. This is
6:25:24not correct. And here you can see that
6:25:26I'm not running this locally. I am
6:25:28running it here on yeah on GitHub just
6:25:32to make modifications quickly which
6:25:33means this is not going to be catched by
6:25:36the pre-commit hooks that we defined
6:25:37earlier. Normally this will will catch
6:25:41that but this is the limitation that I
6:25:43mentioned in the beginning of this video
6:25:45is that yes it's good to have those
6:25:47pre-commit hooks locally to stop the uh
6:25:52some bugs like these but people can also
6:25:56skip them. Uh this is why we create this
6:25:58pipeline to be the gatekeeper. It's not
6:26:00going to allow anything in unless it's
6:26:03good and yeah let's verify that. So
6:26:06let's just have another test commit
6:26:08commit and push. Go back to here. Okay
6:26:12is blocked. And now um the the linking
6:26:17and formatting step in the front end is
6:26:19going to fail. Let's click on it and
6:26:22let's see what's happening. So now we
6:26:24are set up in NodeJS.
6:26:26Good. And let's read the message. You
6:26:29can see that here it says cache restored
6:26:31successfully, which means it it skipped
6:26:34installing NodeJS and it just took it
6:26:38took it restored it from the cache which
6:26:41is great. It confirmed that the caching
6:26:43is working. But let's scroll down a
6:26:46little bit. Ah, you can see that ESLint
6:26:48is not happy with this. It's saying that
6:26:50error parsing uh this token. So it it
6:26:53couldn't find the import and it uh
6:26:56exited with code one. Now let's go back.
6:26:59You can see that because this job has
6:27:03failed uh we we weren't able to run the
6:27:07unit the tests because this depends on
6:27:11the first job and pipeline success
6:27:16failed two because we have one failed
6:27:19job and one cancelled job and now I
6:27:22cannot merge this. So I have to fix uh
6:27:25the problem and they cannot also push to
6:27:28master because it is protected. You see
6:27:30the beauty of having this kind of
6:27:32pipeline in your repository, it's going
6:27:34to be very helpful to catch problems and
6:27:38whenever you merge something to master,
6:27:40it's it is guaranteed that there are no
6:27:42bugs in it and you can merge without any
6:27:46problem and even deploy to your
6:27:49production server. We have reached the
6:27:52end of this tutorial. I hope that you
6:27:54found it interesting and uh you can find
6:27:57the documentation here on this GitHub
6:27:59repository. You will find the link in
6:28:02the video description. And what is next?
6:28:06So we still have two more sub chapters
6:28:09in module 5 and in this one we are going
6:28:12to talk about automated security
6:28:15scanning. We are going to scan
6:28:17vulnerabilities in the libraries that
6:28:20you use in your stack. This is very
6:28:22important. We don't want to push
6:28:24vulnerable code to a production system
6:28:27because you could lose data. There are
6:28:30many problems that can happen if that if
6:28:33you push vulnerable uh libraries and we
6:28:36will perform also this specific test to
6:28:40yeah ensure that uh there are plenty of
6:28:44tests that we can run and I'm going to
6:28:47show you some of them in the upcoming uh
6:28:50tutorial.
6:28:52Welcome back. So far you have built a
6:28:55pipeline that guarantees your code is
6:28:59formatted correctly and your logic works
6:29:02as expected.
6:29:04Today we will focus on automating
6:29:08security scanning.
6:29:10Here is what we are going to talk about
6:29:12in this sub chapter. I will try to
6:29:14explain to you the shift left security
6:29:18model. You are going to run software
6:29:21composition analysis. We will also add
6:29:25static application security testing
6:29:28and finally we will implement a fail
6:29:31fast strategy. CSA and SAS are the two
6:29:35types of tests that we are going to add
6:29:38because that they look for specific
6:29:41security issues in different areas. And
6:29:46the fail fast strategy basically means
6:29:48if the security scan fails, we stop the
6:29:52pipeline. We don't need to run the
6:29:55subsequent jobs because it's just going
6:29:57to be a waste of time because we know
6:29:59that we have a problem in the
6:30:01application. We should never deploy it.
6:30:03And in terms of the shifting security
6:30:06left concept, here it is. I have made
6:30:09this diagram to explain this to you. We
6:30:11have two ways of deploying an
6:30:14application or working and uh we have to
6:30:18work on security. This is very
6:30:20important. But before or traditionally
6:30:22we used to do it this way. First you can
6:30:25see that this is the development cycle.
6:30:28We plan, we design the application, we
6:30:31start coding, we build to make sure that
6:30:33everything is is correct. we have some
6:30:36tests that should pass and right after
6:30:39that we move to security. The problem
6:30:42with this approach is that if you work
6:30:44on a giant application and at the end
6:30:47you realize that you have a lot of
6:30:50problems that will make this development
6:30:54cycle take a lot of time and you will
6:30:58you will be fighting a lot of problems
6:31:00and that will push the release cycle
6:31:03further. uh this is very problematic
6:31:06especially with bigger applications but
6:31:09the shifting left security model fixes
6:31:13this issue by integrating the security
6:31:16checks during this inear cycle let's say
6:31:19so after designing the application now
6:31:23you start coding but while you are
6:31:25coding you are making sure that you are
6:31:28doing the best practices and so you keep
6:31:30thinking about security if you have for
6:31:33example input fields you think about uh
6:31:36sanitizing the inputs. Try to test the
6:31:39application. Can people do SQL injection
6:31:41or other types of attacks while in this
6:31:45tight loop? And you do it before you
6:31:48build you make sure that the feature
6:31:50that you are working on is secure. And
6:31:53if you do this, you ensure that even if
6:31:56you work on large applications,
6:31:57integrating security is going to be an
6:32:00easy task. It's like translation or
6:32:03adding themes in your application. If
6:32:06you do this at the end, it becomes hard.
6:32:09But if you do it in the beginning of the
6:32:11project, it's going to be very easy to
6:32:14uh to integrate. And finally, once uh
6:32:17once you are done with a specific
6:32:19feature, you release and the cycle
6:32:22repeats. Uh this is my GitHub profile
6:32:25and last time you can see that I showed
6:32:28you how to block merging if some of the
6:32:32tests fails. That's good. Now what I'll
6:32:36try to do I will fix these problems and
6:32:38I will come back because we want to
6:32:41ensure that these steps are fine but uh
6:32:44we want uh to add on top of this the
6:32:47security jobs. I am back and as you can
6:32:50see every job has run successfully
6:32:53without any issue. Now let's start
6:32:56auditing both the front end and back
6:32:59end. This is the software composition
6:33:02analysis step and we use it to basically
6:33:06audit the third party libraries that we
6:33:09are including in our projects. For
6:33:11example, in the front end folder we have
6:33:15package.json.
6:33:16It contains all the dependencies that I
6:33:19am using for this project. You can see
6:33:21that we have axios view router and many
6:33:26more. The problem that might happen is
6:33:29that let's say axio might be attacked.
6:33:33Attackers might find a vulnerability in
6:33:36this package which they can exploit.
6:33:39Your application will run without any
6:33:41problem. because that vulnerability is
6:33:44invisible to the checks that we have
6:33:47here. So by adding another job that goes
6:33:51through all your dependencies, checks if
6:33:54there is a vulnerability. When we do
6:33:56this, we can uh successfully block the
6:34:00pipeline. We will not allow deploying a
6:34:04vulnerability a vulnerable package to a
6:34:08to the production environment. we will
6:34:10wait for the maintainers of that
6:34:13specific project to fix the problem
6:34:16before updating to the newest version.
6:34:19And the same story for the back end. If
6:34:21I go back, if I go to the backend
6:34:23folder, you can see that here I am still
6:34:26using the requirements.txt.
6:34:28People nowadays use uv or other package
6:34:31managers. Uh there is nothing wrong with
6:34:34requirements.
6:34:35We define the dependencies that we are
6:34:38relying on and we need a tool that will
6:34:41go through all of them, check the the
6:34:43database, see if there are
6:34:45vulnerabilities and block the pipeline
6:34:48if it finds any. Okay, let's now go back
6:34:51to VS code. You can see that I am still
6:34:54using the web version because these are
6:34:57changes that are simple. Now let's go to
6:35:02the GitHub folder. open workflows and
6:35:06here I need to add two more jobs one for
6:35:09double check the dependencies in the
6:35:11front end and the other one will do the
6:35:13same but for the back end let's start
6:35:16with the front end I will zoom a little
6:35:18bit you can see that I have added this
6:35:20job front end vulnerability check this
6:35:23is the name that I've used here this is
6:35:26a reusable workflow we have explained
6:35:29this in the previous sub chapter and
6:35:32here what are we doing so We are
6:35:34checking the repository here. We are
6:35:37using PMPM version 11. I have changed
6:35:41this from the previous sub chapter. Why?
6:35:44Because if if you want to learn more
6:35:46about this specific version of PMP, make
6:35:49sure to go to Google and search for
6:35:52PMP1.
6:35:54You will land on this page. You can read
6:35:57more about it here. But basically uh by
6:36:00default PNPM is more secure than npm.
6:36:05Recently there were there were a lot of
6:36:07attacks on very popular npm packages and
6:36:11people sometimes they just download the
6:36:14latest version and that uh gets them
6:36:16hacked. PNPM solved these problems by
6:36:20blocking recent updates. So if an update
6:36:23is less than 7 days, it's not going to
6:36:25be installed automatically. you will you
6:36:27will get a warning or an error and you
6:36:30had to decide if you want to download it
6:36:32or not and it comes with other security
6:36:35features by default. This is why I've
6:36:38moved to version 11 to also help us in
6:36:42um blocking the download of malicious uh
6:36:46code in specific packages. And since we
6:36:50moved to PMPM version 11, that requires
6:36:54node version 22 and up. You cannot, as
6:36:57you can see, this is a requirement. You
6:37:00cannot use node 18, 19, 20 or any
6:37:03version below 22. And after that, we
6:37:06install the dependencies and again we
6:37:08are using pmppm to audit the third party
6:37:12packages. If everything is good, this
6:37:15will not fail. But if we if PMPM finds a
6:37:19vulnerability in a specific package uh
6:37:22then this step will fail. Now let's do
6:37:25the same thing for the back end. Uh the
6:37:28job is called backend vulnerability
6:37:31check. We are doing the same steps
6:37:33checking the repository. We installed
6:37:36Python version 3.13.
6:37:39And here we are using a specific GitHub
6:37:41action that uses the pip audit library.
6:37:46And here as input we give it
6:37:49requirements.txt because here we define
6:37:52the dependencies. If you were using uv
6:37:56or other py uh other package managers
6:37:59those they they generate a file called
6:38:03hi project.toml.
6:38:05If this is the case, you just change the
6:38:09input from requirements.txt
6:38:12to pip projects.totml.
6:38:14And you can read more. You can find more
6:38:16examples in their GitHub repository.
6:38:19Always this is what how how you should
6:38:22reuse these uh specific GitHub actions.
6:38:25Just look for their GitHub repository
6:38:29and read their readme. From the readme,
6:38:32they state that this action uses pip
6:38:35audit which is the default library that
6:38:38we use in Python to audit the
6:38:40dependencies. And uh if you want to
6:38:43learn more about the usage, just go to
6:38:45the usage section. And as you can see
6:38:48here uh for example, here they are
6:38:51pointing to the environment folder. Uh
6:38:53here is where how you can point to
6:38:56requirements.txt.
6:38:58And here you can point to more than one
6:39:00file. For example, if you have if you
6:39:02split the dependencies that are used by
6:39:06your application and those that are used
6:39:08just for development, you can provide
6:39:11both files. And let me show you where is
6:39:14the TOML one. Here it is. So if you read
6:39:17this, they say that to audit a project
6:39:19that uses pi project.toml,
6:39:22you use this specific uh notation. You
6:39:25just point to the directory where this
6:39:28file lives. Now let's go back to VS code
6:39:33and voila. You can see that it's very
6:39:36easy. Now we just need to go back to the
6:39:39orchestrator and add these jobs here. I
6:39:43have changed the jobs list. Now I start
6:39:46with the vulnerability checks. we what
6:39:49we have the front end and back end and
6:39:52we point to those two files that we have
6:39:55created in this session and after that I
6:39:59update the front end LinkedIn and
6:40:02backend LinkedIn jobs to depend on these
6:40:06vulnerability checks. You can see that
6:40:08here we added needs and this front end
6:40:12link job needs the vulnerability check
6:40:15to succeed before running. At the end in
6:40:19the pipeline success job I've also
6:40:21updated the needs list by adding the two
6:40:25security jobs. Now we can commit this.
6:40:29Let's say CI add security jobs and let's
6:40:34commit and push. Now we can go back here
6:40:38and uh you can see the pipeline will run
6:40:40again and this time we can go we can
6:40:44click on view details click on summary
6:40:48and here is our graph. Let me zoom in.
6:40:51You can see that we have the backend
6:40:54audit dependencies job and the front end
6:40:58vulnerability job. Both of them run in
6:41:01parallel and as you can see the backend
6:41:04lint and front end lint depend on these
6:41:09two jobs to succeed. If they fail the
6:41:11pipeline will stop. And as you can see
6:41:15the front end has failed. We will see
6:41:19why in a minute. And the back end also
6:41:22has failed which means that the pipeline
6:41:26success job will fail too. Good. And now
6:41:31if we go back to the food request tab,
6:41:35you can see that I cannot merge. This is
6:41:38blocked. And uh yeah, I need to solve
6:41:41these problems. Now let me let's verify
6:41:44what is happening. I will open this in a
6:41:46new tab and I'll open the backend job
6:41:50here. This is the result that we get
6:41:52from pip audit. And here we have a
6:41:55table. So name is basically the name of
6:41:58the package. Here we have flask cores
6:42:01and pi jwt.
6:42:04We have the current version the ID of
6:42:07the security problems. If you want you
6:42:10can take this ID open a new tab and
6:42:13search for it. You should be able to
6:42:15find the details about that
6:42:17vulnerability. And the severity is a I
6:42:21think it ranges from 0 to 10 if I
6:42:24remember correctly. And if it is closer
6:42:27to 10, that's very problematic. And as
6:42:29you can see, for this one, it's very
6:42:31critical. When you see this, it means
6:42:33that the you should update that package
6:42:36as soon as possible. But sometimes it's
6:42:38just medium or low. That's fine. Uh it's
6:42:41not very problematic. Okay. Now, let's
6:42:45look at the front end job. Ah, okay. So
6:42:49here because we are using PMPM version
6:42:5111, we need to yeah migrate from uh the
6:42:56previous version. So we need to first in
6:42:58uh try to update the the pmppm lock
6:43:02file. Let's take this command. Let's go
6:43:05back to VS code. I've opened the
6:43:07terminal. Let me let me go through the
6:43:10front end folder and let's run that
6:43:12command. Okay. So we have we have the
6:43:15dependencies but here I think yeah we
6:43:17are using pmppm version 10. So we need
6:43:20to update it. So to update they give us
6:43:24this command to run. Let's run it. Okay.
6:43:28So let's run pmppm setup to fix that
6:43:32issue. Let's run this command. Okay.
6:43:36Good. Now let's try to update it again.
6:43:40Okay. So we need to use pmppm self
6:43:42update. That's the new uh method. And
6:43:45now we can run pmpm-
6:43:48version. Still says 10.32.1.
6:43:52Maybe I need to close and open a new
6:43:54terminal window. So pmppm- version.
6:43:58Okay. Yeah, that now that worked. So we
6:44:00have version 11.1.3.
6:44:03Let's go back to front end. Let's copy
6:44:07this command again. Let's past space
6:44:10here and hit enter. Let's say yes. And
6:44:14now you can see that PMP is verifying
6:44:16the log file for any supply chain
6:44:19problems. Everything went well. But here
6:44:22we have a problem. By default, PMPM is
6:44:26going to ignore ES build. So if we want
6:44:29to fix this, we need to run PMP approve
6:44:32builds or so. We can either do this or
6:44:35come back here because this is going to
6:44:37work locally. But if we move to CI, it's
6:44:40still going to fail and we cannot uh run
6:44:43this in the CI interactively. Now with
6:44:46PMP version 11, we have a new file
6:44:50called PMP workspace that we can use to
6:44:54modify or to update the settings. Here
6:44:56it is. You can read more about it. And
6:44:58if you want to search for uh specific
6:45:02values within it, you you will find you
6:45:05go to this [clears throat] link pm io/
6:45:08settings and you can see they are
6:45:11talking about this specific file and
6:45:14what we need to do let's go back to this
6:45:17tab. So here they are talking about
6:45:19approved builds. We can search for it.
6:45:24Okay, now it's approved builds. Yeah,
6:45:27it's this one. And as you can see, we
6:45:29need to add this to PMPM workspace and
6:45:33we need to set AS build to true. Let's
6:45:36do that. I will go back to this tab. We
6:45:40are in the front end folder. Let's add
6:45:44PNPM
6:45:46workspacey.
6:45:49Ah, we already have it. Yeah, we already
6:45:51have it. Good. Ah, okay. Okay. So when
6:45:54you run the command, PMP PM will try to
6:45:56help you. Basically, as you can see,
6:45:58it's created the file for us and it's
6:46:01telling us set this to true or false.
6:46:04Let's set it to true. Let's open the
6:46:07terminal again. Yeah, let's try
6:46:10installing this. Copy the command. Paste
6:46:13it. Hit enter. Okay, it worked. Okay,
6:46:17let's commit these two. Let's commit
6:46:20this. Let's go back.
6:46:23to the pipeline. Ah, I need to sync the
6:46:26changes. Let's focus specifically on the
6:46:28front end job because the back end will
6:46:31fail. We haven't fixed that yet. Here we
6:46:34are setting PNPM.
6:46:36That's good. Can see that here it
6:46:39switched to version 11. That's what we
6:46:41want to see. We installed node and here
6:46:44we are installing the dependencies.
6:46:46Okay. So, that failed because we have 17
6:46:50vulnerabilities.
6:46:52Okay. And now let's go back to PMPM
6:46:55workspace again. We need to let's look
6:46:59at npmrc
6:47:00or sorry package.json.
6:47:03You can see that here before I used to
6:47:07to resolve those problems and to do that
6:47:10I added the overrides object because
6:47:13these are transitive dependencies. I
6:47:15don't depend on these libraries
6:47:18directly. But some of my primary
6:47:21thirdparty libraries are using them.
6:47:24That's why we added overrides. But now
6:47:27with pmpm11, we need to move these to
6:47:31this pmpm workspace. And uh we can
6:47:35search for that here. Let's see. Okay.
6:47:38So we click on overrides. And as you can
6:47:41see here is how we do it. We can copy
6:47:43this format. We can let's put it here.
6:47:47And yeah, I just need to copy all of
6:47:51this and put it here. Okay, let's see.
6:47:54We don't need these commas. We can
6:47:57remove them. Yeah, I think this is the
6:48:00format. Now, let's remove all of that.
6:48:02Let's remove this from package.json. We
6:48:06no longer need that. And let's save. And
6:48:08if you are wondering how do I know all
6:48:10of this, you can read the migrate v uh
6:48:14v12 to v11 empm guide. So they have a
6:48:19guide that will help you migrate to this
6:48:22new version. Okay. Now let's save this.
6:48:25Let's type fix move overrides to pmppm
6:48:32work space commit
6:48:35and sync the changes. Let's open the
6:48:38front end job again. Remember before we
6:48:40got 17 vulnerabilities.
6:48:43Let's see how many we will get or if
6:48:46this change will solve will resolve all
6:48:49of them. Okay, we have the same problem.
6:48:52I think because we have moved those from
6:48:54package.json, we need to uh again update
6:48:58the log file. Let me do it again. pmppm
6:49:02install. Okay, so that changed the log
6:49:05file. Good. So this is how how we work
6:49:08with uh GitHub actions. You start in
6:49:13with something, you test it doesn't
6:49:15work, it gives you the error code, you
6:49:18fix it, and you keep iterating until you
6:49:21get something that works. And in the
6:49:23future, let's say that we we have fixed
6:49:26this. In the future, this job might
6:49:28fail. So you open it, you look at some
6:49:32the the step that failed, you read the
6:49:34error and you go fix it. And this is
6:49:38going this is super useful to ensure
6:49:42that the code that you are deploying is
6:49:45secure and that there are no problems in
6:49:47it. Okay. Uh we are the job has failed
6:49:51once more and now we have 14
6:49:55vulnerabilities
6:49:56and most of them are coming from Axios.
6:49:59You can see that all of them are coming
6:50:01from Axios. Uh let's see. Yeah, all of
6:50:04them are coming from Axios and you can
6:50:06see that here the vulnerabilities they
6:50:08show you the range where that was found
6:50:10and the patched version. You can also
6:50:14go to the GitHub repository of that
6:50:17specific project, open their releases
6:50:21uh page and here you can read about the
6:50:24release. Sometimes they might have a
6:50:26security section in the release where
6:50:30they talk about the specific
6:50:32vulnerability that was found. So now we
6:50:35can install version 16.1.
6:50:38Uh it's the latest version. uh it was
6:50:41released for more than a week. So that's
6:50:44fine. PMPM will be able to install it.
6:50:47Okay, let's take this version. Let's go
6:50:50back to VS Code. We no longer need this,
6:50:53but we need to search for Axios here.
6:50:56Let's change the version to 16.1.
6:50:59Let's also again run pmppm install.
6:51:04Okay, good. So the the log file has
6:51:06changed. Let's commit and let's push the
6:51:09changes and hopefully this time this job
6:51:12will succeed. Let's go back to the pull
6:51:14request. Open the job in a new tab and
6:51:18uh let's wait. Awesome. We were we were
6:51:21too close. Uh but now we have one. So
6:51:24this this library we are not depending
6:51:27on it. But there is uh some third party
6:51:30library that uses that uses it under the
6:51:33hood. But you can see we went from 17
6:51:35vulnerabilities to only one. And um
6:51:40yeah, we can add this to the transitive.
6:51:43Let's see. So, copy this. Go back to
6:51:46PNPM workspace where it is here. Here we
6:51:50have that version. And now we can add
6:51:53what was the package name? I think it
6:51:55was WS. Here we put this and and I think
6:52:01the the last version is 8.20.1.
6:52:05So we can fix it to that specific
6:52:08version. You can always verify. You can
6:52:10either type the package name and then
6:52:13add npm or you can add GitHub to the
6:52:16end. As you can see the last version is
6:52:19820.1 and it was published 9 days ago.
6:52:23That's good. If it was updated just a
6:52:25day ago, the MPM will not allow you to
6:52:27download it because by default it's I
6:52:30think you can change the value. If you
6:52:33go to settings, you can override that.
6:52:36Uh you can set it to one day, a week or
6:52:40more. It depends on the severity. If a
6:52:43package is very popular, uh it's it's
6:52:46good to wait for a while before
6:52:47downloading the last version. Okay,
6:52:50let's now again run pmppm install and
6:52:53let's see if the log file will change.
6:52:55And yes, it did. Commit. Save the
6:52:58changes and let's push. This time it
6:53:02should pass. We are installing the
6:53:05dependencies and yes it worked. It took
6:53:09it took a while but we have solved the
6:53:11problem. Now you know how to iterate uh
6:53:15over this. Now we are using PMP version
6:53:1711 which is strict. It uh as you saw it
6:53:22blocks a lot of things and we needed to
6:53:24do uh some migration but it's worth it
6:53:27once you do this. You do this once it
6:53:29takes some time but later it's uh you
6:53:32don't think about it. Awesome. Now we
6:53:34can uh focus on the backend job. Let's
6:53:39go back to the audit section and let's
6:53:42see what we have here. Okay. So we have
6:53:44two packages. We have flask cords and pi
6:53:48JWT. The first package is problematic
6:53:51because they don't have a new version.
6:53:53So this is the last version that they
6:53:55have. And pi JWT I'm not sure about that
6:53:59one. So let's search for it. Uh let's
6:54:02click here. Let's look at releases. Is
6:54:05it an active project? Yes, it is still
6:54:07active. And the last version is 2121
6:54:11which is again we have the same problem
6:54:13with PI JWT. We don't have a new version
6:54:16that addresses this um vulnerability.
6:54:20How do you handle this case? Well, it
6:54:24depends. For example, in my project, I
6:54:27am not using Flask. I am using fast API.
6:54:31And I'm not sure why this is here. But
6:54:33if you dig deeper, you will find that a
6:54:36dependency that I'm using just for
6:54:37development is relying on this package.
6:54:41So I can move that specific package to a
6:54:45to requirements dev and I can scan only
6:54:48requirements. That's that's problematic
6:54:52too. Yeah, you solve the problem with
6:54:54production. So the code that you will
6:54:55put in production is going to be safe
6:54:57because we will not use this library.
6:54:59But if uh you you have if you run this
6:55:02[clears throat] locally, you might also
6:55:04get infected.
6:55:06There is another solution which is to
6:55:08basically remove that package if you
6:55:10don't need it. For example, if you used
6:55:12it only once. Yeah, you can remove it
6:55:15then. It's not it's not necessary. And
6:55:18as we saw in this package, the
6:55:20vulnerability is critical. Maybe if it
6:55:22was low or medium, we could skip it. uh
6:55:26or you could read it. Uh if it requires
6:55:29user input or something like that. Uh if
6:55:32you run it locally, you are safe. There
6:55:34is no going to be any issue. But yeah,
6:55:37this one has a critical problem. And uh
6:55:40so it allows SQL injection. Can see that
6:55:43here they say this vulnerability can
6:55:45lead to unauthorized data manipulation,
6:55:47data exfiltration, DOSs and yeah a lot
6:55:51of problems. But most of them so SQL
6:55:54injection requires an application that
6:55:56users interact with. So that's the
6:55:58problem. And here we we are not we are
6:56:01not we are not using this. We are using
6:56:03fast API for the front end. But it's
6:56:06it's very tricky. Sometimes you have to
6:56:08wait for a specific package to release a
6:56:10fix uh for the that specific problem.
6:56:15What I propose to do is to divide the
6:56:18requirements file to two files. one for
6:56:22production and the other one for local
6:56:24development. Let's go back to GitHub.
6:56:27Let's remove or let me keep that
6:56:30specific workflow because we need to
6:56:32update it. Backend vulnerability check.
6:56:35I'll keep that one here. I'll go to back
6:56:37end here. This is what we currently
6:56:40have. I'll copy this file. Paste it.
6:56:43Okay. So, it's not allowing me to copy
6:56:45the file. Okay, that's not an issue. We
6:56:48will create another one.
6:56:49requirements-dev.txt.
6:56:53Let's copy all of this. I already uh
6:56:57have them here. So, let's put them in
6:57:00this place. How do how do we split the
6:57:03the file into dev and one the other one
6:57:06for production. Basically, these
6:57:08libraries are used for development. For
6:57:10example, for pi test, we use that to run
6:57:12tests. We're not going we don't need
6:57:15this in production. So, that we can
6:57:17drop. We can also drop the other with
6:57:20the other packages by this facing kio
6:57:22and the coverage pre-commit that's
6:57:25another local one we have explained this
6:57:27in the previous sub chapters locust this
6:57:30is we this is new we haven't talked
6:57:33about this uh but we will see it in the
6:57:36upcoming chapter so I will not explain
6:57:39this now but we will see it later uh the
6:57:42other packages are re are uh important
6:57:46python TNV is used to load environment
6:57:49variables. Fast API as our back end. We
6:57:53use Pantic to validate input. UVON is
6:57:57important. Yeah. So everything else is
6:58:00is required by our application. So this
6:58:03is the first thing we will do. Uh now we
6:58:07have two separate requirements files. So
6:58:10let's do this here. Let's commit this.
6:58:13Now let's go back here. Here you can see
6:58:16that we are still pointing to
6:58:18requirements.txt.
6:58:20That's good. Uh let's just sync the
6:58:23changes. Let's see what will happen on
6:58:25GitHub. Uh let's click on backend
6:58:28vulnerability. It should fail but I at
6:58:32least we will have only one
6:58:34vulnerability. Yeah, we have just one
6:58:37which is PI JWT. This one is problematic
6:58:41because if I did uh more research and I
6:58:45found that docker are also flagging this
6:58:49library and here the score is seven.
6:58:51It's not uh like the other site that we
6:58:54saw before. Let me explain to you how do
6:58:58we get these CVE. Basically security
6:59:02researchers they go through packages and
6:59:04they look at the code to see if there is
6:59:06some something that is that should be
6:59:08fixed and in pjwt they say that it has
6:59:13weak encryption but the people that were
6:59:16that work on this project claim that no
6:59:18the the project is fine there is no
6:59:21issue and the problem that those
6:59:23researchers have pointed is that the
6:59:25length of the key uh is not very large
6:59:29so it can be small. So in this case
6:59:32where the maintainers don't want to
6:59:35address the CVE, we can tell this is why
6:59:39I kept this here. We can tell pip audit
6:59:41to ignore that specific CVE because it's
6:59:45just noise and the maintainers they are
6:59:47the ones who understand their project
6:59:50and they have the freedom to either
6:59:52accept the claim the CVE or reject it.
6:59:57because they have rejected it. We will
6:59:59add ignore vulnerabilities and we
7:00:02specify the name of that vulnerability
7:00:05and uh you can find that here uh it's in
7:00:09this in the ID column. Just copy this
7:00:12and paste it in this YAML file. Let's
7:00:16commit this. Let's think and hopefully
7:00:20this time this job should should not
7:00:23fail. Let's click on it again. Okay, we
7:00:26are checking the repository. We install
7:00:28Python. It should be cached. And
7:00:32let's wait for a moment. And as you can
7:00:35see, it says no known vulnerabilities
7:00:38found. And it also tells us that we
7:00:41ignored one vulnerability. And now if we
7:00:45go to summary, everything will run. So
7:00:48now because these security jobs has s
7:00:53succeeded, we now can run LinkedIn.
7:00:54adapt that we run unit tests. But if one
7:00:58step fails, the other one will not run.
7:01:01And here we have a problem with
7:01:03LinkedIn. Why? And we have a problem
7:01:06with unit tests. Interesting. Here we
7:01:09are installing pip install-r
7:01:11requirements.txt.
7:01:13But now we need to also we need to
7:01:15install from both of them. We keep
7:01:17requirements txt and we add requirements
7:01:19there. There is another option where you
7:01:22put - r requirements txt within
7:01:26uh requirementsdev.txt
7:01:28but we will keep it simple like yeah
7:01:30it's explicit it's uh it's not going to
7:01:32hide that uh specific part of the
7:01:34command commit and let's sync the
7:01:37changes. We also have a problem with
7:01:39linting that's uh that's weird. So let's
7:01:42see what is happening there. Well, we
7:01:45can just come let's come here and let's
7:01:47uh run pmppm lint and pmppm format.
7:01:52Okay, so we needed to format workspace.
7:01:54That's the issue. Okay. Uh let's type
7:01:57style apply formatting. I think now
7:02:00everything should pass hopefully. Let's
7:02:03close this tab and this one. Let's go
7:02:06back here. Let's click on view details
7:02:10summary and let's wait. Stage one
7:02:13completed successfully. Let's move to
7:02:16stage two. We have no problem with stage
7:02:19two. Now we can move to stage three. And
7:02:22the unit test has without any problem.
7:02:26Amazing. And the pipeline succeeded.
7:02:29Great. Now we can go back to the pull
7:02:30request and if we are happy we can merge
7:02:33the change the features that we are
7:02:35developing or the bug fix that we that
7:02:38we resolved. Amazing. You can see that
7:02:40uh it took us a while. Uh we had a lot
7:02:43of problems. We fixed an issue here. It
7:02:45generated other problems there. But I
7:02:48wanted to show you the full um the yeah
7:02:52the row experience. That's how you
7:02:54develop this. And uh it's it's it's
7:02:57easy. Yes. But you get a lot of errors
7:03:00along the way. And uh now I I think you
7:03:03you have a better idea of how to work
7:03:05with GitHub actions and how to resolve
7:03:08any problem that you might face. But we
7:03:11are not done yet. We have uh this is
7:03:13this was the CSA software composition
7:03:17analysis. Now we need to do static
7:03:20application security testing. What is
7:03:23the difference? Basically SCA doesn't
7:03:26look at your source code. It checks the
7:03:29third-party dependencies but static
7:03:32application security testing which is
7:03:34which we call SAS for short will go read
7:03:38your Python code and see if there are
7:03:40problems there. If you have password
7:03:43within the source code if you are using
7:03:46weak uh deprecated methods for example
7:03:50if you are using the execute function
7:03:53within your code that's a risky one. So
7:03:56this is going to check for these
7:03:57patterns and it's going to let you know
7:03:59if it found something. Let's go to VS
7:04:03code and let's add another job here. We
7:04:07will call this one backend stat check
7:04:11and we will put this code inside. Let's
7:04:14scroll down. What we have here is this
7:04:17is uh similar to what we have done
7:04:19before. The only thing that changed is
7:04:22bandit. Here we are using a library
7:04:25called Bandit. You can search for it on
7:04:28GitHub. And as you can see, it is a tool
7:04:31designed to find common security issues
7:04:34in Python code. That's the difference
7:04:37between this and uh the other two jobs
7:04:40that we worked on earlier. It is going
7:04:43to read your code. And here we are
7:04:45bandit with these specific arguments. Uh
7:04:48basically dash r means recursive. Here
7:04:51we are. So the working directory is
7:04:54backend. It's going to go in the inside
7:04:56the backend folder and go through all
7:04:58the Python files. Dash LL means so here
7:05:03we have two L's. This means that this is
7:05:05going to flag only medium or high
7:05:09issues. It is going to skip low
7:05:13problems. We will see this in the logs.
7:05:16If you if you don't do this, you will
7:05:18have a lot of issues and they are just
7:05:20false positives like uh what we have
7:05:23done with PI JWT that CVE was blocking
7:05:27us. But if you analyze it well, you see
7:05:30that in the end it's not an issue.
7:05:32That's why we are skipping low problems.
7:05:34We want to focus just on medium or high
7:05:38issues. And if you want to find uh if
7:05:41you want to learn more about this
7:05:43library, you can always go to the source
7:05:45repository and read their documentation.
7:05:48There you will find everything. Okay.
7:05:51Now we need to go back to CI and we need
7:05:54to add that after the backend
7:05:56vulnerability check and I want let me
7:06:00take this. I want backend linting and
7:06:02formatting to run after both of these
7:06:06tests. Okay. And uh I will add that here
7:06:10also. I think that's all we need to do.
7:06:12Now we can we commit this and let's
7:06:16think the changes. Let's go back to
7:06:18GitHub. I think it should pass because
7:06:21uh I am not using any unsecure code in
7:06:24my back end. But let's verify or refresh
7:06:27the page. Sometimes it gets stuck. Uh I
7:06:30have been refreshing the page uh for a
7:06:32few minutes but it is still stuck. And
7:06:36when that happens, sadly, GitHub, they
7:06:38don't show you that there is a syntax
7:06:40error. But here you can see that I have
7:06:43missed a comma. That's the problem. And
7:06:45that's why the pipeline didn't start
7:06:47right away. Okay. And now it's working.
7:06:51Yeah, that they should they should fix
7:06:53this uh somehow because if you don't
7:06:56know that, they should tell you that it
7:06:59failed because of a syntax error. But
7:07:01anyways, now we know if something gets
7:07:03stuck uh that means that you have a
7:07:06syntax error somewhere. And here we are
7:07:09running those jobs in parallel. You can
7:07:11see that backend Linton is uh requires
7:07:16both of these jobs to succeed before
7:07:19running. And as I told you, Bandit will
7:07:21not fail. But let's look at the logs
7:07:23because I want to show you that we have
7:07:26so Banditis found 1,900 lines of code.
7:07:29Here are the metrics. So you can see
7:07:30that we have 159 low issues. If we
7:07:34didn't skip low issues, you can see that
7:07:38it will be a nightmare to look at all of
7:07:40these problems. And most of them are
7:07:43going to be false positive. These aren't
7:07:45real problems in your code. But if we
7:07:48had something in medium or high, now we
7:07:50should look at those seriously. And here
7:07:52you might be conf So don't confuse
7:07:55issues and confidence. So here they have
7:07:58total issues by severity. So these are
7:08:00basically if we have an issue is it
7:08:02severe or not in our case no they are
7:08:05low and confidence is that here it is
7:08:09set to high. This doesn't mean that all
7:08:10of these problems are of a high
7:08:13severity. No it's just the confidence
7:08:15level. So here this means that bandit is
7:08:18confident that all of these are low
7:08:20problems that we that we should skip and
7:08:23don't care about. It seems that this is
7:08:25working. The pipeline has succeeded.
7:08:28Everything has passed and now we can
7:08:29merge without any problem. But before we
7:08:32end this tutorial, I would like to show
7:08:33you how banditis fails. That's
7:08:36important. Um let's open let me open
7:08:38main.py. I have added this because it
7:08:41uses an outdated cryptography module. Um
7:08:45I've just added something just to
7:08:47trigger banditis. So let's see. Sure
7:08:51trigger banditis, but I'm going to
7:08:53remove it right away. I just want to
7:08:55show you what happens when Bandit fails.
7:08:58This will Yeah. Okay. So, let's wait for
7:09:01this. Okay. So, it is it is running. Uh
7:09:05let's let's click on Bandit. That's the
7:09:07one we care about. Did it fail? Yes, it
7:09:10did. So, the exit code is one. And now
7:09:13we have one high problem. And if you
7:09:16look closely, okay, so it is running
7:09:19here. It says issue. Uh so it it found
7:09:22that we used the MDA hash and it's uh
7:09:26yeah it's not uh secure. This is an
7:09:29outdated module that we should not use.
7:09:32So here they show you the where where it
7:09:35found the problem. Here is the location.
7:09:37It's online 18. And you can see that now
7:09:40we have one in the high severity
7:09:43category. That confirms that bandit is
7:09:45working and uh it it is reading the
7:09:48code.
7:09:49Uh it's not like the other the other
7:09:51ones. So now we can remove this safely.
7:09:54Uh we've confirmed that Bandis is
7:09:56reading the code and uh let's wait for
7:10:00this job to finish one more time. I want
7:10:03to ensure that every every job passes
7:10:06and that I can merge without any issue.
7:10:08I am back. The pipeline took a minute
7:10:12and 37 seconds. It succeeded which is a
7:10:16great sign. And uh and yeah, we have
7:10:19reached the end of this tutorial. You
7:10:22know, I've really enjoyed making this
7:10:24video. Uh it was challenging. Uh we have
7:10:27hit a lot of problems and we fixed them
7:10:30uh interactively. I hope that you
7:10:33enjoyed it too and that you have learned
7:10:35how to deal with securing your
7:10:38application. You can see that the
7:10:39pipeline is starting to get bigger and
7:10:41bigger. We are adding we are making sure
7:10:44that we develop a robust pipeline and by
7:10:48the end of the of this module you will
7:10:52get something that uh that is really
7:10:54great and that will help you uh find
7:10:57bugs easily and uh ensure that when you
7:11:00hit deploy that your application will
7:11:02run without any issue. In the next sub
7:11:05chapter which is titled advanced testing
7:11:09and dust. Dust is you can see that here
7:11:12we have sust and dust is dynamic. So we
7:11:15will add that one too. Uh we will still
7:11:17continue working on testing and
7:11:20security. So far we have just unit tests
7:11:23but we will add integration tests and
7:11:26end to end tests. Uh we will we will see
7:11:29we will see that in details but I just
7:11:31wanted you to know what is the next step
7:11:34in our course. Thank you again for
7:11:36watching. See you in the next sub
7:11:39chapter. In this sub chapter we are
7:11:42going to talk about advanced testing.
7:11:46Here we are going to add both
7:11:48integration and end to end tests and I'm
7:11:51going to show you something called dust.
7:11:54Here is the agenda that we will follow.
7:11:56As I said, we will start by writing
7:11:58integration tests. And here we are going
7:12:02to test a live me search instance
7:12:06because this is the search engine that I
7:12:08am using on my personal project. We are
7:12:11going to perform end to end tests using
7:12:14the the playright tool in order to
7:12:17simulate real users. And finally, we are
7:12:21going to look at dynamic application
7:12:23security testing. And we're going to do
7:12:26that with this tool OASP zap. If you
7:12:30want to follow with me, every command
7:12:33will be here. Just go to this GitHub
7:12:35repository, open the book folder and
7:12:38here we are in chapter number five. Just
7:12:42search DOS and you will find the this
7:12:45new sub chapter advanced testing and
7:12:47dust. And here you have all the commands
7:12:50that I will run here. And this is
7:12:53explained uh in detail. I will now close
7:12:56this tab because we no longer need it.
7:12:58And here you can see that I haven't
7:13:00merged this pull request yet. We will
7:13:03keep using it for uh the upcoming
7:13:05videos. And as you can see at the end uh
7:13:09if I click on this icon, you can see
7:13:12that we have all these tests. We can
7:13:14click on details here. Let's click on
7:13:16summary. And currently this is what we
7:13:19have in this video. We are going to add
7:13:21integration tests and end to end tests
7:13:25before the final pipeline success step.
7:13:29Let's start doing this. Open the GitHub
7:13:33folder. Open workflows. And here let's
7:13:36you can see that we have backend
7:13:38integration tests. This is what we want.
7:13:41But before we do that, let's change the
7:13:45branch to what did I call it? Test
7:13:48number two. Okay, let's do it here. Test
7:13:51number two. Okay, now we don't have that
7:13:54file. So, we need to create it. Click on
7:13:56this icon and add this file backend
7:14:00integration tests. I will copy and paste
7:14:03the code. And here, let me explain to
7:14:05you what we have. There are some things
7:14:07that we have seen before like checking
7:14:10out the repository. I will collapse
7:14:13that. Installing Python, installing the
7:14:16dependencies. All of these steps are
7:14:19known and uh we will skip them in this
7:14:23uh sub chapter. We won't we don't want
7:14:25to waste time here. The new stuff is the
7:14:28main search environment variables and
7:14:30the steps to launch that service. And
7:14:34let me explain to you why do we need
7:14:36integration tests. If I go back to
7:14:39GitHub and open the pipeline, as you can
7:14:43see, we have some tests already. We call
7:14:46them unit tests. These are important
7:14:49tests and they test few things in the
7:14:52application in isolation. For example,
7:14:55if I have a button or a function in the
7:14:58back end that converts time into a
7:15:00specific format, I can test that
7:15:02function in isolation. And let's say if
7:15:05I have if I am making requests to m
7:15:08search, I am going to mock that. I am
7:15:11going to uh not talk to M searcharch in
7:15:15unit tests because here we are just
7:15:18testing the individual functions or
7:15:21functionalities. Integration tests are a
7:15:24bit different. They test integrating
7:15:28external services like me search or if
7:15:31you have let's say a postgress SQL
7:15:34database or other services you test them
7:15:37in integration tests. And here when if I
7:15:40want to make a let's say a search query
7:15:43to me search I am going I need an
7:15:46instance of me search running to be able
7:15:48to do that that's why here in this uh
7:15:52workflow we are going to set up m search
7:15:56so that it runs in this job and so that
7:16:00we can make requests to it. Here we have
7:16:02a few environment variables. Uh these
7:16:05are specific to me search. So if you are
7:16:07using other services you need to change
7:16:09this part but we have a master key a URL
7:16:13and a name index name. Here we will
7:16:15store documents. I will explain these
7:16:18two environment variables down below
7:16:20because they are important. Okay let's
7:16:22start. Here we have the download me
7:16:25search binary step. So after installing
7:16:27the dependencies, Python and checking
7:16:30out the repository, we need to install
7:16:32me search and it is a binary file. So we
7:16:35can download it easily using this curl
7:16:38command. And uh this this also needs to
7:16:42change if you use other services. And
7:16:45here we have these two variables max
7:16:48retries and retry delay. Sometimes in
7:16:52the GitHub actions this step will fail
7:16:55and you won't be able to download M
7:16:58search and uh you might say oops the the
7:17:02integration tests have failed but
7:17:04actually the problem was just network
7:17:07and uh we just need to retry this step
7:17:10multiple times until we download mainly
7:17:12search. So we are here uh redefining the
7:17:16variables max retries and retry delay.
7:17:18And here we are using a for loop to uh
7:17:21try downloading me search max retries
7:17:25times and in this case it is set to
7:17:27three but you can change the value to
7:17:30something higher. If this keeps failing
7:17:32and if this step succeeds we will show
7:17:36my search was downloaded and we will
7:17:39break out of the loop. Otherwise we will
7:17:42exit with one and uh tell GitHub that
7:17:45this step has failed. Okay. So here we
7:17:48download me search. After that we need
7:17:51to start it. And in order to start it we
7:17:53just run the binary and we pass the
7:17:56master key. And here notice what I have
7:17:58what I have added at the end. I have
7:18:00this amperand. And this is very
7:18:03important. If you don't add this, this
7:18:06call will block this job or this step
7:18:09and we won't be able to run the next
7:18:11steps. But this by adding an amperand,
7:18:14it will run this as a back as a
7:18:17background job and it will allow the
7:18:19next steps to run without being blocked.
7:18:22And here when we run medarch, we need to
7:18:24ensure that the server has started. And
7:18:27the way we do that is we call the health
7:18:30endpoint. If this endpoint replies it
7:18:34means that M search is ready to accept
7:18:36requests. That's this is what we are
7:18:39doing here. And again we are using a for
7:18:41loop so that we keep pinging this
7:18:44endpoint. If we detect available great
7:18:47we are going to show this in the logs
7:18:50and we will break out. Otherwise we will
7:18:53sleep a little bit to uh retry. We don't
7:18:55want to do this quickly. And here uh at
7:18:58the end we just if this uh doesn't uh
7:19:01work we will show that the me search
7:19:04fails to start after the number of
7:19:06attempts that we have here. If this
7:19:08succeeds we will run the integration
7:19:11tests and uh again we will use piest and
7:19:15in my case I I am storing you can see
7:19:18that I try to group tests by their
7:19:21nature. Here I have all of my unit
7:19:23tests. Here I have integration tests and
7:19:26I also have load testing. We might see
7:19:28this in upcoming uh subchapters. And as
7:19:32you can see here I am using tbhore so
7:19:33that the exceptions are not very verbose
7:19:37and I passed the environment variables
7:19:39because we need them inside the tests.
7:19:42So the only thing that I want you to
7:19:44take from this workflow is that in
7:19:47integration tests you need to start your
7:19:50services. Uh if you have me search this
7:19:52is how you do it. If you have Postgress
7:19:54SQL, you need to do it differently. If
7:19:58you are having services in Docker, you
7:20:00need to run the containers, stuff like
7:20:03that. And after running all of your
7:20:05services, you can run your tests. And
7:20:08the beauty is that here if something
7:20:10broke, it's going to uh be uh very
7:20:13informative because you are testing the
7:20:16live services. Awesome. Now, we can go
7:20:20back to CI.YAML YAML and add this
7:20:24integration test there commit and we can
7:20:27see if that will fail or not. Let's open
7:20:30CI.YAML
7:20:32and here let's add that job. We have
7:20:35backend integration tests. I want to run
7:20:38it after unit tests. And uh here is the
7:20:42file. I need to copy this and also add
7:20:45it here below backend unit tests. And I
7:20:49guess that's it. Let's commit this. I
7:20:52have pushed the code. Now let's go back
7:20:54to GitHub. Let's refresh the page and
7:20:57let's see if this is going to uh fail or
7:21:01not. Let's open the details page and
7:21:04let's see what we have. I want to go
7:21:07back to summary. I want to see how this
7:21:09looks like. And yeah, everything is
7:21:11good. So we have unit tests and after
7:21:13that we will run the integration tests.
7:21:16I will wait for all these steps to
7:21:17finish and I will come back. And as you
7:21:19can see here, we have a problem with
7:21:21audits. Probably something in the front
7:21:23end changed. But this is not a problem.
7:21:25We now know how to solve this. Uh I will
7:21:27just focus on uh the integration tests
7:21:30and then maybe we will we can come back
7:21:32here and solve the problems that we
7:21:34have. Now we are in the unit tests
7:21:37phase. [snorts] Let's wait for it. Okay.
7:21:39Uh let's click on backend integration
7:21:42tests because I want to see the logs.
7:21:44Okay. So far so good. These are the
7:21:47steps that we that we have in every job.
7:21:50Okay, great. Now we are downloading
7:21:51Millie search and as you can see this
7:21:54time it worked. Here we we are showing
7:21:57attempt one of three and yeah we are
7:22:00still trying to download it and that
7:22:02that worked great and you can see that
7:22:05they show us how to run it. Now here in
7:22:08this step we are running it and as you
7:22:11can see uh these are the logs and
7:22:14awesome that worked. So far so good. Uh
7:22:17let's collapse all of these steps. Okay,
7:22:20but I think yeah, we have a problem
7:22:22here. Import error while loading conf
7:22:25test. Okay, so it couldn't import
7:22:27something because it doesn't understand
7:22:30fast API. Interesting. So I guess we
7:22:32need to fix something when we in when we
7:22:36are installing the dependencies. Okay,
7:22:39let's go back here. So we are installing
7:22:42requirements dev and uh what happens
7:22:46what what's the problem? So if we go
7:22:48back to back end we have divided the
7:22:50requirements into two files. Ah I see so
7:22:54I forgot to add - r. Yeah the difference
7:22:57here is that we need to also do this. So
7:23:01we need to do - r requirements
7:23:04txt or we can do it here. So we do r
7:23:09requirements.txt. txt space-ash r
7:23:12requirementsdev.txt
7:23:14txt and u yeah this time I will do it
7:23:17this way pip will go inside requirements
7:23:21dev it's going to find this it's going
7:23:24to open requirements txt and it's going
7:23:26to install fast API okay let's commit
7:23:29this and let's push let's go back to
7:23:33GitHub the pipeline has started let's
7:23:37let's see the details let's open the
7:23:40integration tests and I think this time
7:23:44it should work. Let's collapse all of
7:23:46these steps. We've installed M
7:23:49searcharch successfully. We have started
7:23:52it. Okay. And now we are running
7:23:55integration tests and as you can see we
7:23:57didn't have we didn't get the same error
7:23:59and they succeeded. Great. So that
7:24:02worked. Let's go back here to summary.
7:24:05As you can see, both tests
7:24:09uh run successfully, but the pipeline
7:24:11didn't succeed because we have a problem
7:24:14in the audit dependencies job. And as
7:24:18you can see here, we have a problem with
7:24:20JS cookie vest. So these are the two
7:24:24libraries that we need to fix. Let's do
7:24:26that. While we are here, remember that
7:24:29we have PNPM workspace and here we can
7:24:32add the libraries. uh we need to
7:24:35override them because in my case I am
7:24:37not using JS cookie that's why I need to
7:24:41put it here in the overrides object and
7:24:44what was the the version 306 okay so
7:24:48let's go back here and let's put 306
7:24:52that's the first one and what what else
7:24:55we have test so for test I am using it
7:24:59so if we open package JSON You can see
7:25:04that I have a V test. Here it is. And
7:25:07the here the version is 4014.
7:25:11Okay, that's why. Yeah, that's why it
7:25:13will it is complaining. So, we need to
7:25:15upgrade that to for the 1.0.
7:25:19Let's go inside the front end folder.
7:25:21And here, let's try to install the
7:25:25newest version. Here is the full
7:25:28command. Let me open package.json JSON
7:25:31again just to verify that we have V
7:25:34test. Yes, it is in the dev
7:25:36dependencies. This is why we are adding
7:25:38dash D and yeah let's install the newest
7:25:43version. Okay, it is installing and we
7:25:46are using PMP
7:25:49version 10 because this is a a fresh
7:25:52virtual machine. So that's fine. You can
7:25:54see that vest uh we were able to update
7:25:57it upgrade it to 4.1 and uh yeah let's
7:26:01see what we have here package.json yeah
7:26:04that worked. I have updated pmppm to
7:26:07version 11 because uh I don't want to
7:26:09see any error in uh this PCI pipeline.
7:26:14Let's run pmp install just to ensure
7:26:17that nothing uh oh sorry I need to go to
7:26:20front end and let's run pmppm install.
7:26:24Uh let's say yes. Okay, we are good to
7:26:28go. Let me close the terminal and let's
7:26:32say sure update dependencies. Let's sync
7:26:36the changes and hopefully this time the
7:26:38pipeline will succeed. We will let that
7:26:41one running and I will come back to it.
7:26:43But now we need to add let's see we need
7:26:46to add end to end tests. Let's open
7:26:48workflows again and let's create this
7:26:51file end to end tests. I will copy the
7:26:55code and I will paste it here. Uh let me
7:26:59collapse this and let's look at this
7:27:01environment variables. This is identical
7:27:04to what we have there. And in end to end
7:27:06test we need to have everything running.
7:27:08We need to have the front end, back end,
7:27:10and all services. So here we have all of
7:27:14these environment variables for me
7:27:15search because we need to start the mele
7:27:17search service. And let's see what we
7:27:20have here. Here we added timeout minutes
7:27:2230 minutes. Uh we don't want so if end
7:27:25to end test for some reason take more
7:27:27than 30 minutes, we want to time out.
7:27:29This depends on how many tests you have
7:27:31and how and the complexity, but for me
7:27:3430 minutes is more than enough. Here we
7:27:36are checking the repository. We install
7:27:38Python here. Let's see. So we want to
7:27:42install
7:27:44R requirements and here let me add
7:27:46dev.txt
7:27:48because we want to run uh maybe this is
7:27:51not important here. Yeah, it's not
7:27:53important because we won't run it. We
7:27:56will be right we will be using the
7:27:58playright tool and that one runs in the
7:28:00front end. That's why we just need the
7:28:03production dependencies to run the fast
7:28:05API service or server. Here we download
7:28:09me search. This is identical to what we
7:28:11have in the integration tests. We start
7:28:14search and here we start the backend
7:28:16server. We are using uicorn and again we
7:28:20have the same for loop. We ping the
7:28:23health endpoint and if it returns okay
7:28:25that's great. We know that the backend
7:28:27has started otherwise we exit. And here
7:28:31we have a step that I called see the end
7:28:34to end test data. When we create when we
7:28:37download search and start it, it's going
7:28:40to be empty. We won't have any data to
7:28:42play with. In end to end tests, we need
7:28:45to have data in the browser so that we
7:28:47can interact with uh articles or
7:28:50products or any it depends on the on the
7:28:53project that you are working on. And in
7:28:55my case, I want to put data in my
7:28:58research. So I have created a a specific
7:29:01file that I called test data that will
7:29:04insert documents to many search. You can
7:29:07see that here is the index that I am
7:29:09using for tests. It's called articles in
7:29:13test here and here here are all the
7:29:16documents that I will inject just for
7:29:19end to end tests. If you have Postgress
7:29:22or other databases, you can do the same
7:29:24something similar where you seed your uh
7:29:27where you seed the test data and after
7:29:30that we install here we have we use PNPM
7:29:33version 11 we install node we install
7:29:36the frontend dependencies and now we
7:29:39arrived at the playright tool. Okay. So
7:29:42we install it we check the version the
7:29:45version is coming from again
7:29:47package.json.
7:29:48Uh if we look at playright test here it
7:29:51is we take this specific version and
7:29:55display it in the logs. Okay. After that
7:29:57we cache play right because here we in
7:30:00we need to download browsers and uh this
7:30:03depends on your configuration. If you
7:30:06some if you want to run the test only on
7:30:09chrome this step will download Chrome.
7:30:11If you want to also add Firefox it's
7:30:13going to download Firefox as well. If
7:30:15you want to to do it on Safari, you will
7:30:17install Safari. So, and if the browsers
7:30:20are the same, we need to cache them so
7:30:22that next time when we run end to end
7:30:24tests, they are going to be there and we
7:30:26are just going to use them. And for
7:30:27that, we are using uh an action called
7:30:30cache, which is a very handy uh GitHub
7:30:34action that allows us to cache stuff and
7:30:37uh as you can see the continuation is
7:30:39here. So here we basically define the
7:30:41keys where we want to store um the cache
7:30:44for playright and here we just check. So
7:30:48here if there is no cache we will
7:30:50execute this function that will install
7:30:53here I want to run the tests on Chrome
7:30:56and Firefox I don't want to use Safari
7:30:59otherwise if we if we have downloaded
7:31:02the the browsers previously we will just
7:31:05uh go here and as you can see we have an
7:31:06if block so if we have the cache so we
7:31:10are checking if it is true that means we
7:31:12can just reuse that and we don't need so
7:31:15if we look at the difference between the
7:31:17You can see that here we have with
7:31:19dependencies
7:31:20and this takes more time. Uh but if we
7:31:23have things in the cache that will make
7:31:25things faster. So after all of these
7:31:28steps you can see that this is why I
7:31:30said that end to end tests uh this
7:31:32workflow is going to involve a lot of uh
7:31:35work but we are just reusing what we
7:31:37already have in the other workflows. So
7:31:40after setting up all of these s servers
7:31:44we can run the end to end tests and uh
7:31:47in package.json JSON I have this script
7:31:50that will run end to end tests and here
7:31:53I pass environment CI to true to
7:31:56differentiate between because you can
7:31:58run end to end tests locally on your
7:32:00laptop and or in this case we run it on
7:32:04G using a GitHub runner and finally
7:32:07after the the end of uh the tests we
7:32:11will upload the report we will use the
7:32:13upload artifact action and that will
7:32:17allow us to download the report and
7:32:19reports are amazing. Locally when the
7:32:23tests finish uh playright will open a
7:32:26new tab and show you the report. But
7:32:28here since we are running this on GitHub
7:32:31we can't open a new tab. So we need to
7:32:33download a file that will that contains
7:32:36the report. We can check it later. We
7:32:38can see screenshots. Yeah, it's uh it's
7:32:41really handy. Okay. So this is the end
7:32:43to end tests and this will run as I said
7:32:47on the cloud using GitHub actions. But
7:32:50what if you want to run this locally?
7:32:52For that I mean here on the repository
7:32:54everything is set up already. Uh you can
7:32:57see that we have this playright
7:32:59configuration file that will come when
7:33:01you install playright. And the way you
7:33:03do it is simple. You just you just run
7:33:07pnpm create playrite and this will
7:33:10install playright for you. When you do
7:33:12this the playright configuration JSON
7:33:15will be created with some default
7:33:16values. Maybe there um this command will
7:33:20also create a uh some tests so that you
7:33:23can just uh make sure that playright is
7:33:26running but you can delete all of that.
7:33:29Let's run the command together and let's
7:33:30see what happens. You can see that here
7:33:32it says do you want to use typescript or
7:33:35javascript. In my case this project was
7:33:37using javascript that's why I kept
7:33:39javascript but if you rely on typescript
7:33:43uh change that to typescript and here
7:33:45they tell you where to put your end
7:33:47toend tests. Uh in my case if I look at
7:33:49the front end I always try to group my
7:33:51tests so they are inside the end to end
7:33:54tests. So here the default value is end
7:33:56to end. Okay let's just keep it as end
7:33:59to end test. uh add a GitHub actions
7:34:01workflow. I mean no because we have we
7:34:04did it ourselves. And here this step
7:34:08tells you do you want to install
7:34:10playright browsers? This can be done
7:34:12manually via pmppm execute playright
7:34:15install. And this is the exact command
7:34:19that we have here. As you can see pmppm
7:34:21exe execute playright. Let's install
7:34:24them because in this machine we don't
7:34:26have them. install play operating system
7:34:28dependencies. Why not? And you do this
7:34:32once and later you don't need to uh
7:34:35install the browsers every single time.
7:34:37Okay, so this finished. There was
7:34:40something that was added to uh the logs
7:34:43to the log files, but I I won't change
7:34:46this. It's not important. So I'll just
7:34:48undo this. I just wanted to show you the
7:34:50steps that you need to run. And uh okay,
7:34:53let's see what we have here. Okay. So,
7:34:56we didn't override the file because it
7:34:58found it. But uh in your case, if you
7:35:01don't have this file, this is going to
7:35:02be created for you. So, let's go through
7:35:04it together in order to understand what
7:35:06we have here. First of all, we tell
7:35:09playright that our end to end tests are
7:35:12located inside this folder. We want to
7:35:15run things in parallel so that tests uh
7:35:19finish quickly. Here we set the number
7:35:22of retries. So if we are on CI sometimes
7:35:25tests fail randomly so we want to retry
7:35:28them but if you are locally we won't
7:35:30retry anything here forbid only is
7:35:34sometimes when you run end to end tests
7:35:37you can tell player to run just one but
7:35:40in our CI job we want to run everything
7:35:43so this will fail if you tell player to
7:35:45run one one test only because it's going
7:35:48to skip the other ones and we don't want
7:35:50to allow that. Here we have reporter.
7:35:53This is also important because when you
7:35:55run uh tests locally as I said you can
7:35:58get an HTML page that will show you what
7:36:02failed, what succeeded. Uh but if we are
7:36:05in CI we are running this on GitHub. So
7:36:07we need to to tell play that please show
7:36:09us nice logs on GitHub. Here we define
7:36:12the projects. You can see that I I am
7:36:14using Chromium and Firefox. If you want
7:36:17you can also add Safari down below. Uh
7:36:19you can change this however you like.
7:36:21And finally we have something very
7:36:23interesting which is web server. This is
7:36:26this will allow playright to run the
7:36:28server the front end for you. So if you
7:36:30forget to add a step to run the front
7:36:33end playright will use this command to
7:36:36start it for you and you tell it the URL
7:36:39or and here the use existing server is
7:36:42uh is interesting. So if uh we started
7:36:46the server manually play will see that
7:36:49oh the user has started the server so I
7:36:52don't need to run this command. This is
7:36:54why we use uh reuse existing server and
7:36:56again we have timeout. Awesome. So now
7:37:00we can close this. Let's open uh ci.yml
7:37:04and let's add end to end tests there.
7:37:08Okay we have integration tests. Let's
7:37:11add end to end test underneath it. Copy
7:37:14this and let's paste it here and let's
7:37:18see what did I do in the needs um list.
7:37:23So I am waiting for the integration
7:37:26tests and also unit tests uh in the
7:37:29front from the front end. Okay, let's
7:37:31now commit this add end to end tests
7:37:34commit and push. Okay, uh the tests are
7:37:40running. Let's look at the details page.
7:37:43Let's go back to summary and let's
7:37:45ensure that the end to end test is
7:37:48correctly placed. Yes, it is. You can
7:37:50see that it is waiting for the front end
7:37:53unit tests and the backend integration
7:37:56tests. I will wait for these jobs to
7:37:59finish and I will come back once the end
7:38:01to end tests uh start. I am back and as
7:38:04you can see so far everything is green.
7:38:08Uh and wait yes the end to end tests
7:38:10started because uh front the unit tests
7:38:14in front end and integration tests in
7:38:16back end didn't fail. Let's click here
7:38:18and let's see what is happening. We are
7:38:20now installing the backend dependencies.
7:38:23We are downloading millarch. Good. That
7:38:26worked. Now we are starting the mill
7:38:28search server. That worked also. Let's
7:38:32now start the backend server using
7:38:34uicorn. And as you can see, we were able
7:38:36to ping the health endpoint. We seeded
7:38:40the M search with 15 documents. We've
7:38:44installed PNPM node. And now we are
7:38:48installing the playright browsers. And
7:38:49as you can see, this takes time because
7:38:52now as you can see, we are in the if no
7:38:54cache state in the upcoming commits. Uh
7:38:57this will not run and it's instead it's
7:39:00going to to download to take the
7:39:02browsers from the cache. Okay. Yeah, as
7:39:05I said, this takes some time. Where are
7:39:08we at? Okay, so it took 30 seconds.
7:39:11Okay, and now we are you can see that
7:39:13this the other the the other step was
7:39:15skipped because we don't have any cache
7:39:18and now we are running the end to end
7:39:20tests and as you can see we have 178
7:39:22tests and here we are using two workers.
7:39:25So they are going to run the tests in
7:39:27parallel. You can see that the tests are
7:39:30uh so here we are running the first one,
7:39:32second one, third one but now they start
7:39:34randomly because we are running them uh
7:39:36not in a sequential order but in
7:39:38parallel. Okay, end to end tests usually
7:39:42take minutes to finish and it depends on
7:39:44how you uh design them. Mine uh mine
7:39:48will take around 3 minutes to finish. So
7:39:51I will stop the video here and I will
7:39:54come back. We are almost done. Uh we
7:39:57have 178 tests. Let's see. So they that
7:40:01should finish now. And uh all of them
7:40:03succeeded. You can see it took 3
7:40:05minutes. And here, okay, we are trying
7:40:08to upload the report. And awesome.
7:40:12Everything works. Let's go back to
7:40:14summary. Amazing. You can see that the
7:40:17pi the whole pipeline took almost 7
7:40:20minutes. Okay, let's wait for the last
7:40:23or let's go back. Let's go back here. Uh
7:40:26yeah, everything passed. Amazing. Now
7:40:29what I would like to do is I want to
7:40:31make a small change just to show you
7:40:33that the cache is working. Let's see
7:40:36what should I do here. I will just
7:40:39remove this comma from this file cuz
7:40:43it's not needed. So sure, remove extra
7:40:48comma and let's push this. Let's go back
7:40:51to GitHub. Okay, everything is uh okay.
7:40:57Let's click on summary and let's wait
7:41:00for end to end tests to start and
7:41:02hopefully this time we should see that
7:41:04we will be uh we will uh run the if
7:41:08cache uh step. The end to end tests job
7:41:12has started. Let's wait for Okay, so we
7:41:16are now in the front end area. Okay, we
7:41:19are installing the dependencies.
7:41:22Let's see what we have here. So we are
7:41:24looking at the cache and as you can see
7:41:26that worked. Uh let me collapse this.
7:41:29Before we ran this step if no cache but
7:41:33now we run we are running this other
7:41:36step if cash hit. Before this step took
7:41:4030 seconds but now this one took almost
7:41:43half of that. It's just 15 seconds but
7:41:46it's it's working. It's better than
7:41:48nothing. And yeah, the end to end test
7:41:51will succeed and uh uh yeah, everything
7:41:53will work. Let's create a new workflow
7:41:57called dast scan. I will copy the code
7:42:01and paste it here and let's see what it
7:42:04does. First of all, you can see that we
7:42:07are we also need my search because here
7:42:10we are going to scan the API. First of
7:42:12all, we are checking the repository. We
7:42:15set up Python. We install the
7:42:17dependencies. We download me search. We
7:42:21run it. Here we see the data because we
7:42:23want to have something in me search. We
7:42:26start the back end. And uh finally this
7:42:29is the new step uh that will use the zap
7:42:34project. And if you are not familiar
7:42:37with zap here it is. You can check their
7:42:40website. Uh it's an amazing project.
7:42:42It's free and open source. And uh we you
7:42:45can use it to scan your applications.
7:42:48And they have good documentation as
7:42:50well. We will use this in this video
7:42:52because by default this scan is going to
7:42:57flag a lot of things that are uh that
7:43:01are not problematic because we are not
7:43:03in production. Well, I will show you I
7:43:06will show you that. So let's look at
7:43:08this step. You can see that we are using
7:43:10a GitHub action provided by the Zap
7:43:13team. And here the target is the open
7:43:16API endpoint. Because we are using fast
7:43:20API by default it gives us all the
7:43:23endpoints in our back end. And this is
7:43:26useful because Zap will use this in
7:43:29order to know which endpoints to target.
7:43:32The format is open API. And here you can
7:43:36see that I'm referring to a file uh
7:43:40called zap rules. Right now I will just
7:43:42delete this one because we don't have it
7:43:44and I will I want to do this so that you
7:43:47understand the reason why we add this
7:43:49specific file. I have deleted that line.
7:43:53Now open ci.l because we need to add
7:43:55this scan here in the main pipeline. I
7:43:59will scroll up. Let's look at backend
7:44:02length and format. I want to put it uh
7:44:06or I want to run it after LinkedIn. So,
7:44:09let me let's go down below. Let's paste
7:44:12it here. And as you can see uh it's
7:44:16going to run after the l after the lint
7:44:18job. Now, copy it and paste it here.
7:44:23Now, let's commit this. Add dust and
7:44:26let's push it to the repository. Let's
7:44:30go back. Let's click on the the pull
7:44:32request. Scroll down and let's see. We
7:44:36should see the pipeline running. Okay,
7:44:39that's good. Let's try to see to look at
7:44:42the details. Click on summary and let's
7:44:45wait for dust to run. As you can see, it
7:44:49should be right after. Yeah, it is after
7:44:53uh backend link and format. You can see
7:44:55that right now we have two kinds of
7:44:57tests. We have sust and dust. The
7:45:01difference is that SAS checks your row
7:45:04source code for security flows. But some
7:45:07security problems only appear while
7:45:10interacting with with your application.
7:45:12This is where Dusk comes in and this is
7:45:15why we started mini search and the back
7:45:17end and we gave it as the target the
7:45:20open API spec so that when it looks at
7:45:23that at those endpoints it's going to
7:45:25attack them like any uh attacker would
7:45:28do uh in order to find vulnerabilities
7:45:31in your application. And as you can see
7:45:34right now it failed. Okay. So why did it
7:45:36fail? It failed because it couldn't
7:45:39download me search. You can see that
7:45:41even though we did three attempts, so
7:45:44the first one failed, the second one
7:45:47failed and the third one also failed. Uh
7:45:49if this happens for you, no problem. You
7:45:52can just go back and rerun the job.
7:45:55Okay, so I can run rerun failed jobs. Uh
7:45:59it gives me all of these. So I'll hit
7:46:01rerun. Okay, let's wait for now. Let's
7:46:05click again on this job. Hopefully this
7:46:08time we will be able to download Mary
7:46:10search. Sadly it failed again for some
7:46:13reason. I'm not sure why. I guess we
7:46:15just need to wait a little bit and
7:46:17retry. They tell us to download this
7:46:20manually from GitHub. But yeah, we
7:46:23cannot do that here in the pipeline. We
7:46:26we will need to change that specific
7:46:28step. Okay, I will just I'll try to
7:46:31rerun it one more time. If it fails
7:46:33again, I will wait for a few minutes and
7:46:36come back. I did some research and I
7:46:38think the problem that we have here is
7:46:40that if we look at their latest version,
7:46:43you can see that it was released 14
7:46:46minutes ago. But creating the assets
7:46:49take time. You can see that for example
7:46:51this one was just created a minute ago
7:46:53even though the release was pushed 15
7:46:56minutes ago. So that's why we are
7:46:59getting 404s because um the installer
7:47:03script is trying to download something
7:47:06from GitHub that doesn't exist yet. This
7:47:09is why we are getting uh 404 not found.
7:47:13So we have two options here. We need to
7:47:17wait a little bit to get the new file
7:47:21and then we can run the script again or
7:47:24in the future we need to update this to
7:47:28download a specific version because here
7:47:31when we run the installer script we try
7:47:34to download the latest version or the
7:47:37latest tag from GitHub and in this case
7:47:40it is
7:47:42146.1
7:47:43right now I will keep the script like
7:47:45this. But uh right after uh explaining
7:47:49the dust scan, I will go back and I will
7:47:51show you how to update only this
7:47:54specific step to uh download a specific
7:47:58version. And uh yeah, we will remove
7:48:00this line because it downloads all it
7:48:03tries to download the latest version
7:48:05always. Okay, I rerun the script and uh
7:48:09as you can see it worked because now we
7:48:11we have uh the assets were created and
7:48:15uh the installer script was able to get
7:48:18it from GitHub. Awesome. So now let's
7:48:20see as you can see the zap scan step is
7:48:23running and uh okay so here let's see
7:48:27what we have. We see a deprecation
7:48:30warning that's fine. We can uh avoid
7:48:33that. You can see that here it found 15
7:48:35URLs and it was able to find them
7:48:38because we provided the open API spec
7:48:41and uh here I want to mention that the
7:48:44dust scan that we are using is
7:48:46lightweight because it is targeting just
7:48:48the back end but we can also have a
7:48:52serious one that will check the front
7:48:53end as well. That one takes a lot of
7:48:56time. This is why I didn't include it in
7:48:58the main pipeline. Instead, we will
7:49:01create another file that will uh have
7:49:04the full dust scan and we will run it
7:49:07daily um outside of the the pipeline
7:49:10because we don't want to block
7:49:12everything just to perform that heavy
7:49:15scan and as as you can see this thing
7:49:18has failed and uh I was expecting that.
7:49:21Uh let's see what we have here. You can
7:49:23see that it checks a lot of things and
7:49:26most of them passed. As you can see,
7:49:28everything passed except some problems.
7:49:32And as you can see here, it says X
7:49:34content type options header is missing.
7:49:36Uh let's see what we have. We also have
7:49:38cross origin resource policy missing.
7:49:42And as I told you, we will have these
7:49:44problems because we are running this in
7:49:46a CI environment. It's not running in
7:49:50production. And if you remember in the
7:49:52past we have we have resolved these
7:49:56problems and in our application that we
7:49:58deployed we are using engineext we added
7:50:01the HTTP security headers so everything
7:50:04is set up already and zap is telling us
7:50:07that these are missings that's that's
7:50:09correct because we are running on a CI
7:50:11environment this is why we add that
7:50:15specific file that I was talking about
7:50:18because it allows us to tell zap to skip
7:50:22those specific tests. Here I will just
7:50:25add rules file name and we need to
7:50:28create this one in the GitHub folder. Uh
7:50:32let's create it here. Uh it's already
7:50:34there. Okay. So let's look at it. Here
7:50:37is the zap rules file and this is
7:50:40specific to my codebase. So if you add
7:50:44dust to your codebase, you might have
7:50:47different warnings. So in that case you
7:50:50need to change this list. So how do we
7:50:52do this? If I open let let me show you.
7:50:55So here we have ids and the format is
7:50:59simple. You have the ID of the rule.
7:51:01It's the first column. You have the
7:51:03action and you have a comment. So here
7:51:07we tell zap to ignore all of these rules
7:51:12because here the action is ignore. And
7:51:14this comment is for us so that we can
7:51:16understand what is happening here. How
7:51:18do we get these rule ids? You can get
7:51:21them from their documentation. You can
7:51:24see that here. For example, if we open
7:51:26this active scan rules, you have the ids
7:51:30here. So they they show you the rule,
7:51:33they explain it and they also give you
7:51:35the ID. There is also this website that
7:51:38I will make sure to drop in the video
7:51:41description where you can find uh every
7:51:45every rule. And here if we search for
7:51:47cross origin which is basically this
7:51:50error that we got here. Now you can see
7:51:53the ID is 90,000 and4. Here it is. You
7:51:57can see that we have uh uh a lot of them
7:52:00here. And yeah you can use this website
7:52:03to uh learn more about uh these specific
7:52:07rules. And if you want to read it you
7:52:10can click on it and you will find here a
7:52:12summary solution etc. Okay. And uh we
7:52:15can you can also find the ids here
7:52:17directly from the logs. Uh so for
7:52:19example here I can take this id and uh
7:52:23paste it here in the rules. Uh here it
7:52:26is. You can see 10,21
7:52:29uh and I have added it here. Once you do
7:52:31this the errors are going to go away.
7:52:34But make sure to not skip everything.
7:52:37Skip only the things that are related to
7:52:39a production environment. And as you can
7:52:41see here, many things are handled by
7:52:44EngineX. That's why I've added them
7:52:46here. And for example, this one is
7:52:48failing because we are running this as
7:52:51HTTP. We are not using HTTPS. And again,
7:52:54because we are not in production. This
7:52:56is fine. There is no problem with that.
7:52:59Here we have a problem with timestamps
7:53:01because M search uses Unix timestamps.
7:53:04It doesn't use other formats. And here
7:53:07uh Zap is thinking that this is a
7:53:09problem. But in fact that's how med
7:53:12search works. Okay. Now what I will try
7:53:15to update this uh so let's say CI add
7:53:21zap rules commit this. Let's sync the
7:53:24changes and let's see what will happen
7:53:27this time. I'll go back to the pull
7:53:30requests. Let's look at the pipeline in
7:53:33detail. Go to summary. Let's click on
7:53:37OASP zap API scan. Uh we need to wait
7:53:40for it first to start. Okay, let's see
7:53:44the now we are auditing the backend
7:53:46dependencies. This should pass without
7:53:49any problem. Great, that worked. Now
7:53:52let's go to LinkedIn and formatting. And
7:53:55now let's look at the dust scan. The
7:53:59scan has started again. It found 15
7:54:03URLs. And as you can see, it is using
7:54:06our zap rules file. Okay, let's see now
7:54:09what will happen. Hopefully, it's uh
7:54:12it's not going to fail. And yes, it
7:54:14didn't. Let's see what we have here.
7:54:17Okay, you can see that they tested 197
7:54:22uh rules or URLs.
7:54:24Um so all of them passed and again we
7:54:27have those problems with timestamp,
7:54:29cross origin, etc. because we have added
7:54:32them to zap rules the zap project or the
7:54:37zap tool is not failing it is skipping
7:54:40them and u that's why uh the pipeline
7:54:44has passed something to note is that
7:54:47over time this file will stabilize I'm
7:54:50talking about zap rules because
7:54:52especially if you keep adding more
7:54:54endpoints if you keep expanding your
7:54:56backend application yeah zap is going to
7:54:59find some uh small issues that you can
7:55:02skip. After you do that job once, when
7:55:06you get an error in your CI pipeline, it
7:55:09means that you have a real security
7:55:12issue that needs your immediate
7:55:14attention. Right now, uh you can see
7:55:16that we got a few errors because uh we
7:55:19are just setting up D for the first time
7:55:22and that's normal because uh by default
7:55:25these tools are trying to find anything.
7:55:28So that's why you get false alarms. But
7:55:31once you handle those, you you are good
7:55:34to go. And uh when you get an error,
7:55:37that's a good that's a good signal that
7:55:39something bad is happening. Great. And
7:55:41as you can see, the other steps are are
7:55:43also running without any problem. So we
7:55:46were able to run integration tests. End
7:55:48to end test should also pass. And
7:55:50finally, we will uh the pipeline will
7:55:52succeed. Awesome. We have done
7:55:54everything uh in this sub chapter. But
7:55:58uh let's let me also show you how to
7:56:01update the download step so that you can
7:56:03install a specific version. I'm not
7:56:06going to apply that to the to all files
7:56:08because I like to download the last
7:56:10version and I don't want to go update
7:56:13the the file and then commit to be able
7:56:16to download the last version. But if you
7:56:18are curious about how we can accomplish
7:56:20this, here is how to do this. I will uh
7:56:23remove all of this and I will paste
7:56:26this. Okay. And now what do we need to
7:56:28do? We need to go to GitHub. Click on
7:56:31releases. And here the last one is
7:56:34146.1.
7:56:36I will take this. I will take this one.
7:56:39So here we need to change this variable
7:56:42to uh target this specific tag. Here we
7:56:46are just echoing this message. And uh
7:56:50here in the curl command we have uh this
7:56:53handy arguments that we can use. Before
7:56:57we uh we had to add these environment
7:56:59variables. Yeah. Because we had an
7:57:02external loop. But here we can do that.
7:57:04We can do it directly with curl. So if
7:57:08uh the command fails we retry three
7:57:11times and between each retry we wait for
7:57:135 seconds. And we are we go to the
7:57:16releases page download version and we
7:57:19download this specific binary. After we
7:57:22do that we uh make it executable so that
7:57:25we can start it. And uh you can see it
7:57:28is smaller than the previous solution.
7:57:31But this one requires you to always open
7:57:33the file, update this commit, push and
7:57:37wait for the pipeline to finish. That's
7:57:39the the difference between the two. But
7:57:41if you prefer this, you can keep it. Uh
7:57:44in my case, as I said, I will not keep
7:57:46that one. I I like to always download
7:57:48the last version. We have arrived at the
7:57:51end of this sub chapter. I really hope
7:57:52that you enjoyed it. In my case, I I
7:57:55did. And uh it's always great to
7:57:58struggle to build a pipeline because
7:58:00once you do, uh you you see this uh
7:58:03amazing view where everything is green.
7:58:06And um yeah, it's super super useful.
7:58:10And now we finished the continuation the
7:58:12continuous integration pipeline. In the
7:58:15next uh subchapter we will work on
7:58:18continuous delivery which is also
7:58:20amazing. Uh we will connect this
7:58:23pipeline to our virtual machine on
7:58:26digital ocean. to that if uh this step
7:58:29which I'm I'm referring to continuous
7:58:31integration passes we will click on a
7:58:34button to deploy the built application
7:58:38directly without having to SSH to the VM
7:58:42copy the code run some some commands we
7:58:46will create a a workflow that will do
7:58:48this for us and we will just hit the
7:58:51button wait for a minute or two and see
7:58:54the new version live on the website. We
7:58:57will also update this pipeline because
7:59:00you can see that pipeline success
7:59:01depends on every job before it. So the
7:59:06so the graph looks a little bit ugly.
7:59:08But we just need to look at the last
7:59:12job. We we need to wait for end to end
7:59:14tests because if we wait for this job,
7:59:17we are waiting for this one. This one we
7:59:20are waiting for every previous job. So
7:59:22we don't need to uh make this uh let's
7:59:27go back to CI we can simplify this needs
7:59:30to just uh the end to end tests we know
7:59:33we don't need to have all of them
7:59:36because implicit they are implicitly uh
7:59:39within the end to end tests if we are
7:59:41waiting for that specific job. So we
7:59:43will do some optimizations to make the
7:59:45pipeline uh cleaner and uh yeah that sub
7:59:49chapter will be the the last one in
7:59:51module 5. After that we will move to
7:59:54module six which is the last module in
7:59:56our course. See you in the next sub
7:59:59chapter. We have arrived at continuous
8:00:02delivery. This is the last subchapter in
8:00:06module 5 and here are the things that we
8:00:09are going to talk about. I will show you
8:00:11how to compile and upload the static
8:00:14view front end. You will trigger
8:00:16automated database and search engine
8:00:19backups. Before deploying, we are going
8:00:22to require a human review using GitHub
8:00:26environments. And finally, we will
8:00:28execute an atomic virtual environment
8:00:31roll back strategy. So if something bad
8:00:34happened, we can go back to the previous
8:00:37working version. In the [clears throat]
8:00:38last sub chapter, we have built this
8:00:42pipeline and the last step is end to end
8:00:47tests. What are we going to do before
8:00:49moving to continuous delivery is to also
8:00:51add front end build. So we will try to
8:00:54build the front end to make sure that
8:00:55there is no syntax error that there is
8:00:57no problem in the dependencies that we
8:01:00are using before finishing the
8:01:02continuous integration pipeline. I have
8:01:05I forgot to add this in the previous sub
8:01:07chapter but we are going to add it now.
8:01:09Open the GitHub folder. Open workflows
8:01:13and add this new file content build.yml.
8:01:18But before I do that I need to change to
8:01:21test number two because this is the
8:01:23branch that we were using. Okay, let's
8:01:25now try to create the file. Okay, now we
8:01:28are able to create it. I will copy and
8:01:31paste the code here. It is similar to
8:01:34what we have done before. The name here
8:01:36is build front end. First we check the
8:01:38repository. We use pmppm version 11. We
8:01:43install the dependencies and after that
8:01:45we just run pmppm build. If everything
8:01:47goes all right, that's what we want. But
8:01:49if something bad happens, this job will
8:01:52fail. After that, we are going to upload
8:01:54the artifact so that if we want, we can
8:01:57reuse the build files in a next job or
8:02:01something like that. And the retention
8:02:03days is set to one because because if
8:02:05you add another feature, we are not
8:02:08going to use this uh cache. So there is
8:02:11no need for it. It's going to become old
8:02:13and it's going to be uh removed. Now
8:02:15let's open ci.l and let's add this job
8:02:19there. Here is how we do this. I'm going
8:02:21to come here and I want this to depend
8:02:25on end to end tests because if this
8:02:28succeeds I know that at least there is
8:02:30no problem in the code so we can build
8:02:33and here I will copy the job name and
8:02:38let's put it below dust check and that's
8:02:41all we need to add now commit all of
8:02:44this let's go back to GitHub let's look
8:02:47at the put requests the pipeline has
8:02:49started I will Wait for the pipeline to
8:02:52finish and I will come back. I am back
8:02:54and as you can see the front end build
8:02:57job has started. Let's click on it to
8:03:00make sure that it works. Okay. So now we
8:03:02are building the front end as you can
8:03:04see and it should work. Uh this is just
8:03:08a step that I wanted to add to ensure
8:03:10that building the front end succeeds.
8:03:13Great. And as you can see we get a green
8:03:15check mark. Amazing. Now let's go back
8:03:17to the pull request. Let's click on pull
8:03:20requests here. Let's click on this one.
8:03:24And let's make sure that everything is
8:03:26green. Awesome. So now we have finished
8:03:28working on continuous integration. Now
8:03:31we will work on continuous delivery.
8:03:34Before we start deploying the new built
8:03:39files to digital ocean, I would like to
8:03:42start with uh create working on two
8:03:45things. First of all, a backup
8:03:48management system. This is going to
8:03:50ensure that the servers disk does not
8:03:53fill up because if we keep creating
8:03:56backups for let's say mail search uh and
8:03:59maybe databases if we have them that
8:04:02will eat a lot of storage and because we
8:04:04are using a cheap digital ocean uh
8:04:07virtual machine that's we don't have the
8:04:10luxury to uh waste space. So we are
8:04:12going to create a script that will clean
8:04:14the backups for us and keep only the
8:04:17last five. That means we need to SSH to
8:04:21the server. Let me do that quickly. I am
8:04:24back and as you can see I am logged in.
8:04:27Now what I will try to do is the
8:04:30following. I will create this clean
8:04:32backups script in this location. Now I
8:04:35need to provide my password. Great. Now
8:04:38I will copy and paste the following code
8:04:40and I will explain it to you. First we
8:04:43have this keep count variable. Basically
8:04:46this this is going to uh control how
8:04:49many backups we want to keep. And by
8:04:51default it's it is set to five. I will
8:04:54keep it that like that but if we want we
8:04:56can change it. Here we have a we have an
8:04:58offset variable that will add one to
8:05:02keep count. We are doing this because of
8:05:05this uh huge command that we are running
8:05:07here. So I will explain what we are
8:05:09doing here. Basically this is a function
8:05:11that we are calling here on two folders.
8:05:14In my case in this application that I am
8:05:17deploying I have mainly search and I
8:05:20also have a SQLite database. So for me
8:05:24search the backup is going to be stored
8:05:26in the dumps folder and for SQLite I
8:05:30will store that in the backups folder
8:05:32inside the web application. So what I
8:05:35want is that I want to give this path to
8:05:38this function and keep only the last
8:05:41five files inside each folder and this
8:05:45is how we do it. So the target is
8:05:48basically going to be the path that we
8:05:50are passing to the function. If if the
8:05:52path exists, we are going to show in the
8:05:56screen that we are cleaning it. And um
8:05:59after that here here is what we are
8:06:01doing. First of all, we are using ls-tp
8:06:05so that we can sort the files by time.
8:06:08Here we use grap-v
8:06:10and here we are inverting the search so
8:06:13that we uh skip folders. This command
8:06:16will only keep files. And here we have
8:06:19tail dash n and we pass the offset here
8:06:22and the plus that we are passing here is
8:06:25going to basically give us all the files
8:06:28that are older than uh offsets. So for
8:06:31example if offset is set to six we are
8:06:33going to start from there and it's going
8:06:36to give me the backup that was from day
8:06:38six day seven etc. And we use X arcs to
8:06:42pass each file to the remove command.
8:06:45It's going to be put here so that we can
8:06:47remove it. And this is how this function
8:06:50works. And if you want you can play with
8:06:53this command on your own. You can copy
8:06:55copy paste it and see how it how it
8:06:57works. So if you now I can save the
8:07:00file. I will exit. Let me show you.
8:07:03Let's go back to web app. And as you can
8:07:06see so far I don't have any backups
8:07:08because I haven't deployed from GitHub.
8:07:10But once we start doing that the backup
8:07:13folder will be created and uh in terms
8:07:16of med search we will have the backups
8:07:19in the dumps folder. Now we need to make
8:07:22the script executable using sudo chode
8:07:25plus x. That's done. Great. Now we need
8:07:29to do something interesting. So we need
8:07:32to run sudo visudo. This is going to
8:07:35allow us to run commands that typically
8:07:39require us to enter the password because
8:07:41we need to run them using pseudo. But we
8:07:43are going to skip that. We are going to
8:07:46run them without a password. Why do we
8:07:48need to do that? Because uh in g on
8:07:51GitHub, we are going to have a job that
8:07:54will tell the virtual machine let's say
8:07:57to restart engineext or to restart your
8:08:00uh backend application. And for that we
8:08:02need to run systemctl reload engineext
8:08:05and but this requires us to enter sudo.
8:08:08We are going to run this command. We go
8:08:11at the end of the file and here we need
8:08:13to add the following. I will paste this.
8:08:16I'll go back to the beginning and I will
8:08:18change the username to mine. So for
8:08:21example my username is IAD but for make
8:08:25sure to change this one. So here what do
8:08:27we have? You can see that here we
8:08:29specify that we want to run these
8:08:31commands without a password. And the
8:08:33commands that we that we want to allow
8:08:37are systemctl reload engine x supervisor
8:08:40ctl restart all and the cleanup scripts.
8:08:44Other than that we the other commands
8:08:47are going to require a password. Never
8:08:50never never uh remove password because
8:08:53that's insecure. Let's say that someone
8:08:56was able to steal your SSH keys somehow
8:09:00and they were able to log into the
8:09:01virtual machine. The only thing they can
8:09:03do is either reload EngineX, reload,
8:09:06restart supervisor and uh run the
8:09:09cleanup com backup script. If they want
8:09:12to, let's say remove something from a a
8:09:15directory owned by sudo by the root
8:09:18user, they won't be able to because uh
8:09:20Linux will prompt them for the password.
8:09:23Okay. Now save the file and exit. Now I
8:09:26would like to also explain to you this
8:09:28specific file. Let's go. Let's use cat
8:09:33and let's call that specific file rs
8:09:36sync exclude. Here is what we have. What
8:09:39why do we need to do this? Because we
8:09:41want to copy the source code from GitHub
8:09:45and put it here on digital ocean in this
8:09:48specific virtual machine. And the way uh
8:09:51we are going to accomplish this is with
8:09:53a command called r sync. This is an
8:09:56amazing tool that allows you to copy
8:09:58files. But the problem is that it's
8:10:01going to make sure that what we have
8:10:04here on the virtual machine matches what
8:10:06we have on GitHub. And the this causes a
8:10:09problem because let's say that I added a
8:10:11file that I want to have here on the
8:10:14virtual machine but that I don't want to
8:10:16push on GitHub. The problem is that
8:10:18rsync is going to delete this specific
8:10:20file that I added here on the virtual
8:10:23machine. This is why I have added all of
8:10:26this here. You can see that in rs sync
8:10:28exclude I tell rync to never delete
8:10:33these files from the virtual machine.
8:10:36Keep them and uh keep them on top of
8:10:39what we have on GitHub. Okay. So for
8:10:42example we are keeping backups, we keep
8:10:44the same file, we keep node modules etc.
8:10:47These are things that I want to keep
8:10:49here when I run the rs sync command. I
8:10:52am showing you this because now we are
8:10:54just preparing. We haven't yet wrote
8:10:56wrote the deploy job but we will see
8:10:59these things in that yl file. So don't
8:11:02worry about this. We are just preparing
8:11:04for that step. Great. So now we need to
8:11:06go to go to GitHub because we need to
8:11:08create an environment. I will show you
8:11:10how to do this. Click on settings. Click
8:11:13on environments. Click on new
8:11:16environment. Here let's call it
8:11:18production and click on configure
8:11:21environment. Now click on required
8:11:24reviewers. We are doing this so that as
8:11:26I said when before deploying we will
8:11:29have certain number of people that will
8:11:31receive an email so that if they want
8:11:34they can deploy the application. We
8:11:36don't want the deployment to be
8:11:38automatic. So that we just have one
8:11:40layer of confirmation before confirming.
8:11:44Here I will put my own account but if I
8:11:46have other team members I can add them
8:11:49here and just click on save protection
8:11:51rules. Now we need to create a few
8:11:54secrets. For that we need to go here
8:11:57click on secrets and variables and click
8:12:00on actions. These are secrets that are
8:12:03going to be used in the deploy job. For
8:12:05example, to connect to the virtual
8:12:07machine, we need an SSH key. So that
8:12:09this is why we need to create these
8:12:11secrets inside the repository secrets
8:12:15section. We click on this button. I will
8:12:17add the following keys. So we need to
8:12:20have four of them. We need to have the
8:12:23IP address of the virtual machine, my
8:12:26user, and uh we need the SSH key and
8:12:30finally the mainly search master key.
8:12:32These are all the keys that we need to
8:12:34create. Let's start with adding the SSH
8:12:38key. Now I will open my terminal. I will
8:12:41create this a key using this command.
8:12:44And here I will just add test test to
8:12:48make it obvious. Yeah, let's hit enter.
8:12:51So now we are generating the public and
8:12:54private keys. And here the the command
8:12:57is asking us to add a passphrase. We
8:12:59should not add a passphrase. So just hit
8:13:02enter. Hit enter again and the uh the
8:13:04keys were generated. Why don't why we
8:13:07shouldn't add a passphrase? Because when
8:13:10we run the deploy job on GitHub, the the
8:13:12job should run without being asked for a
8:13:15passphrase. That's why we created a new
8:13:17key just so that we can use it on GitHub
8:13:20to connect to the virtual machine. Uh
8:13:23use the cat command to print the public
8:13:26key to screen and then copy it. I will
8:13:29do that and I will come back. After
8:13:31copying the key, go back to the virtual
8:13:34machine. Run nano SSH authorized keys.
8:13:39After you run this, paste the key in a
8:13:42separate line. I will do that and I will
8:13:44come back. My public key is now inside
8:13:48the virtual machine. Now I need to go
8:13:50back to my to the local terminal or to
8:13:54the local machine and this time copy the
8:13:58private key. Let me do this quickly. So
8:14:01I'll just remove public and I'll run the
8:14:04command. And here I have the WL copy
8:14:07program that allows me to take the
8:14:09output of this command and just uh copy
8:14:12it to my clipboard. Now go back to
8:14:15GitHub and here you need to paste the
8:14:18private key in the secret input field
8:14:21and use digital ocean SSH key as the
8:14:24name. I will do that and and I will come
8:14:26back. Awesome. So we are we were able to
8:14:29create our first repository secret.
8:14:32Later we can either update it or delete
8:14:35it. Now let me add Digital Ocean
8:14:38username. In this case it's a Mac. Let
8:14:41me also add the host IP. And finally, I
8:14:45need to add me search master key. If you
8:14:48don't remember where we stored the me
8:14:52search API key, run this command. sudo
8:14:56nano etc me search environment. Copy it
8:14:59from there. Paste it here and click on
8:15:02add secret. As you can see, I have
8:15:04created the four secrets that I will use
8:15:07in the deploy job. We are now ready to
8:15:11create the deploy job. First, open the
8:15:15workflows folder and create a file
8:15:19called deploy.yaml.
8:15:21Now, let's look at the content of this
8:15:23file. The name is manual deploy because
8:15:26after merging a boot request, we want to
8:15:29have the possibility to click on a
8:15:32button manually to deploy the
8:15:34application. The way we do that is by
8:15:36adding workflow dispatch. We will see
8:15:39this later when we merge this put
8:15:41request on GitHub. We will have a button
8:15:43that will allow us to manually run this
8:15:46particular job. And here under jobs, we
8:15:49have two. We have build and deploy.
8:15:53Build is reusing the work the front- end
8:15:56build workflow that we have created uh
8:15:59in the CI pipeline. Now under deploy we
8:16:03have the name is deploy. This needs to
8:16:06run after the build process. And here
8:16:09under environment, we have production
8:16:12and we point it at the URL. If you have
8:16:15different environments, make sure to
8:16:16change this. But in our case, we have
8:16:19just production. And now here are the
8:16:21steps. We will start with the same steps
8:16:24that we have seen in the previous
8:16:26workflows. First, we check the
8:16:28repository. We download the front end
8:16:31artifacts because now in the in the
8:16:34deploy job we are going to take the
8:16:37files that were built in this job and we
8:16:39are going to put them in digital ocean.
8:16:43This is why we are using this GitHub
8:16:45action to download the front- end build
8:16:48from this specific path. Up to now as I
8:16:51said these are the these are the steps
8:16:53that we have been following before and
8:16:56the new step which is related to the to
8:16:58the deploy job is create backups. What I
8:17:02will try to do is first I will just
8:17:04commit this and then I will add the
8:17:06other step. So let me try to do this. So
8:17:10here because we are u this is something
8:17:13new I will add add deploy job and I will
8:17:17commit this. Now let me publish this
8:17:20branch and I want to publish it to
8:17:24basically my own fork. I don't want to
8:17:26push it to upstream because this is just
8:17:29a test repository to show you how things
8:17:32work. And now on GitHub I can click on
8:17:34this button to create the pull request.
8:17:37And here okay so let's see. No I don't
8:17:39want to I want to merge it with my own
8:17:41branch or yeah and here. So yeah, let's
8:17:44just keep things simple and let's just
8:17:47hit create put request. Now the first
8:17:50thing that will run is basically the CI
8:17:53job and then if everything passes, we
8:17:56can uh yeah, we can merge this and try
8:17:59the deploy job. But while this is
8:18:02running, let's go back to here and let
8:18:05me undo the changes to show you the
8:18:07first step that we are going to perform
8:18:10in this deploy job. As you can see this
8:18:13specific step is called create backups.
8:18:17And here I am going to create backups
8:18:19for both SQL light and mele search. But
8:18:23you can adapt this to create backups for
8:18:25other databases if you are using them.
8:18:28And we are using an new uh GitHub action
8:18:32called SSH action. This is an amazing
8:18:36GitHub action that you can use to
8:18:38connect using SSH. Here it is. If you
8:18:42want to read more about it and they have
8:18:44a few examples here in their GitHub
8:18:47repository, if you scroll down a little
8:18:49bit, you will find there this quick
8:18:51start example that you can just copy and
8:18:55uh paste in your workflow and you can
8:18:57just tweak a few settings there. And as
8:19:00you can see here for example in order to
8:19:01connect they are using a username and a
8:19:04password. But if you look at my own
8:19:07version I am not doing that because we
8:19:11have disabled the ability to connect
8:19:13using a password and instead we are
8:19:16using an SSH key because as we said this
8:19:19is more secure than using a password.
8:19:23Okay. And here we are using version
8:19:251.2.4.
8:19:27So if you find that there is a new
8:19:29version just go to GitHub look at the
8:19:31releases tab and see if there is a new
8:19:33release. If that's the case then updates
8:19:35the version. Here under environment we
8:19:38are specifying so we are adding mail
8:19:41search uh and we take that from the
8:19:44secrets and under with we have the host
8:19:48which is basically the IP address of our
8:19:50virtual machine. We have the username
8:19:52key and here we pass the environments.
8:19:55So here we have me search. We need this
8:19:57because we are going to communicate with
8:20:00the me search service or server in order
8:20:03to create the backups and under script.
8:20:06So this is the what I like about this
8:20:08GitHub action. This allows me to run a
8:20:10script inside the virtual machine. So if
8:20:13I collapse this, you can see that's the
8:20:15only thing left in this step. Let's
8:20:17let's uh go through it step by step.
8:20:20First we are creating some variables. We
8:20:24have backend directory, backup directory
8:20:28and how many backups we want to create.
8:20:30So here the maximum amount of backups is
8:20:33going to be five. But if you want you
8:20:35can change this. And here we create a
8:20:37time stamp. We are going to use this to
8:20:40create uh when we are going to create
8:20:42the backup. So that's the file names are
8:20:44going to be different. The the first
8:20:46thing we need to do is basically to
8:20:48create the backup directory because if
8:20:51we go to the virtual machine you will
8:20:53find that we don't have a backups folder
8:20:56inside web app. This is something new
8:20:58and using the make directory command we
8:21:01will create it and here you can see that
8:21:04we are using this option dashp. It's
8:21:08amazing because uh what it does that if
8:21:11the back if the backups folder doesn't
8:21:13exist, it's going to create it. But if
8:21:16it if it finds that we already have a
8:21:19backups folder, it's going to skip this
8:21:21operation. And after creating the
8:21:23backups directory, we are going to
8:21:26start. So the first thing here is uh so
8:21:29this block is going to is related to
8:21:32SQLite. You can see that in my virtual
8:21:35machine or in my project I have this
8:21:38database inside the backend directory.
8:21:41Let me go back to the virtual machine to
8:21:43show you how it looks like. As you can
8:21:45see I am now inside the virtual machine.
8:21:48Let me zoom in a little bit so that we
8:21:50can see this better. Let's type ls-la.
8:21:54Now let's go inside backend and let's
8:21:57see what we have. You can see that we
8:22:00have the visitors database. This is a
8:22:03SQLite database that I am using uh in my
8:22:06application. This is what we want to
8:22:08back up uh when we deploy. Let's go back
8:22:11to VS Code and let's see. So here we
8:22:15have an if block. We are checking if
8:22:18this file exists. If it does, we are
8:22:21going to move we are going to copy that
8:22:23file and we are going to create a uh
8:22:26this new file. And as you can see here
8:22:28we are using the timestamp as a suffix.
8:22:32So we have visitors underscore and we
8:22:34put the timestamp. This is helpful
8:22:37because after that we are going to sort
8:22:39these files and remove the older ones if
8:22:42we have more than five backups. Then we
8:22:46echo basically that we we were able to
8:22:49back up the database. Otherwise, if this
8:22:52file doesn't exist, we are going to show
8:22:54this warning to let us know that we
8:22:57couldn't uh create a backup. That's it
8:23:00for SQLite. It's simple. It's just a
8:23:02file that we copy. What happens if you
8:23:05want to do this with me search? It's a
8:23:07little bit tricky because we need to do
8:23:09more work. As you can see here, first we
8:23:12check if we have the me search key.
8:23:14Normally we should because as you can
8:23:16see we have passed it to the
8:23:18environments uh option here and if we
8:23:22look inside we are using this option to
8:23:25check if this environment variable does
8:23:27exist. If it doesn't we are going to
8:23:30exit and return an error. Now here is
8:23:33how we this is how we are going to
8:23:36basically make a backup. we are going to
8:23:39uh make a post request to this specific
8:23:43endpoint which is called dumps. And with
8:23:46this we we are going to tell me search
8:23:49to create a backup for us. But in order
8:23:52to do that we need to pass the me search
8:23:54key because if you remember we have
8:23:57created this server we have created a a
8:24:00API key so that no one can just make
8:24:04requests to the server. they need to
8:24:05have the API key first. After making the
8:24:08post request, we add a header that will
8:24:10pass that uh me search API key. And here
8:24:14you can see that we are running curl
8:24:17with the s option which is basically
8:24:19silent. And here we what we are doing
8:24:22here is that we are tell we are telling
8:24:24curl to fetch the HTTP code. And if you
8:24:27are not familiar with HTTP codes, you
8:24:30can go and search for HTTPS HTTPS status
8:24:34code and you can learn more about them.
8:24:37You can go here to Mozilla. They have a
8:24:40good documentation that explains how
8:24:43this works. HTTP status codes will tell
8:24:47you if if your request finished
8:24:49successfully or not. In this case, uh
8:24:52you can see here are the ranges. So we
8:24:54have something between 100 to 200 etc.
8:24:58and we are going to use this information
8:25:00to uh basically check if what we have
8:25:03done succeeded or not. Now let let me go
8:25:06back. So after capturing the HTTP code
8:25:10we will have all of that we will put all
8:25:12of that response inside this variable
8:25:15and from that we are going to get the
8:25:18HTTP code. After getting the HTTP code,
8:25:22we will uh compare it to see if it is
8:25:25within this range. So if it is greater
8:25:27than or equal to 200 and less than or
8:25:30equal to 300, this means that we made
8:25:33the request successfully. I can show you
8:25:36this. If if this seems complex, I will
8:25:39copy this and paste it in the virtual
8:25:42machine and let's see what we will get.
8:25:44First, let's try this um help endpoint
8:25:48because it doesn't require us to pass a
8:25:51an API key. You can see that we have uh
8:25:56-x get. This is not necessary because by
8:25:58default when you run a curl command, a
8:26:01curly request, it's going to be get.
8:26:03It's not we add the -x option when we
8:26:06want to perform other actions such as
8:26:09post. And here, let me just run curl
8:26:12directly to see what we get. So for
8:26:14example here we get status available but
8:26:17what happens if I add the -w option
8:26:20after curl let's add dashw
8:26:24and let's see what we get you can see
8:26:26that here let me retry the command you
8:26:28can see that here we have 200 and this
8:26:31is the the status code that we are
8:26:33looking for and let's put this inside a
8:26:36variable like we did uh in the deploy
8:26:39job so you can see that I can wrap
8:26:41everything around this Let's do it like
8:26:44this and let's add the variable name.
8:26:48And I need to close this like so. Now
8:26:51let me hit enter and let's see what we
8:26:52get. You can see that uh now we need to
8:26:55echo the content of this variable to see
8:26:58what it what it holds. Let me copy it
8:27:01and paste it here. And as you can see we
8:27:03are getting the same output. Here we get
8:27:05the response from that endpoint telling
8:27:07us that the M search server is up and
8:27:10running and 200 is the response that we
8:27:13got after making this HTTP request. Now
8:27:17let me copy this second line. Let's take
8:27:21HTTP code. Let's put it here and let's
8:27:24again echo that variable. So let me run
8:27:28echo and let's copy this and paste it
8:27:32below. Let's see what we get. You can
8:27:33see that we are getting 200. Why?
8:27:36Because here we are taking this output.
8:27:39We give it to tail and the tail will
8:27:41take the last line which is in this case
8:27:44is 200. And this is what we are getting
8:27:47here. And the next uh action or command
8:27:52is will get just so let me copy this.
8:27:55It's going to get the status. It's uh
8:27:58because here we are using the set
8:28:00command. It's going to remove the status
8:28:03code. It's going to take just the other
8:28:05part which is the response body. Okay.
8:28:07Uh as you can see the these commands are
8:28:10simple. They look complex but if you run
8:28:12them like what like uh I did uh you will
8:28:15find that um they are easy to understand
8:28:19and what they are getting is uh going to
8:28:21be used here to basically check if the
8:28:24request uh was made successfully or not.
8:28:27So as you can see here we got 200 as the
8:28:31status code. If we compare if we look at
8:28:34this condition it's true because the
8:28:36HTTP code is greater than or equal to
8:28:39200. So that means that we will show
8:28:42this in the console showing that the
8:28:45request was accepted. But in me search
8:28:48when we call for example the dumps
8:28:51endpoint it is not going to create the
8:28:53endpoints directly because it depends on
8:28:56the size of your database. So med search
8:28:59will give you an ID so that you can
8:29:01track the progress and uh this is what
8:29:05we are getting here in the response body
8:29:07we will try to look for the task UID.
8:29:11Let me try to do this so that so that we
8:29:13can see how this works. Instead of
8:29:16hitting the health endpoint, I'll try to
8:29:18hit the dumps endpoint to see the real
8:29:20output that we will get from uh that uh
8:29:23specific request. Let's run curl. Let's
8:29:26add dash x because as I said here we are
8:29:30going to perform a post request instead
8:29:33of a get request. And let me add this
8:29:36and let's see what we get. You can see
8:29:38that now we weren't able to create to
8:29:40make the post request because we are
8:29:42missing the uh yeah the header. So we
8:29:46need to add that. Here is the the other
8:29:48missing part. So let me copy it and
8:29:51paste it after this. Here is the full
8:29:55command that we can run. Now it should
8:29:57work because we are passing the API key.
8:30:00Let's run this and let's see what we'll
8:30:02get. Okay, here is the output and as you
8:30:04can see I've told you that me search
8:30:06will return the task UID and this is
8:30:10what we will try to get in this nest
8:30:13command. So if we go back here you can
8:30:15see that we have this variable that will
8:30:18take the response body and we'll search
8:30:20for the task UID using this uh
8:30:24expression and uh yeah if we print this
8:30:28we will get the task u ID. So again
8:30:31let's do the same thing. Let's store
8:30:33that information inside dump dump
8:30:37response and let's try to get the
8:30:38response body. Let's put the response
8:30:41here. Let me add this. Okay, that
8:30:44worked. Now let's take the response body
8:30:49and let's now run the task view ID.
8:30:52Let's try to print the content of this
8:30:54variable to see if we are getting the
8:30:56correct view ID. Okay, that didn't work
8:30:59because I forgot to add the other
8:31:01options. So, here I have just d - x. So,
8:31:05let me add the other ones dash s and
8:31:09dash w. Let me add them here. Okay. And
8:31:12now it should work. So, let's go back to
8:31:15response body. And now let's go back to
8:31:18task UID. Let's now try to echo. And as
8:31:23you can see, we got the ID. This is what
8:31:26we want to basically try to communicate
8:31:29with Misarch to keep asking it did the
8:31:32task finish or or not. Let's see how we
8:31:35can do this. So after getting the task
8:31:38ID here we are going to check if it
8:31:41exists. So if it doesn't we are going to
8:31:43throw an an error and uh basically fail
8:31:46the deploy pipeline. Otherwise, if we
8:31:48have the task ID, we are going to tell
8:31:51mainly search uh by making this curl
8:31:54command if the operation finished or
8:31:56not. So, let's see what we will get if
8:31:58we run this. So, let's do that. And
8:32:01okay, so here they say that the
8:32:03authorization header again is missing.
8:32:06So, we need to add that one. Let's add
8:32:08it like this. Let me clear the screen.
8:32:11Let's add the authorization header and
8:32:13let's see what we will get what we get.
8:32:15Okay. So the output is we have a lot of
8:32:18things but here you can see that they
8:32:20confirm that the ID exists and here we
8:32:22have the status which is succeeded. This
8:32:25is what we want to see but in the
8:32:27beginning it might be incued or
8:32:29processed or sometimes it might fail. So
8:32:31we are going to take this status
8:32:33information and we are going to check uh
8:32:36to check it inside the Y loop. Okay. So
8:32:38after making the request we are going to
8:32:41basically grab the status from the
8:32:44output. If it says succeeded like in
8:32:48this example, we are going to basically
8:32:50show that the dump completed
8:32:53successfully and we are going to break
8:32:55out out of the while loop. Otherwise, if
8:32:58it failed, we are going to fail the job.
8:33:02But if it is still uh processing, we are
8:33:05going to increase the attempt variable
8:33:08and sleep for 2 seconds. And here with
8:33:11the max uh amounts of the max retries is
8:33:14set to 30 but you can change this. It
8:33:17shouldn't take that much but this is how
8:33:19we are uh this is the condition that we
8:33:22have in the y loop. So if we reach 30 we
8:33:24are going to uh break. Okay. So now here
8:33:28we are inside this y loop. We can
8:33:30collapse it. And after that if for some
8:33:33reason we have exhausted these attempts
8:33:35we are going to exit. And now we can
8:33:38close this and we can focus on the else
8:33:41block. So if the status code is not
8:33:44within this range, we are going to exit
8:33:46because something bad happened. And
8:33:48finally we are going to run the this
8:33:52specific script that we have created to
8:33:55basically uh ensure that we keep at most
8:33:59five backups for SQL light and mele
8:34:02search. Okay. And uh yeah that's as you
8:34:06can see uh that's all we are doing here
8:34:09and this is this step is going to run
8:34:11whenever we want to deploy our
8:34:13application and this is very important
8:34:15especially if you have sensitive data in
8:34:18your database. Great. So this worked.
8:34:21Let's go back to GitHub. And the CI
8:34:23pipeline also worked without any
8:34:25problem. Amazing. Uh let me commit this.
8:34:28Let's say CI add backup job or step. Uh
8:34:35let's commit it and let's sync the
8:34:37changes. Let's go back again to the
8:34:39virtual machine and let's ensure that we
8:34:42don't have a backups folder. You can see
8:34:43that there is no backups folder and if I
8:34:46go to back end let's see so we have the
8:34:49visitors DB. Great. Now let's I will try
8:34:53to wait for the pipeline to finish and I
8:34:56will come back because I need to wait
8:34:58for this to finish uh in order to merge
8:35:02the p request so that I can go to the
8:35:05actions tab and uh click on the deploy
8:35:08uh job so that I can deploy the
8:35:11application. Okay, I'll wait for this to
8:35:13finish and I will come back. I am back
8:35:16and as you can see now we can merge the
8:35:18pull request because all the tests have
8:35:21passed successfully. I'll hit now
8:35:23confirm merge and okay that's done.
8:35:27Let's also go back here and uh let's go
8:35:31to master to fetch the new changes.
8:35:34Let's do that. Okay, great. So now we
8:35:37have the new deploy file. Let's go back
8:35:42to GitHub. Let's click here. And as you
8:35:45can see, we have merged the third pull
8:35:47request. Now let's now refresh the page.
8:35:51Let's go to actions. Let's click on
8:35:56manual deploy. This is what I explained
8:35:59to you. So now we have merged. As you
8:36:02can see, we have a CI step. It
8:36:04succeeded. Now we have a second step
8:36:07which is continuous delivery. And this
8:36:10is how we do this. So as I said we will
8:36:14do this manually. We click on run
8:36:16workflow and here are the two jobs.
8:36:19First of all we are going to build the
8:36:21front end. Once this finishes we are
8:36:24going to deploy the application. This is
8:36:26not going to run automatically. No
8:36:30GitHub will stop the deploy job and it's
8:36:33going to basically send us a
8:36:35notification to let us know if we should
8:36:38let this continue or not. And this is
8:36:41what we have configured uh earlier. You
8:36:45can set as many people as you want so
8:36:47that they can click on the deploy uh
8:36:50confirmation. And this is what I was
8:36:52talking about. You can see that we built
8:36:54the front end without any issue. That's
8:36:56great. But now we have uh you can see
8:36:59that they requested a review from us. We
8:37:02click on the button, we review. So we
8:37:04want to deploy to production. We confirm
8:37:07that and we hit approve. Let's click on
8:37:10that and let's see what will happen. So
8:37:13the deploy has started and you can see
8:37:15that we have four steps. Let's click on
8:37:17them to see what what is happening. So
8:37:19now we are creating the backups. You can
8:37:21see that everything went well. Nothing
8:37:24bad happened. Uh and we can read through
8:37:27the console. We can see these are the
8:37:28messages that are um that we see because
8:37:31we have uh we are using the echo
8:37:33command. And let's read through this.
8:37:35You can see that SQLite backup created
8:37:37successfully. Here is the task ID. Here
8:37:40is the HTTP status code that we got from
8:37:43media search. And at the end, we say
8:37:46that backup process finished
8:37:48successfully. Amazing. That's what we
8:37:50want to see. But let's confirm. Let's go
8:37:52to the virtual machine to see if that is
8:37:54correct. Uh before running the the
8:37:56deploy job, we didn't have a backup
8:37:58folder. But now, if I go back to uh web
8:38:02app, let's run ls-la.
8:38:06As you can see, we have the backups
8:38:07folder. That's amazing. Now, let's go
8:38:10inside that folder. Backups. Normally,
8:38:13we should have just one file because
8:38:15this is the first time we are running
8:38:17this deploy job. And as you can see,
8:38:19here is the time stamp. Now, what I will
8:38:22try to do is I will copy this file and I
8:38:24will paste it
8:38:26four four times because I want to have
8:38:29five files. when I will when I will
8:38:31deploy the next time and we are not go
8:38:34we are going to create the sixth file
8:38:35but we are going to remove one of the
8:38:38old ones because the script will keep at
8:38:41most five files let's use the copy
8:38:44command let's copy the file and here
8:38:47I'll just change uh 12 to let's say 13
8:38:52let's change this again to 14 15 and 16
8:38:57let's see how many backups we have 1 2 3
8:38:594 and five. Now let's go back to GitHub
8:39:03and I will try to rerun all jobs. Okay,
8:39:08this will trigger the deploy job again
8:39:10and it should create a new backup but
8:39:14it's going to delete this one because it
8:39:17is the oldest. Let's confirm this. We
8:39:19have the output and uh after uh this
8:39:23deploy job finishes I will run again ls
8:39:26and we will compare um the two lists.
8:39:30Let's wait for the build job to finish.
8:39:32We have the artifact. Great. And now
8:39:36let's hit deploy. Let's click on that
8:39:40specific job. We are checking the
8:39:42repository. We download the artifacts.
8:39:44And here we are creating the backups.
8:39:47And yeah, that finished. Let's see. So
8:39:50here you can see that we have this
8:39:51specific step. Starting cleanup
8:39:54retention policy is to keep the last
8:39:56five files. Well, let's confirm if
8:39:59that's correct. So let's run ls. And as
8:40:02you can see, we don't we no longer we no
8:40:05longer have this specific database. We
8:40:08have 13 up to up to 16. And now we have
8:40:11a new upto-date backup of the database.
8:40:15This is great. And if we want we can
8:40:18also check the dumps folder to see what
8:40:21we have there. Normally we should have
8:40:23multiple uh backups. So if we go there,
8:40:26okay, we don't have the permission, but
8:40:28we can run sudo and pass the password.
8:40:31And as you can see here, we also have a
8:40:33few backups. Awesome. So this is working
8:40:36really, really well. Let's keep going.
8:40:39This is not the end of the deploy job.
8:40:41We still have a few jobs to add. Let's
8:40:44click on here and let's create a new
8:40:46branch. Let's call it R sync. Okay,
8:40:50because this is what we want to do. Let
8:40:52me collapse these jobs. So I will
8:40:55collapse the checkout step, the download
8:40:59front end artifact and also uh the
8:41:02creation of the backups. And now we will
8:41:05add this new job that will sync the
8:41:09files from the from GitHub to the
8:41:12droplet. For that we are using SSH
8:41:15deploy and rs sync is a command that
8:41:19will sync the files. So basically it
8:41:22will take whatever we add in GitHub and
8:41:25make sure that we have that on the
8:41:27virtual machine and uh by default it
8:41:30will try to create a mirror. Basically,
8:41:32whatever we have on GitHub should be on
8:41:35the virtual machine. If we have new
8:41:36files on the virtual machine, the rsync
8:41:38command will delete them. But we don't
8:41:40want that because as you can see, we
8:41:42created a backups folder and this folder
8:41:45doesn't exist on GitHub. So, if we run
8:41:48this, we will risk deleting the backups
8:41:50folder and any other file that doesn't
8:41:53exist on GitHub. For example, thev file,
8:41:56that's another problem. That's why we
8:41:58created that rync exclude file to tell
8:42:01rync yes try to create the mirror but if
8:42:04you find something new in the virtual
8:42:06machine please keep it. You can also
8:42:09search for this new GitHub action that
8:42:11I'm using here. We pass the private key.
8:42:14We pass the host and user. So the host
8:42:18is the IP address. The username that I
8:42:21am using um in my virtual machine. Here
8:42:25source will basically copy everything we
8:42:27have here and the target is web app and
8:42:31here are the arguments that we will pass
8:42:33to the rs sync command. Uh so here by
8:42:36default we will delete and but we
8:42:38exclude these files and folders that are
8:42:40inside the rsync exclude command. Okay,
8:42:43you can see that this job is simple. So
8:42:46let's try to commit it and let's test
8:42:48it. Uh yeah, let's just call it ci add r
8:42:52sync job. Let's commit this and let's
8:42:56publish the branch. We want to publish
8:42:58it in to origin. Uh let's go back to
8:43:01GitHub. Let's click on pull requests.
8:43:04Compare and pull requests. Let's target
8:43:08my master branch. And yeah, let's just
8:43:11uh create the pull request. I will wait
8:43:14for the pipeline to finish and I will
8:43:15come back. And like uh the creation of
8:43:18the backups, I will deploy manually to
8:43:20confirm that the step is working. This
8:43:22is the best way to create complex jobs
8:43:25is to test each step independently to
8:43:28confirm that it works before moving to
8:43:30the next step. If you try to write the
8:43:32full pipeline, it's going to be yeah,
8:43:34it's going to be hard to test because uh
8:43:38multiple jobs will fail at the same time
8:43:40and it's going to be a nightmare. I
8:43:42highly suggest to work on complex
8:43:45pipelines like this. Create a step, make
8:43:48sure that it works and then move to the
8:43:50next one. Same as before, I'll wait for
8:43:52the CI pipeline to finish before hitting
8:43:54the deploy button. I am back. I can
8:43:57merge the pull request. Let's do that.
8:44:00And let's click on uh the yeah the
8:44:03GitHub action to uh trigger the R sync
8:44:07job to verify that it works uh
8:44:09correctly. Okay. Now again click on run
8:44:12workflow. Make sure to choose master. I
8:44:15forgot to mention this before but that's
8:44:17very important because master is
8:44:19protected and whenever we merge a new
8:44:21thing it is ready to be deployed. What
8:44:23we want to confirm in this step is that
8:44:26the backups folder shouldn't be deleted.
8:44:28This is uh very important. We can see
8:44:30that here we have a few backups and if I
8:44:33go back uh the backups folder shouldn't
8:44:36be deleted. Let's verify that this is
8:44:38the case. Uh okay let's confirm the
8:44:41deployment. Click on production and
8:44:44approve and deploy. Let's click on the
8:44:46deploy job. We now should have a new job
8:44:49which is uh going to run the rs sync
8:44:52command. And as you yeah as you can see
8:44:54here it is. Okay. So we can see that the
8:44:57rsync command is going through
8:44:58everything uh in the GitHub repository
8:45:01and it says done. Okay. Uh we we
8:45:05deployed successfully. But the question
8:45:06is do we still have the backups folder?
8:45:09Okay. We still have it. Let's see what
8:45:11we have inside it. Amazing. So, we have
8:45:14everything there. But if we have removed
8:45:16this rync exclude file, things will have
8:45:19changed. Uh but basically, you get the
8:45:21idea. If you want to keep uh files that
8:45:24are inside your virtual machine but
8:45:26don't exist on GitHub, you should try to
8:45:29add a file like this and specify it in
8:45:32the arguments. Now, the next step is to
8:45:36populate the environment variables.
8:45:38Okay. So first of all let me try to add
8:45:41the backups folder again to r sync
8:45:44exclude it was so it was backups uh save
8:45:47the file and exit. Now if we go to back
8:45:50end uh sorry let's go to back end let's
8:45:53run ls-a you can see that we have thev
8:45:57file and I have created this uh manually
8:46:00when I when we were testing the the back
8:46:03end but we can also create this uh
8:46:06directly from the deploy job because we
8:46:08have all the secrets we want and if we
8:46:11uh look at env.example example. You can
8:46:15see that this is the only thing we need.
8:46:17We just need to change the mini search
8:46:19master key because the other ones are
8:46:22not yeah they are not secrets. This is
8:46:25just the endpoint and uh this is the
8:46:27index name and the environments. It can
8:46:29be either development or production but
8:46:31the secrets uh is the master key. This
8:46:34is what we will try to add. Let's try to
8:46:37do that first. Let's go back here. Let's
8:46:41click on the repository. Uh okay. So we
8:46:45have merged. That means we need to go
8:46:46back again to master and let's try to
8:46:50pull the latest changes. Okay. As you
8:46:53can see we have added the new step here.
8:46:55Amazing. Now I will create another
8:46:57branch and here let's call it feature
8:47:00and let's say create environment file.
8:47:06This step is also small. Uh let's add
8:47:09this below uh the rs sync job. You can
8:47:14see that the name is create a the
8:47:16backend env file. Here we are again
8:47:19using the SSH action command because it
8:47:21allows us to run scripts. Here we are
8:47:24using cat and we are redirecting all of
8:47:26this content to this file we are
8:47:29creating. So this is the the endpoint we
8:47:32are this it's local that's why we are
8:47:34using local host. The index name this
8:47:37you can change it based on the name of
8:47:39your index. And here we are using that
8:47:42environment variable that we get from
8:47:43secrets to populate this environment
8:47:45variable inside this file. And finally
8:47:48environment is set to production. And we
8:47:50end the file. That's all we need to do
8:47:52to create a to create thev file. Uh
8:47:55let's uh commit this civil
8:48:00file. Commit this. Publish the branch to
8:48:04origin. Let's go to GitHub and let's
8:48:08create the pull request. Okay, we have a
8:48:11problem. I need to change this to ghost
8:48:13like this. Okay, great. Now, let's
8:48:15create the pull request. I'll wait for
8:48:18the pipeline to finish and then I will
8:48:20come back. Let's merge the pull request
8:48:23because the CI pipeline succeeded and
8:48:27let's test the deploy job. Click on
8:48:31actions. Let's find manual deploy. Let's
8:48:35run the workflow on the on the master
8:48:37branch and let's see if the environment
8:48:39file will be created. This is a simple
8:48:42job so it shouldn't fail but yeah it's
8:48:45always uh a great idea to test. Let's
8:48:48click on review deployments, choose
8:48:51production and deploy. Click on the
8:48:53deploy job and let's wait for the final
8:48:57step to run. As you can see it is there.
8:48:59So that means that we uh yeah we are
8:49:02using the latest version of that deploy
8:49:05file. The RC command has finished and
8:49:08now we and we we were also able to
8:49:10create the backendv file. That's
8:49:13amazing. Yeah, we can verify that we
8:49:16have the file there. Um let's see. So
8:49:19let's run ls-la.
8:49:22And as you can see, today is it's June
8:49:2528th, um 11:24 a.m. So that means that
8:49:29it's uh correct. Great. Let's go back to
8:49:33VS Code again. Let's switch to the
8:49:35master branch. Let's pull the latest
8:49:38changes.
8:49:40Now, we need to install the Python
8:49:42dependencies. Let's say that we added a
8:49:44new library that the back end depends
8:49:47on. If we don't install it, the back end
8:49:50will fail. So we need to add a step to
8:49:53install the dependencies. But the
8:49:54problem that we need to account for is
8:49:56that let's say that we have 20
8:49:58dependencies and uh we install all of
8:50:01those inside the virtual environment
8:50:03folder. Uh let me show you. Let's go to
8:50:06um v to the virtual machine. You can see
8:50:08that we have the virtual environment
8:50:10folder. Let's say that something bad
8:50:12happened and we were able to install 10
8:50:15libraries but the rest failed. If we
8:50:18change this virtual environment folder
8:50:21and um yeah this not all not all
8:50:24dependencies were installed this will
8:50:25cause a problem. So what we need to do
8:50:27is this step should be atomic meaning if
8:50:30it fails we need to roll back and reload
8:50:34uh the virtual environment. This means
8:50:36that we are going to take a copy of this
8:50:38virtual environment folder. We will
8:50:40rename it something else. Uh for example
8:50:42VNV uh backup. Then we will use pip to
8:50:46install the dependencies. That will
8:50:48create a new virtual environment folder
8:50:50for us. If uh everything went well,
8:50:53great. We will delete the backup folder
8:50:54that we created and we will keep the new
8:50:57one. But if something failed, we will
8:50:59roll back and reuse the the backup
8:51:02folder so that um we don't accidentally
8:51:06break the back end. First of all, we
8:51:08need to update the rsync exclude file.
8:51:12we need uh to uh update the backend
8:51:15section. Uh here are the things that we
8:51:18need to add. Remember that I showed you
8:51:20that we need to create the logs folder,
8:51:23the J unicorn start etc. And here we
8:51:26also have the socket. All these things
8:51:28do not exist on GitHub. And if we deploy
8:51:32that was the problem that I had is that
8:51:34these files and folders were deleted.
8:51:37But now I have added them here and uh
8:51:40yeah they shouldn't be deleted anymore.
8:51:42Um so I have gone through the steps that
8:51:44we um followed in the previous sub uh
8:51:47chapters and you can see that I have the
8:51:50socket file I have the scripts and I
8:51:53also have the logs folder and I have
8:51:55added them to rsync so that they won't
8:51:58be deleted anymore in the future and to
8:52:01uh also verify that the back end is
8:52:03working fine you can see that we can use
8:52:05supervisor ctl and here this is the
8:52:08application and it is saying running
8:52:10which is uh Great. This is what we want
8:52:12to see. And if I also open the browser
8:52:16and go to this website, you can see that
8:52:18it is working. If I click on blogs,
8:52:21yeah, they are coming in. And if we
8:52:23click on uh let's go to the conclusion
8:52:27and let's start clapping normally it
8:52:29should work. Yes, you can see that if I
8:52:31refresh the page, the the count uh was
8:52:34updated because the data is stored. Uh
8:52:37and yeah, it means that the back end is
8:52:39running. Maybe search is running.
8:52:41Everything is working perfectly. Okay.
8:52:44Now let's go back. We let's commit this.
8:52:47Uh let's say sure update r sync exclude.
8:52:52Uh let's commit this. Now we need uh we
8:52:54have created the branch. Great. Now we
8:52:56need to add the new job inside this
8:53:00file. Now let me add let me paste the
8:53:03code here. It's long but I'll try to
8:53:05explain it. uh to you. The new job name
8:53:08is called install Python dependencies.
8:53:11And here I have tried to emphasize that
8:53:14we are going to uh atomically swap the
8:53:17virtual environment. We use the SSH
8:53:20action GitHub action to run a script. So
8:53:24let's look at what we have here. First,
8:53:26we define this virtual environment uh so
8:53:29that we can go inside back end. And here
8:53:31is the first step. We want to create a
8:53:34backup of the virtual environment
8:53:36folder. But first, we need to check if
8:53:38that directory exists. If it does, we we
8:53:41use the move command to create a backup.
8:53:44And the new folder name will be virtual
8:53:46environment backup. After that, we will
8:53:49create a brand new virtual environment.
8:53:51We activate it and we use pip install to
8:53:55install the dependencies. And here so
8:53:58yeah if uh this command failed we are
8:54:01going to show the else case which is
8:54:03going to remove the virtual environment.
8:54:07This is the roll back strategy that I
8:54:09was talking about. If it if it if things
8:54:12doesn't work as expected we will fall
8:54:15through the else block. We will show the
8:54:18error to understand what happened and
8:54:20then we roll back. But first we need to
8:54:23deactivate the virtual environment. We
8:54:25remove it and then just to double check
8:54:28we check if the virtual environment
8:54:30backup exists. If it does, we move it
8:54:32back to virtual environment and we exist
8:54:35because the installation failed. But if
8:54:39everything went well, we will do the
8:54:41cleanup. So we have created a new
8:54:43virtual environment. It has all the
8:54:45dependencies. So now we no longer need
8:54:47the backup. We can just delete it. And
8:54:50at the end we echo that the virtual
8:54:52environment setup has completed. And now
8:54:55I can we can commit this um let's say
8:54:58pet and here let's say add uh virtual
8:55:02environment or let's change this to CI
8:55:05because this is what we are working
8:55:06with. We can commit and remember that
8:55:10the rsync exclude file is now looking
8:55:13like this. We can publish the branch. We
8:55:16go to origin and let's now click on pull
8:55:21requests to create another one that we
8:55:24can deploy to pass this new step in the
8:55:27job. Okay. Now let's change this to uh
8:55:31origin. Let's remove all of this. So
8:55:34let's see. So this is the branch we want
8:55:36to merge it with master. Okay. And here
8:55:39let's change the title to CI add virtual
8:55:42environment atomic setup. Click on
8:55:45create pull request. Now we just need to
8:55:48wait again because uh yeah we need to
8:55:51ensure that the CI passes before testing
8:55:54the new deploy job. I am back. As you
8:55:57can see we can merge the pull requests.
8:55:59Let's do that quickly. Now let's click
8:56:01on actions. Let's scroll down to manual
8:56:05deploy. Click on run workflow and hit
8:56:08run workflow. And at the same time I'll
8:56:11try to uh looking at the virtual machine
8:56:15to see if um when the backup folder will
8:56:19be uh created. But first let's wait for
8:56:22the build the job to finish and then we
8:56:25can run the deployment job in order to
8:56:29test the atomic virtual environment uh
8:56:33creation. Let's hit review deployment.
8:56:37Click on production and approve. Okay,
8:56:40let's click on the deploy job and here.
8:56:44Okay, let's do clear and ls-la.
8:56:49We are syncing the files. Let's hope
8:56:52that the logs folder and the other ones
8:56:53do not get deleted now. Okay. And now we
8:56:56are installing the dependencies. And as
8:56:58you can see, we have the virtual
8:57:00environment backup. And now we created a
8:57:03new virtual environment where we are
8:57:05going to install the dependencies. And
8:57:07we can confirm this by the date. You can
8:57:09see that this virtual environment was
8:57:11created uh a week ago. But now this one
8:57:14was created today. And we can also see
8:57:16that the logs file that the logs folder
8:57:19these files were not deleted because we
8:57:21have updated the content of the rync uh
8:57:25exclude file. Amazing. This is working
8:57:28really well. And you can see that the
8:57:30deploy job um finished successfully.
8:57:33Which means that now if I run ls-la,
8:57:35this folder will be removed. Let's
8:57:38confirm that. And yep, that's it. That's
8:57:40what we what happened. And you can see
8:57:43that the date on this folder is um June
8:57:4728th. This is working really well. Now,
8:57:49let's go back to VS Code here. I need to
8:57:53go back to the master branch cuz we need
8:57:56to pull the new changes. Let's hit okay.
8:58:00Um let's wait for a while. Okay, we have
8:58:03a new job here. Great. Now the next step
8:58:06is to uh reload the back end as well as
8:58:11the engineext configuration because if
8:58:13you have made changes to the front end
8:58:15or back end, we need to restart the
8:58:18application so that the changes take
8:58:20effect and for that we will let's go
8:58:23back to the website. You can see that
8:58:24here in the navbar I have so this is my
8:58:28name. I will try to change that and we
8:58:31will hit deploy and then come back here
8:58:33to the website to see if um yeah the new
8:58:36modifications uh are going to be live.
8:58:40Let's try to search for the navigation
8:58:42bar component and here. Okay. So, let me
8:58:46search for my name. Here it is. And I
8:58:49will add uh let's say 2026. Okay. So, I
8:58:53have changed the uh changed that file.
8:58:56Let me create a new branch. Commit that.
8:59:00Close the file. We will close the rsync
8:59:02excl exclude because we no longer need
8:59:04it. And now I will try to add this job.
8:59:09Uh let me indent this correctly. So it
8:59:12says restart the backend service. Here
8:59:15again we are using the SSH action GitHub
8:59:19action and inside this job. Here is what
8:59:22we are doing inside script. We check we
8:59:25try we run the restart all command using
8:59:29supervisor ctl. If this succeeds it
8:59:33means the back end restarted
8:59:36successfully. Otherwise we fail the job.
8:59:39Uh after running this we will wait for
8:59:41let's say 5 seconds so that uh we can
8:59:44run again the status command to check
8:59:46that the output will say running. This
8:59:49is another layer of confirmation that
8:59:52will tell us if the back end is is
8:59:54working. Okay. So we will try to push
8:59:57this and then I will add another job
8:59:59after this that will reload X. Let's
9:00:02commit ci restart back end and below
9:00:08this I will add uh let me go back here
9:00:12let's go down below and let's paste the
9:00:15new job uh I will collapse this because
9:00:19uh we have uh seen it and uh now we just
9:00:22want to focus on the reload engine x job
9:00:25here inside the script first we need to
9:00:27check that the configuration is correct
9:00:29if the configuration is not correct we
9:00:31are not going to reload EngineX because
9:00:33if we do then we are going to break the
9:00:35yeah we are going to break the website
9:00:37and we don't want to do that. If the
9:00:39syntax is okay we move to the next if
9:00:42block we run again here we run the
9:00:45reload engineext command. If this
9:00:48succeeds great otherwise we are not
9:00:50going to run it because the yeah the
9:00:53configuration is not correct. And at the
9:00:55end we show deployment complete. This is
9:00:58the last job in this deploy if workflow
9:01:02and if this succeeds we have um created
9:01:06uh a good CD workflow that when I hit
9:01:10deploy after changing my applications
9:01:12code it's going to be live in a few in a
9:01:15minute or two. Uh so here let's add
9:01:18reload enginex. Let's commit this. I'll
9:01:21publish the branch and create a pull
9:01:24request. Then I will wait for a few
9:01:26minutes for the pipeline to finish and I
9:01:29will come back. I am back. Let's merge
9:01:32the merge requests or the pull request.
9:01:34Let's now go to actions. Click on manual
9:01:39deploy. Let's uh where is it? Yeah. And
9:01:42here let's click on run workflow on the
9:01:44master branch. Let me split the page on
9:01:48two so that we can see the changes uh
9:01:51quickly. Okay. Now let's click on review
9:01:55deployments and let's approve and
9:01:57deploy. Let's click on deploy. We'll
9:02:00follow the steps. We are checking the
9:02:02repository. We download the front end
9:02:05artifacts.
9:02:06Here are the backups. Okay, we are are
9:02:10syncing the files.
9:02:12We have installed the dependencies
9:02:14without an issue. And now we are trying
9:02:17to restart the backend service. You can
9:02:20see that the message is saying
9:02:21restarting the fast fast API back end.
9:02:24Let's hope that this works. We have seen
9:02:27that we were able to stop and start the
9:02:30the backend service but we have a
9:02:33problem. So here it says sudo a terminal
9:02:36is required to read the password meaning
9:02:38that the v sudo uh change that we did
9:02:42didn't work. Let's try to investigate
9:02:45why. So here what was what was the last
9:02:49message we have seen pass API backend
9:02:52that restart is started successfully.
9:02:53Okay now let's go back here and let's
9:02:56search for that message. Okay so
9:02:58supervisor ctl start all worked well but
9:03:01I think what happens is that maybe the
9:03:04status uh is not allowed because here we
9:03:07need to run sudo. So let's verify that.
9:03:10Let's open the terminal again. Let's run
9:03:13sudo v sudo and let's see what we have
9:03:16inside this file. Let's scroll down.
9:03:19Okay, you can see that here we have uh
9:03:22yeah that's the problem. So here we are
9:03:23allowing just restart all. We don't have
9:03:26status. That's the problem. So what we
9:03:30can do is copy the command. I'll try to
9:03:33copy this. Let's go back to the
9:03:36terminal. And here let's try to add it
9:03:39after this one. Let's type user bin and
9:03:43I will paste this command. Supervisor
9:03:45ctl status all. Save the file. Exit. Now
9:03:51go back to uh let's go back to GitHub
9:03:54and let's restart the failed jobs. So
9:03:58here uh let me try to make the screen
9:04:00bigger to so that I can see what's
9:04:01happening here. I will click on rerun
9:04:04failed jobs. Let's uh rerun that. Let's
9:04:08now click on review deployments and
9:04:11let's hope that this time the we won't
9:04:14have an error when trying to reload the
9:04:16back end. Okay, now we have reached the
9:04:19other the step that failed before. But
9:04:21now as you can see it uh it didn't fail.
9:04:24That's great. And um okay, so now we
9:04:27have the same problem while trying to
9:04:30reload EngineX. And I think I know the
9:04:33um uh how to solve it. I think yeah we
9:04:36have the same issue if we look at here
9:04:40let's see so we have reload engine x but
9:04:43we forgot to add uh engineext-t because
9:04:47we are checking the syntax if we go back
9:04:51to vs code let's collapse this because
9:04:54it succeeded it didn't fail now we need
9:04:57to basically uh try to work on this one
9:05:00you can see that systemct ctl reload
9:05:02engine x is already here it systemctl
9:05:06reload engineext. But when we try to uh
9:05:10check the configuration, we run
9:05:12engineext-t.
9:05:14We need to also add this one. To fix
9:05:16this, we need to find where the binary
9:05:19of is located. But I think normally it
9:05:23is located at user sbin
9:05:26/enx.
9:05:28But we can verify that. I have saved the
9:05:31file and now I can run which engine x.
9:05:34And as you can see this is the path that
9:05:35I have added in b sudo and um okay let
9:05:39me just verify it one more time user/
9:05:42spbin/engineext and then I added the
9:05:45comma. Okay great. So now with this we
9:05:48won't um yeah we won't have an error
9:05:51when trying to run that specific step.
9:05:54Okay let's go back again to GitHub and
9:05:56let's try to rerun this failed step. Uh
9:06:00let's use rerun all failed jobs and uh
9:06:03again let's uh rerun this. I will
9:06:06refresh the page to get the confirmation
9:06:09is production and approve. Let's click
9:06:11on the deploy job. Hopefully this time
9:06:14it should work and then I will go back
9:06:16to this tab refresh the page and
9:06:18hopefully we should see that 2026
9:06:22uh will appear here. Let's collapse this
9:06:25rync step the create uh backups etc. Now
9:06:30we are installing the Python the Python
9:06:32dependencies. Uh we arrived at um the
9:06:36last steps. This one will not fail
9:06:39because we have confirmed that. Uh let's
9:06:41just hope that reloading engine X will
9:06:43also work and yes it did. Amazing. Uh
9:06:47let's uh look at this. So as you can see
9:06:49everything is green now. And as you can
9:06:51see here, GitHub will also give you the
9:06:53link so that you can click on it to see
9:06:55if everything uh is looking great. Okay,
9:06:58this is the moment of truth. Let's
9:07:00refresh the page. And as you can see, we
9:07:02have 2026. And this will work when um it
9:07:07doesn't matter if you change your front
9:07:09end or back end. Both will both uh will
9:07:11work. If you change your back end, we
9:07:13restart supervisor. We restart yes
9:07:16supervisor using the supervisor ctl
9:07:18command. And if your front end changes,
9:07:21we also reload the engine X. Awesome.
9:07:24So, as you can see, we have built the
9:07:26full pipeline. And this is how uh how I
9:07:28want you to build it. Don't try to write
9:07:31it fully. No, go step by step. Try to
9:07:34add one step at a time. Verify it
9:07:36because as you can see right now, we try
9:07:38to add these two steps. They fail. It's
9:07:40not a problem. We can go ahead and fix
9:07:43them before moving to the next steps.
9:07:46And once you do that you you learn how
9:07:48to handle errors and it's uh also
9:07:52amazing when you try when you fix the
9:07:54problem and see uh the green check mark
9:07:57uh it makes you happy and developing it
9:07:59like this is also fun. Uh so yeah we
9:08:02have finished the deploy job and this is
9:08:04how it works. So now in the future you
9:08:07have seen now that I was creating put
9:08:09requests and then I merged them. That's
9:08:11the first step. create your uh create a
9:08:14new feature branch. Develop your uh the
9:08:17yeah your feature or bug fix or
9:08:19whatever. Mer make sure that the CI
9:08:22pipeline passes. Then merge that request
9:08:26with your master branch. Then after that
9:08:29move to the second uh step which is to
9:08:32deploy. So go to actions and then hit
9:08:35the button on the master branch. Don't
9:08:37forget that. And uh after that your
9:08:40steps are going to go one by one and
9:08:42then your application will uh update
9:08:44accordingly with the new code that you
9:08:46pushed to master. And just like that we
9:08:49have finished chapter 5. It was a long
9:08:52one and uh this this was the I I would
9:08:55say the most interesting chapter because
9:08:57we have merged everything we have
9:08:59learned before and we try to automate as
9:09:01many things as possible. We have one
9:09:03more chapter left. In that one we are
9:09:04going I will it's a it's not a long one.
9:09:07uh I will just show you tools that you
9:09:09can install uh on your server so that
9:09:12you can monitor let's say uh HTTP
9:09:14requests that are hitting your server
9:09:17etc. So these are just tools that are
9:09:19nice to have and that are easy to
9:09:21install and configure.
9:09:23We have arrived at module 6. We are
9:09:26going to focus on observability. You
9:09:30will learn how to parse native engineext
9:09:33traffic using go access. I will show you
9:09:37how to deploy a dashboard where you can
9:09:40see this engineext data using HTTP basic
9:09:44authentication. We are going to learn
9:09:46about rotating logs and we are going to
9:09:50use a chrome job for that. We are going
9:09:53to visualize the CPU and memory usage as
9:09:57well as process trees using BTOP. And
9:10:01finally, I will show you how to find
9:10:04people that are or bots that are trying
9:10:07to get data from your virtual machine
9:10:11and we are going to block those IPs. Now
9:10:14let's go to the terminal and let's SSH
9:10:17to the virtual machine. Here is my
9:10:19terminal and as you can see I am inside
9:10:21the machine. I am going to start by
9:10:24installing go access. Here we are using
9:10:27ibuntu. We can install it very easily
9:10:30using this command sudo apt install go
9:10:34access. I already have it on my system.
9:10:36So that's why it didn't take too much
9:10:38time to install it. And how do we use
9:10:40this? Okay, let me show you. First, we
9:10:42need to try to look at the this path
9:10:46that stores the logs. You can see that
9:10:49here we have a lot of logs because we
9:10:51have just one website on this server. It
9:10:54is called my website_com.
9:10:57We are seeing this. But if you are if
9:10:59you have a virtual machine that is that
9:11:01is hosting multiple websites then pick
9:11:03one or multiples. So here we are going
9:11:06to pick my website. We want to look at
9:11:09the access logs and you can see that
9:11:11some of them are regular files but some
9:11:13of them were compressed and this is what
9:11:16I was talking about in the beginning.
9:11:18So, Linux is trying to rotate the the
9:11:20logs, meaning it it tries to compress
9:11:23them to save disk space. But first, we
9:11:26are going to look at just this one
9:11:28because it's a regular file that we can
9:11:30read. And later, I will show you to in
9:11:33how to include the zipped files also.
9:11:36Now, we know the logs are stored here
9:11:38and we know that we need to pick this
9:11:40specific uh website. Here is the command
9:11:43that we need to use. I will paste it and
9:11:46then I will explain what it does. Here
9:11:48we are using pseudo because we need to
9:11:49read files from this specific directory.
9:11:53And here I need to copy this and uh
9:11:56replace this placeholder with this
9:11:58specific file name. And here the log
9:12:01format is set to combined. This is the
9:12:04for the default format that enginex
9:12:06uses. So if you have if you are using a
9:12:09different format, make sure to change
9:12:11this option. That's all we need to do.
9:12:14Now if I hit enter, let's see what
9:12:16happens. You can see that we get this
9:12:18amazing in real time dashboard. This web
9:12:23server received 122 requests today. And
9:12:28here we have 54 unique visitors. Go
9:12:31access is showing you the data that is
9:12:34stored in the engineext logs. And here
9:12:37it is also shows you where it where it
9:12:40found the data. And this is the normal
9:12:43file that I was talking about. And here
9:12:45it shows you so the uh the unique
9:12:48visitors we I we have just one person
9:12:50that was that was hitting the the web
9:12:53server. We can keep scrolling or you can
9:12:55hit tab to move to the second uh tab or
9:13:00to the second um section. Here we have
9:13:03requested files. Some were requesting
9:13:06the site map. Some were just visiting
9:13:08the root directory. Here we have the
9:13:12static requests. So you can see that
9:13:14some were checking the robot cxt file.
9:13:17We don't have 404s. Okay. Here we get
9:13:20the IP addresses of people or bots that
9:13:24are trying to reach the web server. Here
9:13:28it shows you the operating systems. You
9:13:29can see that we have a lot of useful
9:13:31data that we can read and try to
9:13:34understand what is happening. And here
9:13:36it also shows you where the traffic is
9:13:38coming from. You can see that some were
9:13:41able to find the website from Google.
9:13:43Some were uh some traffic was coming
9:13:45from our website itself. And here we got
9:13:48the HTTP status codes per request. So
9:13:52most requests here succeed but some of
9:13:54them have client errors. Great. And as
9:13:57you can see all of this data is coming
9:13:59from one single file. And later I will
9:14:03show you how to increase this range by
9:14:07incorporating the files that were
9:14:09compressed. But before we do that, I
9:14:11will show you if you don't if you prefer
9:14:13to see a dashboard instead of sshing to
9:14:19the machine having to run this command
9:14:21to get this terminal dashboard. We can
9:14:24tell go access to generate a an HTML
9:14:27page that we can access from our
9:14:30browser. And this is what we are going
9:14:32to do now. But since this data contains
9:14:35sensitive information like IP addresses
9:14:38and other stuff we don't want everyone
9:14:40to see that page. So we are going to use
9:14:43HTTP basic authentication to block
9:14:46access to that specific page. So you
9:14:49need in order to see the data you need
9:14:51to provide a username and a password.
9:14:54And in order to do that here I will hit
9:14:57Ctrl C to exit. We need to install a
9:15:00tool called Apache2 utils because it has
9:15:05that functionality that allows us to
9:15:07create to generate an encrypted password
9:15:11file. Great. We have installed Apache to
9:15:15utils. Now let me clear the screen and
9:15:18let's use it. Here is how we use that
9:15:21utility. So here we tell it that we want
9:15:23to generate a as I said an encrypted
9:15:26password file and we will put that here
9:15:28in this location and here the user is
9:15:31going to be uh admin. I hit enter and
9:15:35now we need to provide a password for
9:15:37that specific admin user. Now I need to
9:15:40retype the new password and it says add
9:15:43in password for user admin. Now we can
9:15:46use this username and password to lock
9:15:50that specific page. But before we do
9:15:52that, we need to generate the go access
9:15:57html page. And in order to do that, let
9:16:01me go back to this specific command
9:16:04where we generate where we get the
9:16:07terminal dashboard. And we need to
9:16:09adjust a few things. The first part is
9:16:12going to stay the same. So we are
9:16:14pointing at a specific log file but we
9:16:16need to add the following option before
9:16:19log format. So here I will add this and
9:16:23as you can see here we add dash O which
9:16:26basically will try to generate this
9:16:30specific HTML page from the data. Once I
9:16:33hit enter, you can see that it is going
9:16:36to put it inside front end this. And
9:16:38this allows me to be able to use this
9:16:42file and to check it from my browser.
9:16:44And if I want, I can check the engineext
9:16:48configuration. It says okay. And now I
9:16:50can try to reload it so that engineext
9:16:54can serve that specific file. So let's
9:16:57go to Firefox. And here let's go to uh
9:17:01the website. And now because that
9:17:04analytics html file is in the this
9:17:08folder, I can access it. But you can see
9:17:10it didn't work here because I need to
9:17:12create I need to update the engineext
9:17:14configuration in order to serve this
9:17:17file. First let's u make sure that when
9:17:20we go to web app front end and this we
9:17:25need to ensure that we have the file
9:17:27there. As you as you can see we we do.
9:17:29So we have analytics.html HTML. Let me
9:17:33make sure that I have typed it
9:17:34correctly. Yes, I did. Now that's great.
9:17:37We need to open the enginex
9:17:39configuration
9:17:41and let me try to do that. We use the
9:17:45nano. So here I am using nano but you
9:17:47can use other text editors. And here we
9:17:49are pointing at my my website com inside
9:17:54the sites available folder. And here you
9:17:57can see that we have we need to come
9:17:59here. So here we are serving uh index
9:18:02html, we are serving assets and we are
9:18:06serving other stuff. So what are we
9:18:08going to do is we are going to add
9:18:11another location block but we are going
9:18:14to do that before the spa fallback
9:18:18routes. So we can do it maybe before
9:18:20assets or before index html. It it
9:18:24doesn't matter. Okay. Uh I will just
9:18:26copy the following code and I will paste
9:18:29it here. And now I need to try to indent
9:18:33this. And here you might see that I have
9:18:35changed something here. Root instead of
9:18:38specifying this path I have this v this
9:18:42variable that I'm using here. I did this
9:18:45because uh before I didn't have this
9:18:48coming soon page. So when I was uh
9:18:51because here I had I was serving other
9:18:53stuff. So I just uh changed this
9:18:56variable to point to this coming soon
9:18:59but basically it's just a variable that
9:19:01is holding a path and here so here you
9:19:04can see that root is pointing at web app
9:19:06front end this and this is why this was
9:19:09not working now I can save the file exit
9:19:14let's check engineext-t
9:19:17it says that the configuration is okay
9:19:19and now we can reload the page sorry I
9:19:22made a mistake Let's try to run this
9:19:25again and let's hit enter. Okay, now
9:19:27let's go back to the browser and let's
9:19:29refresh the page. And as you can see, I
9:19:32can't access that page because I need to
9:19:35provide my password. And this is what we
9:19:37are using in the configuration. I forgot
9:19:40to explain it. So let's go back to this
9:19:43file and let's see. You can see that
9:19:45root is pointing at the folder because
9:19:48there we have the analytics.html
9:19:51file. And here we add this o basic to
9:19:56tell engineext that this endpoint or
9:19:59this file is restricted. We cannot
9:20:01access it. And here is what we are
9:20:04using. We are using also and here is the
9:20:07encrypted file that we have created. So
9:20:09if the user or if the person that is
9:20:11trying to get this resource if they
9:20:13provide the correct user and password
9:20:16then serve this file for them otherwise
9:20:19don't serve it and show them this
9:20:22challenge which is what we are seeing
9:20:24here. Now I type admin because that's
9:20:26the user that I have created and here I
9:20:29need to provide the password. If I make
9:20:32a mistake let's say I remove the last
9:20:35character it's not going to let me in.
9:20:37So I need to provide the correct
9:20:39credentials in order to see that
9:20:41dashboard. And as you can see now I am
9:20:44in. So we need to wait uh for a few
9:20:46seconds. Um I'm not sure what is
9:20:49happening. Do we have an error here?
9:20:51Okay. Okay. Okay. So I see the problem.
9:20:53Remember that we are using content
9:20:55security policy which is restricting
9:20:59this project from loading some some
9:21:02data. That's why it is not working. If
9:21:05we go, so we need to fix this and this
9:21:07is really cool because as I told you
9:21:09when you apply the CSP policy, you have
9:21:13to keep attention to it and if you
9:21:16change your front end, you need to
9:21:18always test it to see if there is some
9:21:20new dependency that you added or
9:21:22something that is not loaded correctly.
9:21:25And this is what what happened here. But
9:21:27we can solve this easily by going back
9:21:30to the engineext configuration. And uh
9:21:33let's go up to see what we have. You can
9:21:35see that here we have the content
9:21:37security policy. And this is what we
9:21:40have done. And what we could do here is
9:21:42we can uh [clears throat] h have the
9:21:44header here just for analytics. HTML so
9:21:47that we can override this. Uh okay. So
9:21:50we can come here and I will I will try
9:21:54to add that. I will paste that here. Let
9:21:57me save the file. Exit. Let me check
9:22:00that the configuration is okay. And now
9:22:04I can reload. I will I will just make
9:22:07sure that uh the website is going to
9:22:09load correctly and then I'll try to
9:22:11explain to you what is happening there.
9:22:13Okay. So, we still have a problem here.
9:22:15It says that the page settings blocked
9:22:18an inline script uh because it violates
9:22:21something. Okay. Let's go back and let's
9:22:23see what we have here in the
9:22:24configuration. I think what I will do
9:22:26here since this is a page that only
9:22:30admins can access. I will add unsafe
9:22:34inline just so that we can see how the
9:22:36dashboard looks like. So here we have
9:22:38unsafe eval and after that I will add
9:22:41unsafe inline. But please never do this
9:22:44on your main production site because
9:22:46this is a page that is protected. it's
9:22:50not it's not going to be an issue.
9:22:52That's why I'm adding this and to just
9:22:55make sure that it is working. So here
9:22:57now I refresh the page and as you can
9:22:59see we get the dashboard. It's the same
9:23:01information that we will see in the
9:23:03terminal. But if you prefer to say that
9:23:06you want to see this from your phone or
9:23:08you don't have access to the virtual
9:23:10machine, you can use the dashboard to
9:23:13look at the data. And here you have you
9:23:16can for example open the settings. You
9:23:18can change the theme if you like. If you
9:23:20want a bright theme, you can do that
9:23:22here. And here you can see all the
9:23:25panels that we talked about. Here it
9:23:27shows you the unique visitors per day.
9:23:30It shows you the files that were
9:23:31requested,
9:23:33etc. And if you want to learn more about
9:23:35Go Access, you can go to their GitHub
9:23:37repository. They have a ton of resources
9:23:40and it's an amazing project. You can
9:23:42also if you want you can uh as I said go
9:23:45to GitHub and here they have their own
9:23:49example the so on their website they
9:23:51show how this might look like. So if you
9:23:54want to to come here to read the
9:23:57documentation uh that that would be
9:23:59great because they have a lot of
9:24:01features and if you click on this image
9:24:03on GitHub you see that they also have
9:24:06their own dashboard and you can see that
9:24:08they have a ton of information and you
9:24:11can also add maps. It's amazing. It's an
9:24:14amazing project. Great. Now let's go
9:24:16back to this to my website. You can see
9:24:20that we were able to add this and uh as
9:24:24I mentioned here we have we just did a
9:24:27shortcut just to be able to solve the
9:24:30problem with the CSP but as I mentioned
9:24:34and never ever use unsafe inline because
9:24:37that goes against security. Now let me
9:24:41show you how to do this correctly. First
9:24:43we need to add nons. Uh so yeah we need
9:24:47to add this. Let me show you. We need to
9:24:50add nons dash requests ID to the CSP
9:24:55only in the script source. Okay, because
9:24:58this is related to loading inline
9:25:00scripts. And after that, this was the
9:25:02missing piece. I forgot to add it. This
9:25:06is what we need to add to uh this
9:25:08location block. Here we are passing sub
9:25:11filter once off because we want
9:25:14engineext to go through the content of
9:25:16this file and we don't want it to stop
9:25:20uh when it finds the first occurrence of
9:25:22the things that we want it to search.
9:25:24And here here are the filters we are
9:25:25telling it to search for script tags
9:25:28that don't have attributes. We want to
9:25:30have to search for script tags that have
9:25:33attributes. So for example, this one is
9:25:35not closed but the first one is closed.
9:25:38And for these two we want engineext to
9:25:42include the nons value. And this is a
9:25:46dynamic ID or yeah request ID that is
9:25:49generated whenever we request this file.
9:25:52And because of security we want yeah we
9:25:56want to make sure to uh include this
9:26:00stuff. We are telling EngineX to replace
9:26:03these strings with these values and this
9:26:06should resolve that problem. So now I
9:26:09can save the file. Exit. Let's first
9:26:12make sure that the configuration is
9:26:15correct. Good. And now we can reload the
9:26:18page. Awesome. Now let's go back to the
9:26:20page and I will do I will do a hard
9:26:23refresh. We can do that by using control
9:26:26shiftr. And as you can see it is working
9:26:29now. and here. So yeah, we have just
9:26:31warnings. We can remove them. So we can
9:26:33see that we don't have any errors and we
9:26:35can confirm. This is great to play to
9:26:37try to play with this because once you
9:26:39add a CSP, you will have problems like
9:26:42these. And in order to uh understand
9:26:44them, you need to read the error that
9:26:46you are getting to understand what needs
9:26:48to be changed. And here for example, I
9:26:51can remove this and see if it is not
9:26:53needed. And now I can reload the reload
9:26:55the configuration, come back, refresh
9:26:58the page. As you can see, we were able
9:27:00to load the dashboard but not fully. And
9:27:02this is how you do it. Now we see that
9:27:05okay, we have a problem. So now we can
9:27:07zoom in and read a little bit. It says
9:27:09that here uh okay so the pages settings
9:27:12blocks an inline script and here from
9:27:15being executed we can also click on this
9:27:18specific file and as you can see here it
9:27:20shows us that we have this specific
9:27:23script and what do you see? We have we
9:27:26have the type attribute. So it's not a a
9:27:28closed script like the one we have here.
9:27:31Let's go back to the configuration. You
9:27:34can see that here the scripts that we
9:27:36captured is closed. Here it is closed
9:27:40but this one was not we weren't able to
9:27:42load it because now it has an attribute
9:27:45and that was the second thing that I've
9:27:47added in order to capture that. So let
9:27:50me go back and paste the file and here
9:27:52let's indent it. And uh we have seen how
9:27:55to work with CSP. But I just wanted to
9:27:57remind you how we do it. Uh you can use
9:28:00LLM to help you with this. Just copy the
9:28:03console error, give it your current
9:28:05configuration and tell it to give you a
9:28:07secure CSP. Sometimes it might tell you
9:28:10to uh do the what we did before to get
9:28:13rid of the errors quickly, but that's
9:28:15not secure. So try to tell it to give
9:28:18you a secure CSP and that's also going
9:28:21to be very helpful. Okay. Now we can
9:28:24save this file after adding this sub
9:28:26filter. Let's reload it. I know that the
9:28:29configuration is correct. And let's
9:28:31refresh. And as you can see everything
9:28:34is working perfectly. Amazing. Now let's
9:28:36go back to the terminal because now I
9:28:38want to talk about log rotation. We have
9:28:41seen that when we did ls and looked at
9:28:45the var directory, we found that we have
9:28:49a lot of zipped files here. And if we
9:28:52want to use them, it's not possible. So
9:28:56that's why here in the dashboard, we see
9:28:58just what is happening today. But let's
9:29:00say that I want to see what happened
9:29:02during the this month or uh during the
9:29:05past year. I need to change the command
9:29:07a little bit. So let's see how we can do
9:29:10this. For that, we have a utility called
9:29:13zeb uh grap. You can search for it.
9:29:17Normally it is installed on Linux. So uh
9:29:20you can use it and this is how it works
9:29:22and it allows you to search inside the
9:29:25zipped file. So you don't need to unzip
9:29:27them and then search within them. This
9:29:30utility allows you to uh search for
9:29:32patterns within the file directly and
9:29:35it's super useful. We can try we can try
9:29:38to do that uh in order to show you how
9:29:40that works. I will copy the full command
9:29:42and then I will show you what it what it
9:29:45does. Let me go back to the terminal. I
9:29:48will paste the command here. I need to
9:29:50change that placeholder using this
9:29:52website. Let's do that quickly. Okay.
9:29:56Let's see what we have here. Here you
9:29:58can see that we are using a wild card.
9:30:00This means we are going to capture every
9:30:03file that starts with my
9:30:05website_-access.log.
9:30:09And this will capture the normal file
9:30:11and the other ones that are compressed.
9:30:14Now let's look at this Z grab-h
9:30:17dot. If we go back to Google, here we
9:30:20see the this uh AI generated content. It
9:30:23says that here we pass the options and
9:30:25the term that we want to search for. And
9:30:27here we put dot because we want to
9:30:29capture the whole file because that's
9:30:31what uh go access is going to use in the
9:30:34dashboard. And here we pass the dash h
9:30:37option in order to basically tell it to
9:30:41hide the file names. And here if you
9:30:44search online you'll see that's the this
9:30:47confirms it. So dash h means d- no file
9:30:51name. And here yeah it's it's not going
9:30:53to show the uh the file names in the
9:30:55output. But that's how it works. So you
9:30:58you give it the the files. Here we use
9:31:01the wild card to capture all the the
9:31:04files and here we pass that to go access
9:31:08and here the dash is telling go access
9:31:10to read the unccom log data from the
9:31:14standard input because here we are using
9:31:16pipe it's going to take the output from
9:31:18this and give it uh to go access uh from
9:31:22the standard input again so we here we
9:31:25have - o web app and we put that inside
9:31:27the and we use the combined log format
9:31:30that and here we get a problem. So it
9:31:32tells us permission denied. That's uh
9:31:35understandable. So we need to add sudo
9:31:37and uh here is what it is saying we get
9:31:41permission denied because when we do
9:31:43sudo like this it is going to be applied
9:31:45to just z grab and uh after piping that
9:31:49we are going to use that permission. So
9:31:51we need to change that command to uh to
9:31:55look something like this. We need to run
9:31:56it inside this shell execution script.
9:32:00We are using p sudo and we use bash and
9:32:03we pass it the full command that we want
9:32:04to that we want to run. And this will uh
9:32:07use the pseudo privileges. And this
9:32:09normally should work. And as you can see
9:32:11we don't get the permission denied error
9:32:13like we did before. And now since this
9:32:16is inside the this folder I don't need
9:32:18to re to reload the engineext
9:32:20configuration. I can just update yeah
9:32:22refresh the page. And as you can see the
9:32:25the duration changed. So now it is
9:32:28saying 28th of June to 12th of July
9:32:32which is amazing. Uh normally it will be
9:32:352 weeks max. But we will solve this
9:32:37issue because Linux is going to first
9:32:40compress the files but after two weeks
9:32:43it is going to download uh to remove or
9:32:45to delete old files. And uh yeah now you
9:32:49can see we get more data. Now you can
9:32:51see the number of visitors per day and
9:32:54the number of hits. A visitor can hit
9:32:57the server multiple times. That's why
9:32:58the number of hits is usually higher
9:33:00than the number of visitors. Amazing.
9:33:04But what do we want is that we want to
9:33:06automate this generation process. We
9:33:08don't want to log to the virtual machine
9:33:10and every single day generates this uh
9:33:14updated uh static HTML file. We want to
9:33:18automate this and we know how to do how
9:33:20to do that. So we are going to use chron
9:33:22tab and we are going to add uh this uh
9:33:25file or this command to the end of that
9:33:27file. Okay, let me copy the full command
9:33:30and I'll paste it here. Here we are
9:33:33going to run it uh hourly. So that's why
9:33:37we have zero here. It's going to make
9:33:39sure to run zap every hour for us. And
9:33:43now the only thing I need to change is
9:33:45this. Let me save the file and I'll just
9:33:47copy that from here. Let's go back to
9:33:50Chrome tab and let me change this and
9:33:54paste it here. Great. Now we can save
9:33:56the file. Exit. And now this is going to
9:34:00automatically generate the dashboard.
9:34:03And let's say tomorrow if I want to see
9:34:06um the stats. I can just go back and uh
9:34:10refresh the page. But we still have a
9:34:12problem where Linux is deleting old
9:34:15files. But we are going to come back to
9:34:17that issue and resolve it. For now, we
9:34:19will uh stop a little bit and we will go
9:34:22to GitHub. I want to show you an amazing
9:34:25project that you will find very useful.
9:34:27It's called Btop. So, if you go to Btop
9:34:31GitHub, it's a way to monitor resources
9:34:34and it's super super amazing. The
9:34:37application that they have developed
9:34:39looks very nice. What I want you to do
9:34:41is open the releases page and here the
9:34:44last version is 1.4.7 4.7 and here they
9:34:47have the pre-ompile binaries uh that we
9:34:50can download but since we are here in
9:34:53the terminal we need to use the wget get
9:34:56command to download that so I'll just
9:34:58download this one and I will use um
9:35:01first let's use ls okay we have the file
9:35:04there I will use tar to unzip that okay
9:35:07now we have btop so we need to go inside
9:35:10there sorry let's go inside btop and
9:35:14this is the source code we can just use
9:35:16pseudo make install to compile this
9:35:19project. Okay, you can see that this is
9:35:21the installer and it installed the
9:35:24binaries uh in their respective folder
9:35:27so that we can just type btop and uh the
9:35:29program uh will run without a problem.
9:35:33So now if we type btop it says that okay
9:35:35we need to resize the window and uh
9:35:38voila as you can see we have everything
9:35:40here but sometimes
9:35:43Linux will not allow btop to get uh
9:35:46sensor data. So for that we can you can
9:35:50run this command. So if you get that
9:35:51problem you run pseudo set cup and that
9:35:55will allow btop to read the sensor data
9:35:59without an issue. After doing this, we
9:36:02can go back one directory and remove the
9:36:07the the files that we have inst uh
9:36:09downloaded because now btop is running
9:36:12without an issue. And as you can see,
9:36:14btop is a replacement for htop or top or
9:36:19any uh one of these commands because
9:36:22here you can see yes, we get the same
9:36:24information but it's hard to read
9:36:26visually. Uh it's hard on the eye. But
9:36:29Btop is uh is an is an amazing uh
9:36:32terminal user interface. As you can see,
9:36:35it looks really clean. Let me zoom in a
9:36:37little bit just to see what is happening
9:36:39here. You can see on top we have the
9:36:41CPU. Since this is a lowend virtual
9:36:46machine, we have just one core. Uh here
9:36:48is that CPU. We have uh 512 MGB of RAM.
9:36:54You can see is here. We see that
9:36:55currently we are using 48% of memory and
9:36:59uh 50% 50 52% is available. We also have
9:37:03swap. This is what we have created in
9:37:05the beginning. So we are allocating 2 GB
9:37:08from disk as a swap memory to help us
9:37:11because we don't have a lot of uh RAM
9:37:13and uh here so we have disk we have 8 GB
9:37:18and here we have network. So it show it
9:37:20shows us download and upload speeds and
9:37:23here we have the processes and here you
9:37:26see you can you can filter you can uh
9:37:30switch between tree view so if you hit E
9:37:33it's going to show you individual
9:37:35processes uh but if you want to see the
9:37:38tree you can hit E to yeah yeah group
9:37:41the processes uh like so if you want to
9:37:45sort you can use the arrow keys
9:37:48currently we are sorting by memory but
9:37:50you can sort with other stuff. I usually
9:37:52keep it to memory because here as you
9:37:54can see I don't have a lot of it. So if
9:37:56I see that my server is struggling I
9:37:58would like to see what is the process
9:38:01that is using a ton of resources so that
9:38:03I can go here maybe hit K to kill that
9:38:08specific process if it is using too much
9:38:10memory and uh is making my application
9:38:13work u poorly. Okay so that's how we do
9:38:17it. You can also filter. So for example
9:38:19here I you know that we are using J
9:38:21unicorn. So we can look at that. And as
9:38:24you can see here we have all of this and
9:38:26we can hit E. And here you can see that
9:38:29our application is using these three J
9:38:33unicorn workers because that's what we
9:38:36have specified. But here you can see
9:38:38that we have some orphaned uh processes
9:38:41that are left here maybe when we were
9:38:44reloading the the back end but we forgot
9:38:46to delete them and these are consuming
9:38:49memory. So what we can do here we can
9:38:51just hit K and kill them one uh one by
9:38:55one because you can see that some of
9:38:56them are using 5 MB of memory and they
9:38:59are doing nothing and we know that here
9:39:03when we set up the J unicorn file we
9:39:06have specified that we want to have
9:39:08three Unicorn workers that's why we have
9:39:11three of them there and you can see that
9:39:13they are grouped by the master J unicorn
9:39:16process that's how you know if those are
9:39:20orphaned or not. And we can also verify.
9:39:22So if I go back to my website, it should
9:39:26I yeah it should work without any
9:39:28problem. So if I go back here, you can
9:39:31see that here I have the coming soon
9:39:33page. But that's just something that I
9:39:35have added recently. So if I open
9:39:37another terminal window and update. So
9:39:41let me show you what I'm talking about.
9:39:43Here I have this maintenance file just
9:39:46to allow me to change between this page
9:39:49that you are seeing right now and uh
9:39:51this page that contains my uh website.
9:39:56So let me reload EngineX. Let's go back
9:39:59refresh the page and you can see I get
9:40:00the website. And remember last time we
9:40:03did the change where we added 2026 here.
9:40:05And if I go back to blogs you can see
9:40:07that the back end is still running. And
9:40:10uh we can also verify something because
9:40:14now if I try to kill one of these
9:40:16workers what will happen? Let's see. It
9:40:18is going to be killed. But as soon as
9:40:20that happens J the manager so this
9:40:24process manager is going to see that hey
9:40:27we have two workers now but uh the the
9:40:31configuration says that I need to have
9:40:32three workers. So it is going to spin a
9:40:34new one to uh fulfill that. If you don't
9:40:38remember where we have that, you can
9:40:39always go back to web app. Let's go
9:40:43there. We open backend and let's go to
9:40:47back end. Let's use cd instead of ls.
9:40:51And let's see here we have you can see
9:40:54that we have j unicorn start. Let's look
9:40:56at that file. And as you as you can see
9:40:59here we have specified that the number
9:41:02of workers should be always three. And
9:41:06here what when we are on J unicorn it is
9:41:09going to uh ensure that we have three
9:41:13UVORN workers. That's why when I tried
9:41:16to kill uh one of these workers I I
9:41:21don't see two. So one gets deleted and a
9:41:24new one comes in. And how do we know
9:41:27that? Just look at the process ID. So
9:41:29here is the current process ID. Let me
9:41:31take a screenshot quickly just to verify
9:41:34that this is the case. Um here is the
9:41:37screenshot. I will try to kill it again.
9:41:40And here is the number. So you see
9:41:42before it was uh 827 but now it changed
9:41:45to 990.
9:41:47And that's amazing. This is the what we
9:41:50have in the configuration. And now if I
9:41:52go back and refresh the page, you can
9:41:53see this is still working because and uh
9:41:57these processes are what's handling the
9:42:00the requests that are coming from the
9:42:03browser. And you can see this is why
9:42:05Btop is amazing because it allows you to
9:42:07interact with it uh using your mouse and
9:42:10keyboard simply instead of top and other
9:42:14programs that are hard to use. Okay, I
9:42:17have now showed you how to use Btop.
9:42:20Now, let's talk about identifying
9:42:24bots and uh people that are trying to do
9:42:28malicious stuff with your server. Okay?
9:42:31And let me also show you how to
9:42:35understand what I'm talking about. So,
9:42:37if I go back to analytics.html,
9:42:40you can see it is still working even
9:42:42though we changed to my website. You can
9:42:45see that if we focus on requested files.
9:42:49Let me see if I can zoom in. Yes, we
9:42:52can. But I think what we can do, we can
9:42:54also hide some panels. So we can see
9:42:58that here we have unique visitors. Can
9:43:01we hide that? Yes, we can. Okay. And now
9:43:03we have requested URLs. Here you can see
9:43:06that this is what we currently have.
9:43:08Some people are trying to get the site
9:43:10map. Okay, that's fine. But look at
9:43:11this. Someone is trying to do some fast
9:43:14traversal inside the virtual machine.
9:43:17This is malicious. This is not something
9:43:18that a normal user will do. We can keep
9:43:21looking. This one is this person is
9:43:23trying to login somehow. Uh let's see.
9:43:26This person is trying to get Yeah, you
9:43:29can see there are a lot of malicious
9:43:31things. This guy is trying to get
9:43:33password and u some configuration
9:43:36environment files etc. These are not
9:43:40normal users. Basically, there is a
9:43:42person or a group of people that are
9:43:45building programs that are going to the
9:43:49through the internet and trying to find
9:43:51web servers. Once they they find your
9:43:53web server, they try to find if there
9:43:55are vulnerabilities. Basically, if you
9:43:57are not good at security, you might leak
9:44:00some files. And if they find that you
9:44:03are leaking some passwords or some
9:44:05common files that we see here in the
9:44:07requested files, they will try to uh get
9:44:11as many things from you as possible. You
9:44:13can see that here they are also trying
9:44:15to get some AWS credentials or stuff
9:44:17like that. And these bots they might hit
9:44:20your server a lot. So if you if you see
9:44:23here here so for example this is a new
9:44:25web server. I am not getting a lot of
9:44:27hits but this will increase from time to
9:44:30time but we want to solve this. We want
9:44:32to block these people because they are
9:44:35just wasting bandwidth. Now let's see
9:44:37how to do this. We are not going to do
9:44:38it manually. Meaning I am not going to
9:44:40come here. Okay look at this request
9:44:44come down see which IP uh because here
9:44:47we have IP addresses. You can see that
9:44:50here we have the IP addresses. So, we
9:44:52need to correlate them together and see,
9:44:54okay, this IP address is hitting my
9:44:56websites a lot. Like, look at this one
9:44:58for example. It says hitting the website
9:45:01mo almost 2,000 times. That's inhuman. A
9:45:04human person will not go through your
9:45:06websites and try to refresh the page
9:45:082,000 times in uh the span of two weeks.
9:45:11So, that's how you know that you have
9:45:13some outliers, some uh automated
9:45:16programs that are trying to get
9:45:18something from your website if it if
9:45:20they found it. So here this our security
9:45:23is great. So and we don't here we are
9:45:25not using PHP. So the resources that
9:45:29they are trying to get are not in on the
9:45:31website. So that's fine but just so that
9:45:34uh we make their life hard. We can't
9:45:37block them. So let's see how to do this.
9:45:39Now we understood the problem. Let's see
9:45:40how to solve it. Let's exit now from
9:45:43Btop. We no longer need it. You can do
9:45:46two things. You can either look at the
9:45:48dashboard here and see the the IP
9:45:51addresses that are that have the high
9:45:54amount of hits or we can ex uh tell go
9:45:58access to give us a JSON file. So if we
9:46:02do sudo go access let's look at this
9:46:05command we can change it a little bit
9:46:08here we are looking just uh at the the
9:46:11today's log and here we do dash o to
9:46:15specify the output but we can here in
9:46:18this case it is an html file but we can
9:46:21also generate a JSON file instead of
9:46:25html let me do that I'll go here and I
9:46:29will remove this path and replace it
9:46:31with this one. You can see that I have
9:46:33parsed the log file and I have generated
9:46:36a JSON inside uh the web app folder.
9:46:40Let's ensure that we have the file
9:46:42there. And yes, we do. So now we can
9:46:45write a simple Python file to analyze
9:46:49this data. Here is the content of that
9:46:51script. And let's see what we are trying
9:46:53to do. First we we load the file. So if
9:46:56we find this great we will have this
9:46:58data dictionary and otherwise we exit if
9:47:02we don't find it we we load the hosts
9:47:04and from that we get data and we will
9:47:07print the IP addresses. I will save the
9:47:10file and I will run it just to see what
9:47:13we will get. Okay so this is what we are
9:47:16getting. This is the traffic that we get
9:47:18today. Okay. So these are all the IP
9:47:21addresses that uh reached the web
9:47:25server. And here you can see for example
9:47:27this IP address sent 76 uh hits today
9:47:32and these these uh these ones send just
9:47:36three four five. Okay. So these are low.
9:47:38That's not a problem. Even 76 it's not
9:47:40that big of a problem. But if I go to my
9:47:44own personal website which which is um
9:47:48an old [clears throat] one uh it has
9:47:51been deployed for more than a year. In
9:47:53that case I have some cases where one IP
9:47:57addresses one IP address will send in
9:48:006,000 requests 10,000 requests. Those
9:48:03are clearly automated bots that I
9:48:06usually block without thinking about
9:48:08that. But let's say that this IP address
9:48:11uh is a bot. It's not a normal user.
9:48:14What we will do? We analyze this output
9:48:17and we see okay so these are not doing
9:48:19bad things or they are not wasting
9:48:22bandwidth. We can keep them now but if
9:48:24in the future they change their behavior
9:48:26we can spot that. So now we copy the IP
9:48:29address and we use the uncomplicated
9:48:32firewall to plug them. So we use sudo
9:48:35UFW.
9:48:37We type deny from this IP address to any
9:48:41and this will block that specific IP
9:48:43address. And you can see UFW says that
9:48:46we have a new rule. Let's confirm that.
9:48:49I will clear the screen and I will run
9:48:51sudo UFW status. And as you can see, we
9:48:56denied this IP address from accessing
9:48:59anything from the web server. So now
9:49:02when they try to make a request is going
9:49:04to be dropped. it's not going to reach
9:49:06Ajax at all and that's what we want to
9:49:08see and this is how we block nefarious
9:49:10actors that are trying to do us harm.
9:49:14This was the first sub chapter we have
9:49:16talked about observability. Uh but in
9:49:19the next sub chapter we are going to
9:49:21focus on reliability. That's one is also
9:49:24important. We are going to try to put
9:49:27your server under stress. Meaning that
9:49:30you need to know this virtual machine
9:49:33how how much traffic can it support. So
9:49:36if I have 100 people who are trying to
9:49:39access the website at the same time,
9:49:41will my application be able to handle
9:49:43that? I cannot just say it like this. I
9:49:45need to test that. I need to put my
9:49:47server under stress to confirm that yes,
9:49:50we can support that. Otherwise, no, we
9:49:53need more resources. So we need to use a
9:49:55tool that will help us answer this
9:49:57question. We will also see solve the
9:50:00problem with engineext logs. So we we
9:50:02are going to tell index hey please don't
9:50:03delete the files that are older than 14
9:50:06days. Instead delete files that are
9:50:08older than a year or something like
9:50:09that. And I'll also show you a tool that
9:50:12will that will allow you to analyze the
9:50:14disc the disk space to see the uh what
9:50:19is what are the things that are
9:50:20consuming a lot of storage so that you
9:50:22can delete them if you want to. and uh
9:50:25more things. So that's what we are going
9:50:27to talk about in the next subchapter. In
9:50:30this sub chapter, we are going to talk
9:50:32about reliability. Let me show you what
9:50:35I mean by that. I've mentioned that we
9:50:37need to simulate concurrent user traffic
9:50:40locally using a any tool. And in the
9:50:44Python world, we have locust, which is
9:50:46an amazing tool that will allow us to
9:50:49simulate a lot of people trying to
9:50:51access our website. to answer the
9:50:53question, can my website handle x amount
9:50:56of traffic? So with locos, we have a
9:50:59class called fast http user that will
9:51:02allow us to boost execution capacity by
9:51:05five times. So that's something that I
9:51:06want you to use to benefit from it. And
9:51:09we are going to use this tool called
9:51:11NCDU to analyze the disk space to see
9:51:16what are the things on our virtual
9:51:19machine that are consuming space to
9:51:20determine if we can delete it or not. If
9:51:22we are running out of storage of course
9:51:24we are going to use systemd journal D.
9:51:28So we are going to enforce memory
9:51:30constraints and build a monthly
9:51:33automated cache and package cleanup
9:51:35system. So we want the system to remove
9:51:39things that are normally not used by our
9:51:42application monthly. So this is just
9:51:44really uh something that we can add and
9:51:47it's going to run automatically uh for
9:51:49us to save us some space. Amazing.
9:51:52That's what this these are all the
9:51:53things that we are going to look at. Now
9:51:56let's go to the terminal to apply this.
9:51:58So last time we saw that uh the last
9:52:01thing we did is to block an IP address
9:52:04that is trying to do bad things to our
9:52:07virtual machine. Okay. Now we can
9:52:10continue from this. We need to install a
9:52:12tool called locust. But we need to be
9:52:16careful. We should not install that here
9:52:18on the virtual machine. Instead we need
9:52:21to install it on our laptop. Because if
9:52:24you install it here on the virtual
9:52:25machine and you point it at your server,
9:52:28the data that you are going to get is
9:52:30going to be incorrect because Locos is
9:52:33going to consume some memory and your
9:52:35web server is trying to fulfill all
9:52:38those requests that are sent by locos
9:52:40and it's going to fail or it's not going
9:52:42to fail but it's not going to handle all
9:52:44of them because locos is there is
9:52:46consumer uh CPU and memory. So we want
9:52:49to put that on our laptop and then
9:52:52perform the the test. And in order to
9:52:55install it, you use pip install locust.
9:52:58That's how we do it. And um after that,
9:53:02you create a Python file uh locally to
9:53:05perform the test. Okay. I will open the
9:53:07terminal, a new terminal locally, go to
9:53:10the correct directory and I will come
9:53:13back. I am back and as you can see, I
9:53:16have opened a new terminal. I am now
9:53:18inside the project. I can go to back end
9:53:21and here I have created that file inside
9:53:25tests load because this is a uh load
9:53:29test and this is the file that I was
9:53:31talking about. It is called locust file.
9:53:33You can put it wherever you want but in
9:53:35my case I decided to put it here because
9:53:38it is some type of testing. Now let me
9:53:41open VS code to show you uh the contents
9:53:44of that file. So let me zoom in and
9:53:48let's see what we have here. Obviously
9:53:50you need to change this based on your
9:53:52application and you want to test a few
9:53:55endpoints. So here I have a few
9:53:57variables and but these are uh the
9:54:00things that you can that you can tell
9:54:02locus to test. So you create specific
9:54:05tasks. Here I have a few of them and
9:54:08each task receives a weight. So a task
9:54:12with higher weight is going to be
9:54:14preferred over a simple task. So here I
9:54:17am giving a higher weight to the uh to
9:54:20the latest and uh recommendation
9:54:23endpoints as well as the search
9:54:25endpoints because these are tasks that
9:54:28are heavy that are going to do a lot of
9:54:30work and I want locus to uh focus on
9:54:34them so that uh because if we have a lot
9:54:37of requests that are doing search that
9:54:39will use more resources and here in your
9:54:42test let's just look at one uh just as
9:54:44an example so we tell block us to uh go
9:54:47to this endpoint here. We uh we take
9:54:50random inputs. So we take a random
9:54:53query. Here are the search terms that I
9:54:55have here. So I can search for Inkscape,
9:54:57Python, M search and um I want to uh
9:55:02randomly take the article type, sort
9:55:05field and sort order. And because this
9:55:07is a post endpoint, I need to have a
9:55:10payload. And after doing that I can uh
9:55:14send uh a post request using the client.
9:55:18And here the so you can see that this is
9:55:20the the client that we are that we have
9:55:22here. And here is that fast HTTP user
9:55:26class that I was talking about that will
9:55:28boost the execution capacity by around
9:55:30five to six times. And uh you can see
9:55:33that it's simple and uh you can do the
9:55:35same thing for your other endpoint. But
9:55:38again this is specific to my
9:55:39application. These are the endpoints
9:55:41that I have there. But if you have other
9:55:43things, you can change um the endpoints
9:55:46to suit your application or your back
9:55:48end. Okay. So this is how we write that
9:55:51locus file. But now what do we need to
9:55:52do? Let me show you. So after coming
9:55:55here first I need to activate the cond
9:55:57environment because uh locally I I like
9:56:00to use Anaconda and here inside that
9:56:03virtual environment I already have
9:56:04locus. So what I need to do just type
9:56:07locus and hit enter. Okay. So there is a
9:56:09problem with loading imports. That's not
9:56:11an issue. I think I need to go back to
9:56:14back end and then run it like this. We
9:56:17use locos and we tell it here is the
9:56:19file. Okay, now it is working. And as
9:56:22you can see by default locos will pick
9:56:26the this port 8,89.
9:56:29But if you want to specify a port, you
9:56:33can do it like so. You can come at the
9:56:36end and add the host option uh to add
9:56:39the endpoint. But let's just keep things
9:56:42simple and let's uh go to this website.
9:56:46Okay. So this is how locus looks like.
9:56:48Let me use light mode so that you can
9:56:50see things clearly. Now what do we need
9:56:54to do? Okay. So let's change the number
9:56:57of concurrent users. At peak we will
9:57:00have 50 people trying to access the
9:57:03website. Here the the spawn rate or the
9:57:06ramp up rate is going to be five. This
9:57:08means uh locos will try to add five new
9:57:13users every second. So that's uh what it
9:57:16means and host here we need to put the
9:57:18production website and make sure to do
9:57:22this testing only when uh your website
9:57:25is not used by people. So for example
9:57:28before launch you will come and do the
9:57:31testing or if you have a staging area
9:57:33before production you can test that but
9:57:35never test production because if you
9:57:38increase the number of users that might
9:57:40create problems and if you want you can
9:57:43also open the advanced options but let's
9:57:45keep things simple. Once we do this, we
9:57:48will have two things here in this
9:57:49terminal. We will see some uh some data
9:57:52after stopping and because here we also
9:57:55are using the dashboard, we can see it
9:57:57here. So that's fine. But we can also I
9:57:59can also use go access to see that
9:58:03interactive terminal dashboard uh to get
9:58:07the data. So let's try to do that. So we
9:58:11we have sudo go access. So this is what
9:58:14we have used right. Let's run that again
9:58:17and let's see. So currently we have 214
9:58:20requests. Okay, that's fine. Uh I'll try
9:58:23to put this uh below like this. Let me
9:58:27also put uh this on top like that. Uh
9:58:32but let me decrease uh the font size a
9:58:34little bit so that we can see this
9:58:36better. Okay, are you ready? So this
9:58:39page is fine but since it is going to be
9:58:42generated every hour we are not going to
9:58:45see realtime data that's why I am going
9:58:48to close it and close these tabs because
9:58:50we don't need them. Okay let's hit start
9:58:52and let's see what will happen. So okay
9:58:54so now we are get we are creating the we
9:58:58are generating the requests and look
9:58:59here you can see that we are getting
9:59:01some traffic and here we are also uh so
9:59:04this is the amount of data that we are
9:59:06consuming right now and uh okay so it is
9:59:10running and make sure to run this uh for
9:59:135 to 10 minutes don't let it running for
9:59:15an hour or more because that might
9:59:18create a problem maybe your ISP or if
9:59:21you are using cloudflare or uh other
9:59:24stuff they might detect that you are
9:59:25under a DOS attack and they might block
9:59:28the IP. So just run this for 5 minutes
9:59:30and then get the data and see uh if your
9:59:33server was able to handle all that
9:59:35traffic or not. So I will wait for 5
9:59:37minutes and then I will come back and
9:59:39let me try to make this bigger so that
9:59:41we can see things better. Okay. So here
9:59:43you see the number of requests. You can
9:59:45also stop for example if you reach 8,000
9:59:48or something like that. So currently we
9:59:50have we we served 1,000 requests. Here
9:59:53you get the the data that we are going
9:59:56to look at after that. So you here we
9:59:58have the average response time, minimum,
10:00:01maximum and the current requests per
10:00:04second. So for example, this endpoint is
10:00:07receiving three requests. Again, as I
10:00:09said, I will wait for 5 minutes and then
10:00:11I will come back. I am back. We have
10:00:14reached 5,000 requests. I will stop now
10:00:17because I have the data. Okay. So we can
10:00:20read this table or we can also check the
10:00:23other tabs. If we click on charts that
10:00:26gives us uh that will plot the data. Uh
10:00:28so we can see that we started at 123 and
10:00:32we finished at 1211 and uh so here what
10:00:35do we have? So we have the total
10:00:36requests per second. Uh on average we
10:00:40were getting 16 requests per second and
10:00:42it also shows us the requests that
10:00:45failed. So that's interesting. We also
10:00:48has 16 requests that were failing per
10:00:51second. We have here the response time.
10:00:54So normally it's small um but sometimes
10:00:57we get these spikes. That's interesting.
10:00:59This is the ramp up. So this is the
10:01:02number of users. So here we started
10:01:04small but then we reached the maximum
10:01:06value set to 50. Okay that's good. But
10:01:09if you are curious why sometimes we get
10:01:12spikes in the response time that's
10:01:14normal. This is what we call tail
10:01:17latency. And if you are curious, there
10:01:19is this amazing article that you can
10:01:21read and uh so by Andrew you can see
10:01:24that here on this image it it shows you
10:01:27that most requests are going to have low
10:01:30response time but some of them are going
10:01:32to have
10:01:34long response times and this is what we
10:01:36call tail latency. It has nothing to do
10:01:39with your application so don't worry
10:01:41about it. Uh but if you want to uh read
10:01:44more about it to see why that happens
10:01:46even though the request is going to take
10:01:49the same amount of time to process, you
10:01:52can read this article. I'll make sure to
10:01:55uh it is already on the documentation.
10:01:58So you can find it there uh if you go to
10:02:00the repository. And again just a
10:02:04reminder if you don't if you don't know
10:02:06what I am talking about, this is the
10:02:08repo that uh that I'm that you can go
10:02:11to. So it's called fullstack deployment
10:02:13handbook and inside the book folder you
10:02:16can find the content for this specific
10:02:20module and here if you search for tail
10:02:24latency you'll find that article here we
10:02:27can also go to uh the terminal let's see
10:02:31what we have here and we can stop this
10:02:32and this will also show us some data. So
10:02:35if you it's the same data that we have
10:02:37in the dashboard. So you can see that
10:02:38here we have 5,000 requests in total and
10:02:42these are the error. So you can see that
10:02:44we have 429.
10:02:46Ah so now I understand why we have this
10:02:49error because in my API endpoint I have
10:02:52rate limiting. So that's why we get 429
10:02:56because once we reach I think 20
10:02:58requests per uh per minute uh I block
10:03:01the request. So that's why we were
10:03:03getting this. So I was confused why we
10:03:06have a lot of errors, but that's fine.
10:03:08So again, that's just another layer of
10:03:10security or it's a layer to block people
10:03:14that are that want to scrape your data
10:03:16or something like that. So that's why we
10:03:18have this. So that's completely
10:03:20understandable.
10:03:22That's amazing. I told you that the rate
10:03:23limiting is done by minutes. So this is
10:03:26why we have this specific recurring
10:03:28shape because here if you look at 12 six
10:03:31okay so the this specific user was below
10:03:34the limit but once they reach that the
10:03:36number of errors starts to increase
10:03:38because they made a lot of requests but
10:03:41after the end of that minute we reset so
10:03:43that's why they are able to make
10:03:46successful requests but then it
10:03:47increases. So now let's go back to the
10:03:50statistic to the statistics page to see
10:03:52what we have here. So on aggregate the
10:03:55average response time is 100
10:03:57milliseconds. That's amazing. So if we
10:03:59look at the search uh endpoint it's done
10:04:03is in less than 100 milliseconds. That's
10:04:06that's really great. But if we go to max
10:04:08which is the SP the tail latency we uh
10:04:11we have to wait for 2 seconds. But that
10:04:14happens very rarely. So what you want to
10:04:16look at is basically the 50th
10:04:19percentile. So that's what we have here
10:04:21in this chart. this green. Yeah. So, if
10:04:24I remove this, if we look at this green
10:04:27graph, you can see that this is what we
10:04:29want to care about because most requests
10:04:31are going to be here. And you can see
10:04:33that the response time is slow is small.
10:04:35It's less than 100 milliseconds. So,
10:04:38that's that's really great. Uh that's
10:04:40all I wanted to show you here in locus.
10:04:43You also can look at failures,
10:04:45exceptions. We don't have those. And if
10:04:47you want, you can download the data. So
10:04:49now we can close this because we have
10:04:51stopped the server here and I can also
10:04:54close that. And if you look at Btop you
10:04:56can see that the server was doing a lot
10:04:58of work uh here. But after that yeah now
10:05:01uh it is sitting idle because there are
10:05:03no requests. I can also close that. And
10:05:06here if you look at this dashboard you
10:05:08can see that here are the yeah that's
10:05:115,000 requests that we received today.
10:05:14And here are the end points that were uh
10:05:17that that received a lot of work. And
10:05:19you can see this is based on that those
10:05:22weights that we specified in the tasks.
10:05:25So track search were the ones that had
10:05:28the highest weight. That's why the we
10:05:30have a lot of uh hits. Amazing. Now we
10:05:34can close this and we can look at the
10:05:36NCDU utility. You know if you want to
10:05:40look at this disk this space you can use
10:05:42du-h
10:05:43and that gives you this information you
10:05:45can aggregate it stuff like that but
10:05:47again it's not uh a tool like btop where
10:05:51you can interact with it so that's why
10:05:53there is a tool called ncdu that allows
10:05:56us to inspect the disk util us usage uh
10:06:00with a terminal user interface and that
10:06:03tool is called ncdu and on ibuntu you
10:06:06can use a to install it and once that's
10:06:09done you can use sudo incdu and here we
10:06:13are going to do a full scan starting
10:06:16right at the root directory so that will
10:06:19take some time but as you can see
10:06:21already we have a nice in application
10:06:24running in the terminal yeah we can use
10:06:26to analyze the disk usage and as you can
10:06:30see I can use the arrows to go through
10:06:32all of these and here let me show you
10:06:34what we have here so currently the total
10:06:37disk usage is 6.2 GB. Here you can see
10:06:40that under the user folder we are use we
10:06:43have we are using 2 GB and here my web
10:06:47application is using 300 mgabytes. If I
10:06:49want to understand what is inside user I
10:06:51can hit enter. Now I am inside can go to
10:06:54library. Okay. So here we have things
10:06:57related to Linux. I should not touch
10:06:58that. Here we have Python. I should not
10:07:00touch that because I am using it. Um so
10:07:03that's how you use it. If I go want to
10:07:05go back, I use I go up and uh uh so that
10:07:09will take me uh to the user folder and
10:07:12now I can go to maybe home. Let's see
10:07:14what we have there. Okay, so we have my
10:07:16user there and this these are all the
10:07:18things. So here we have NVM because we
10:07:20are using node, we have cache. So maybe
10:07:24if you want to remove the pip cache, we
10:07:26can come here and hit D. That will
10:07:29delete it. Okay. And if you want to
10:07:31learn more about the commands, you can
10:07:32hit you can press the question mark and
10:07:35that will give you this help panel that
10:07:38shows you how to move. So if you want to
10:07:40move your cursor, you can use the up and
10:07:42down keys. If you want to open a
10:07:44directory, you hit enter. If you want to
10:07:47delete, you use the D key, etc. And you
10:07:50can uh use the up and down arrow to see
10:07:53the other commands. So let's hit Q to
10:07:56close. And let's close out of incu. And
10:08:00to do that we use Q in order to close.
10:08:02And that's how we use incu to analyze
10:08:05disk space and to delete uh any folder
10:08:08that is no longer needed. Okay. So now
10:08:11let's try to fix the issue with uh
10:08:14rotating logs. In order to do that we
10:08:17will open the journal D configuration.
10:08:20And here I want you to search for this.
10:08:22So in order to search, you type, you hit
10:08:25Ctrl W and you paste the term that you
10:08:28want to search for and just hit enter
10:08:31and there it is. So what we need to do,
10:08:33we need to uncomment this in order to
10:08:35tell Linux to change the amount of of
10:08:39logs that we want to use because by
10:08:41default on Ubuntu, we allow uh systemd
10:08:46journal D. So this damon is going to
10:08:48consume up to 10% of your total disk
10:08:51volume just to store logs which is a
10:08:55lot. So if you want to change that
10:08:57default you come here you unccommend
10:08:59this option and you set it to 100
10:09:02megabyte or any other value. So now
10:09:04after we save this we need to restart
10:09:07the service. So how do we do that? We
10:09:10use systemctl and we restart it and we
10:09:13now we can run status to see if it is
10:09:16running and yes it is. That will uh
10:09:19reduce the amount of space that we
10:09:21allocate to logs. But in order to fix
10:09:24the engineext issue we go here uh etc
10:09:29log rotate and we go to engineext. And
10:09:32as you can see here under rotate we have
10:09:3514 days. So after for everything older
10:09:38than 14 days will be deleted. I will
10:09:40change that to 365.
10:09:43And this value depends on the disk space
10:09:46that you have left. So if you don't have
10:09:48a lot, this is going to cause you a
10:09:50problem because storing 365 files, it's
10:09:54a lot. Verify that you have enough space
10:09:56before doing this. Okay. Yeah, I'll just
10:09:58do it because in my case, it's uh it's
10:10:00not an issue. And I will exit. And it
10:10:03also depends if your website receives a
10:10:05lot of traffic because let's say that I
10:10:07only received let's say a thousand
10:10:09requests a day that's not a lot. So the
10:10:11log file is going to be small but if I
10:10:13receive 100,000 requests that's a lot.
10:10:15So one file is going to be very huge.
10:10:18Multiply that by 365
10:10:20and uh imagine the amount of disk space
10:10:23that you will be consuming. So yeah now
10:10:26we won't we won't delete older files and
10:10:29in the dashboard we will be able to see
10:10:31a large uh duration so that we can see
10:10:34what how what is happening across months
10:10:37or weeks. Finally we are going to create
10:10:40an automated script. I will put it
10:10:43inside user local bin because if you
10:10:46remember when when you put a a bash
10:10:48script there you can run it in any
10:10:50location. So that's the benefit of
10:10:52putting it right there. And this
10:10:55specific file it's small it's not that
10:10:57big and this is what it is going to do.
10:11:00So here set e is awesome. So if we have
10:11:04an error is any step we are going to
10:11:06terminate and we are going to exit
10:11:07because we want to run all of these
10:11:09steps without an issue. First we clean
10:11:13um anything from apt and we use auto
10:11:17remove with dash y to accept the the
10:11:21things that we are going to delete. The
10:11:23journal ctl vacuum time is a command
10:11:27that will delete logs that are older
10:11:29than 2 days. Here we are removing pip
10:11:33cache and pmppm cache. And we print echo
10:11:36system cleanup completed. And I want to
10:11:39do this let's say once a month. Okay, so
10:11:41I will save the file, exit. I will use C
10:11:44cho to make it executable. Let's go back
10:11:48to chron tab. Let's go to the end of the
10:11:50file and let's add the following line.
10:11:54And this is going to run this um script
10:11:58at 3:00 a.m. in the morning on the first
10:12:02day of every month. This is what we have
10:12:05why we have this. And the logs are going
10:12:07to be stored inside this file. But we
10:12:10can see if we are getting the correct
10:12:13echo message. Okay. Now I can save and
10:12:16exit and that will be run that that
10:12:19specific script will run uh once a
10:12:21month. But you can also test it because
10:12:23as I mentioned once we put that thing
10:12:25there you can see that once I hit tab it
10:12:28detects it. And now I can hit enter and
10:12:30it says that I need to use pseudo. Okay
10:12:33that's understandable. And because we
10:12:35have put it inside p sudo that's fine.
10:12:37So it's it is going to run without an
10:12:38issue. So now it is uh using apt get to
10:12:42clean the cache. This is the vacuum. So
10:12:45you can see that we deleted logs that
10:12:47are older than 2 days and you can see
10:12:48they are using a lot of space. So here
10:12:51we have maybe seven uh so 82 megabytes
10:12:55and we have the correct echo message and
10:12:58this is what we should see in that log
10:12:59file. Amazing. We have finished this
10:13:02reliability subchapter. Now we have one
10:13:06more subchapter that I want to add
10:13:08before ending this book or this course
10:13:10and that one will be will be about
10:13:13continuous security. If you remember
10:13:15before uh we have created a pipeline
10:13:20that runs uh let's see so we have SUS
10:13:23and we also have a light security scan
10:13:26using OASP zap but that one runs in 2
10:13:29minutes so that's why we have included
10:13:31it in the pipeline because it is not
10:13:34going to block us but there is a a heavy
10:13:37where it is so this one is lightweight
10:13:39but there is a heavy version of this
10:13:42that takes a lot of time that's why we
10:13:44should not put it in the pipeline
10:13:45because otherwise instead of waiting for
10:13:478 minutes we might wait for 30 minutes
10:13:49or more that's why I didn't include it
10:13:51in the pipeline but in the next sub
10:13:53chapter I will show you how to create a
10:13:56GitHub action that will run daily and
10:13:59that action is going to run that heavy
10:14:02security dynamic security scan. So
10:14:05that's what we are going to see in the
10:14:06next sub chapter. It's uh it's a bonus
10:14:09that I wanted to add to show you um how
10:14:12that happens. And the beauty about that
10:14:15one is that it's going to run
10:14:16automatically. We don't need to trigger
10:14:18it with a pull request or something like
10:14:20that. It is going to run let's say every
10:14:22day at 3:00 a.m. or uh something like
10:14:25that. We can specify maybe even hourly
10:14:27if you want. And uh once when we see an
10:14:29error there, it means hey we have a
10:14:31problem that we need to solve. But never
10:14:33include that in your pipeline because
10:14:35that will make it very very slow. We
10:14:37arrived at the last sub chapter in
10:14:40module 6. It is titled continuous
10:14:43security. Let's see what we are going to
10:14:45talk about. We have mentioned in the end
10:14:48of the previous sub chapter that in the
10:14:50pipeline we are performing a light
10:14:52dynamic testing. So we want to move from
10:14:55that state into performing a complete
10:14:58synchronized stack. And that's what I
10:15:01mean by a heavy security scanning
10:15:04because we need to run the front end, we
10:15:06need to run the back end and other
10:15:08services. In my case, that other service
10:15:10is M search. So we want to have all of
10:15:12these things interacting together while
10:15:14running the dynamic testing. We are
10:15:17going to also use OAS, but we are going
10:15:19to configure it a little bit to perform
10:15:21that heavy scan. And as I said, we are
10:15:25going to use a an automated GitHub
10:15:28action that will use the chrome syntax
10:15:30like we did in the chrome tab uh
10:15:32recently to run this scan following a
10:15:35specific uh frequency or interval. And
10:15:39like before we are going to use that uh
10:15:42zap rule set that specific TSV file to
10:15:46uh avoid uh false alarms. So things
10:15:49related to engineext security headers
10:15:51stuff like that. We are here in my
10:15:54second account. This is my fork. I need
10:15:57to uh click on the repository. Uh let me
10:16:00remove these uh tabs. We don't need
10:16:03them. And here on GitHub, you can hit
10:16:06the dot key to open uh VS Code directly.
10:16:10So if you want to do things quickly, uh
10:16:13you can do them that way. So just hit uh
10:16:16the dot key and that that will open VS
10:16:19Code. Uh so here it says the extension
10:16:22yes allow login. Let's continue. The
10:16:26only thing we need to add is a new uh
10:16:28workflow and here we are in the master
10:16:31branch. So we cannot push directly to it
10:16:34because it is protected. So let's create
10:16:36a new branch. Let's call this feed daily
10:16:40bas scan and let's create it from the
10:16:44head. Let's switch to that branch. Okay.
10:16:47Now open the GitHub folder and add this
10:16:51specific file inside the workflows
10:16:53folder daily deep security scan. I will
10:16:58copy the full content and I will paste
10:17:00it here. You will see that this is
10:17:02similar to the other file where it is
10:17:04this dust scan. Uh I'll put them next to
10:17:07each other so that we can see we can see
10:17:09the difference. The only thing that we
10:17:11have here, you can see that here we have
10:17:13on workflow call. But here we also add
10:17:17this option which is to schedule this.
10:17:20So this means that this GitHub action
10:17:24will run daily at 6 UTC. We don't need
10:17:29to. Yes, we have the option to run it
10:17:32manually, but on top of that, GitHub
10:17:35will read this chrome syntax and it is
10:17:38going to understand that we wanted to
10:17:41run this for us every day at 6 uh UTC.
10:17:45And that's the the only the the first
10:17:48difference, but we have another one. So
10:17:50here uh so because this is a deep scan,
10:17:52we want to time it out. So if something
10:17:55bad happened maybe uh we didn't install
10:17:58dependencies correctly or something was
10:17:59not working uh well we don't want this
10:18:03job to get stuck. So we are going to
10:18:05fail if it takes more than 30 minutes
10:18:08and this value can be adapted based on
10:18:10the scale of your application. So if you
10:18:13have a lot of endpoints and a lot of
10:18:15services the scan will take a lot of
10:18:18time. So you want to run it maybe two,
10:18:20three or five times to get the average
10:18:22time and then put it here. But if we see
10:18:25if we look at the other stuff, let me uh
10:18:28decrease the font size. First of all, we
10:18:30define the environment variables for me
10:18:33search. But then the next steps are
10:18:35similar. We check the repository. We
10:18:38install Python. We install the backend
10:18:41dependencies. Uh we are here. We after
10:18:45that we install me search. Uh okay then
10:18:49we start it we use that script to see
10:18:53the so let's see so here we are starting
10:18:55mill search and uh here you can see that
10:18:57we have that seed script here it is here
10:18:59is that same step and we also start the
10:19:02back end so here maybe I could uh put
10:19:06this here to have the same order let's
10:19:09put it there you can see that here we
10:19:10have se test data we start the backend
10:19:14server and after that so here we stop
10:19:17because we were testing just the back
10:19:19end. We weren't testing the other things
10:19:22in the stack. So we can see that here
10:19:24after arriving at this we just pointed
10:19:26at the open API JSON because we have a
10:19:28fast API back end that will tell Zap all
10:19:32the end points. But here we continue. We
10:19:35install we set up PNPM. We install
10:19:38NodeJS. We install the front end
10:19:40dependencies. We build the front end.
10:19:43And then we start it. Here we are using
10:19:45pmppm preview because we want to
10:19:47simulate a production environment. We
10:19:49are not using pmppm dev because that is
10:19:51a dev server. We want to use pmppm
10:19:54preview. And here we uh like before we
10:19:57are using a for loop just to ensure that
10:19:58the front end started and and it is
10:20:01ready to uh be used. And finally we
10:20:04perform the full scan this time. And if
10:20:07you look at the image that we are using
10:20:09it's different. So here we have action
10:20:11API scan because this is uh used for API
10:20:15endpoints but here we are using the full
10:20:18scan and here you can verify the version
10:20:22to see if there are new ones. If there
10:20:24are make sure to update and here the
10:20:27target also changes. So we are pointing
10:20:28at the front end. It's like a user that
10:20:30will go there and try to do malicious
10:20:33stuff. uh we are we don't we don't just
10:20:35point at the back end and we are reusing
10:20:38that TSV file and uh yeah the this the
10:20:42other options are are the same but after
10:20:44doing this you can see that we are going
10:20:46to test everything in our stack mainly
10:20:48search the front end and the back end as
10:20:51you can see both of them are similar we
10:20:52just uh added running the front end and
10:20:56uh we changed the image that we are
10:20:58using here to action full scan amazing
10:21:02now let's commit this file. So let's
10:21:05come here and let's say that field add a
10:21:09new workflow for full dust scan commit
10:21:13this and push it to this new branch that
10:21:15we have created. And now we can create a
10:21:18new merge request and point it at my
10:21:21master branch. And I hope that we don't
10:21:25have a problem with the dependencies. I
10:21:28think we will because it's been a few
10:21:30weeks since I've worked on this. But if
10:21:33we yeah if I get some problem while
10:21:36auditing the dependencies I'll try to
10:21:38resolve them and I will come back and uh
10:21:40Okay. So let's wait for this to finish
10:21:43and then we can go to the actions tab
10:21:46and look for that new file and then we
10:21:48can run it manually from here. Let's
10:21:51wait for this to yes thankfully the
10:21:54auditing step passed. That's amazing. Uh
10:21:58so that means that this will will not
10:22:00fail hopefully because we haven't
10:22:01changed anything in the source code. I
10:22:04am back and as you can see we can merge
10:22:06the pull requests because we don't have
10:22:09any error. Let's confirm the merge and I
10:22:12can also close this tab because I no
10:22:15longer need it. Let's go back to master
10:22:18and yeah as you can see it merged now.
10:22:21Now we can click on actions and we
10:22:24should see this new scan. Yes, here it
10:22:27is. Daily deep security scan. Let's
10:22:30click on it. And as you can see,
10:22:32normally if we wait, it should run at 6
10:22:34in the morning every day. But now it is
10:22:3812:00 a.m. or 12:00 p.m. So I cannot
10:22:41wait for that. So let's just hit run
10:22:43workflow. Make sure that it is on master
10:22:45and hit run. And uh yeah, it's just one
10:22:49simple job. If we click on it, we will
10:22:52see the the steps. We installed Python
10:22:55and now we install all the dependencies
10:22:58that we need for the back end. We
10:23:00downloaded M search. Now we start it and
10:23:04we need to seed it with data so that the
10:23:06end points will work and won't return
10:23:10errors. We have restarted the started
10:23:12the back end and as you can see we sent
10:23:14a request to help to ensure that server
10:23:16has started. Now we are installing
10:23:19NodeJS
10:23:21the front end dependencies. We are
10:23:23building the front end. That shouldn't
10:23:25take a lot of time. Okay, that's done.
10:23:27Now we start the front end using PMP
10:23:29view. And we see the front end is ready.
10:23:32Great. And now we are starting the full
10:23:35scan. Hopefully that I haven't forgot
10:23:39anything to add to that TSV file.
10:23:43Normally this should pass because uh I
10:23:45haven't changed anything in the back end
10:23:47and I have tested this before on my
10:23:49personal project or on my websites that
10:23:52I have deployed to production. So it
10:23:54shouldn't fail. But since this will take
10:23:57some time, I will wait for it and then I
10:23:59will come back. I am back and as you can
10:24:01see the scan didn't fail. That's a good
10:24:04sign. And in my case it took 6 minutes.
10:24:07You can see that this is already uh
10:24:09taking three times more time than the
10:24:12light one. The light one took 2 minutes
10:24:15and this one took six. So, and it runs
10:24:17successfully and this is going to run uh
10:24:19every single day. But as I said, if you
10:24:22have a a large application, my personal
10:24:24work is not that big. So, I have a few
10:24:26endpoints that's why it took 6 minutes.
10:24:28But if you have a lot of applications, a
10:24:31lot of services in your app, this time
10:24:33will increase drastically. And that's
10:24:36all I wanted to show you in this
10:24:37subchapter. Thank you for watching. The
10:24:40next video will be about we will look at
10:24:43all the things that we have learned
10:24:45together across this long journey that
10:24:48we took from the first sub chapter where
10:24:50we were just talking about what we are
10:24:53going to do. And now as you can see we
10:24:55are able to perform scans, block IP
10:24:59addresses and do a lot a lot of things.
10:25:03And I hope that you have learned a ton
10:25:04from this course. The the next video
10:25:06will be simple. It's not going to be
10:25:08very long, but we will try to at least
10:25:10give you something to that that you can
10:25:13use to basically tell you um maybe
10:25:16remind you of all the things that you
10:25:18have learned. Congratulations on
10:25:21reaching the end of the course. You have
10:25:24successfully journeyied from a bare
10:25:27Linux server to a fully hardened
10:25:32self-sustaining production application.
10:25:34These are all the things you have built.
10:25:37In the first two chapters, I have showed
10:25:41you how to harden SSH access by removing
10:25:46the ability to authenticate with
10:25:48password, removing the ability to
10:25:51connect as root user. You also learned
10:25:55how to play with the uncomplicated
10:25:58firewall. We have learned how to use
10:26:01engineext socket proxying to talk to our
10:26:04J unicorn/ Uvic application. On chapter
10:26:09three and four we have focused on
10:26:12performance and data. Here you have
10:26:15learned how to isolate the the me search
10:26:20engine by creating a new user that we
10:26:23called me search and that user was
10:26:26managing the this service and uh we also
10:26:31made sure that the environment variables
10:26:34that med search requires are stored in a
10:26:37way that cannot be read by other users.
10:26:40We have played with go access which
10:26:43allows us to read the enginex logs to
10:26:47create really beautiful dashboards that
10:26:49shows us what is happening on our web
10:26:52server. We have also learned how to use
10:26:54cloudflare CDN to serve the static files
10:26:58quickly across the world across the
10:27:00globe and we have we have used SSL
10:27:03certificates to make your website
10:27:06secure. And finally in the last two
10:27:08chapters we talked about automation and
10:27:11security. Here you have learned about
10:27:14pre-commit hooks so that you don't push
10:27:16code that is broken. So you need to pass
10:27:20a few tests in order to be able to
10:27:22commit and push. You have learned about
10:27:25CI/CD testing pipelines, atomic
10:27:28rollbacks, automated
10:27:31dynamic application security testing and
10:27:34also uh testing the libraries that we
10:27:38are using in our application. After
10:27:40doing everything, now our application is
10:27:43ready for production. Now your system is
10:27:46observable. You have go access which
10:27:49allows you to read the logs and get
10:27:52realtime metrics. It's automated meaning
10:27:56we will have zero downtime. We have seen
10:27:59that even if we kill a UVON worker the J
10:28:04Unicorn process manager will create one
10:28:06for us. So even if you have a lot of
10:28:09traffic and maybe one of the workers
10:28:11dies everything is automated. So you
10:28:14will have zero downtime when we deploy
10:28:17our our application. If something
10:28:20happened while installing the
10:28:21dependencies, the application is not
10:28:24going to crash. We are going to roll
10:28:26back and use the all dependencies that
10:28:29we had before the deployment. And
10:28:32finally which is this is the most
10:28:33important point security. We have made
10:28:36sure that people cannot log in using SSH
10:28:40as root or by using a password. We you
10:28:43have SSL certificates that renew
10:28:45automatically so that people who connect
10:28:47to your website are connected securely
10:28:50and we have done a lot of testing jobs
10:28:53in our CI/CD pipeline. I really hope
10:28:56that you enjoyed going through this
10:28:58course with me and now we are completely
10:29:01ready to build, launch and scale your
10:29:03features with absolute confidence. Thank
10:29:06you for following along and happy
10:29:08building.

About this transcript

This page contains the full transcript of How to Deploy, Secure, and Automate Full-Stack Web Apps – Course for Beginners by freeCodeCamp.org, generated from the public captions YouTube serves with the video. The transcript has 100,405 words across 13,243 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.