YT YouTube2Text

YouTube2Text → Transcripts → #3 Creating Compute (Cluster) Step-by-Step|Azure Databricks Full Course in Telugu #telugu #course

#3 Creating Compute (Cluster) Step-by-Step|Azure Databricks Full Course in Telugu #telugu #course — Transcript

by Code chasers · 2,781 words · 418 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Hello everyone, welcome to our channel
0:02Code Sages. I am your Madhu Lakshman
0:05Atiketi, and welcome back to the world
0:07of Azure Databricks. In this video, we
0:10will see how to create compute. In the
0:12previous video, we saw how to create a
0:15Databricks workspace. We also looked at
0:18an overview, okay? So the next step is
0:20to see how to create compute in this
0:23video. Compute can also be referred to
0:26as a cluster, okay? A cluster or
0:28compute is the main core part where the
0:32execution takes place. Okay, no matter
0:35how much code we write, there has to be
0:37an engine to execute it, right! So,
0:39it's not like I can directly say it's
0:41just an engine place. So this is the
0:44main place where the execution part is
0:47concentrated; that is compute. Okay, so
0:50without any delay, let's start the
0:52video. Hello everyone, in the last
0:55video, we created a Databricks
0:57workspace and saw an overview. We saw
1:01how the Databricks workspace looks and
1:03some of its key terms. As a beginner,
1:05we looked at some terms to know and
1:08some areas we need to be aware of. And,
1:10one small thing. Even if you didn't
1:13watch that overview, or if you didn't
1:14understand the terms in that overview,
1:15it's okay. Because in future lessons
1:18and upcoming videos, we will understand
1:21every single term clearly, okay? So,
1:24what I will do now is configure our
1:26cluster in our Databricks workspace.
1:29Okay, so we will see how to do that. So
1:32, first of all, I am launching our
1:34workspace. After launching, it will
1:38take some time to launch the Databricks
1:40workspace. Okay, I have launched it.
1:44After launching, there are horizontal
1:47lines here, right? If I click on them,
1:50I can see options like Workspaces,
1:53Recents, Catalog, Jobs, Pipelines, and
1:56Compute. So, I will click on Compute
1:59here. Compute, generally, compute is a
2:02place. As I mentioned earlier, when
2:05talking about compute, in this place,
2:08we can create compute. To be specific,
2:11it is about creating and configuring a
2:13cluster. Okay, so we will do that. So,
2:17click on this Compute, and there will
2:18be a "Create Compute" button here.
2:20Under the plus symbol, there is a "
2:22Create Compute" button; click that
2:24button. Okay, after clicking, it will
2:27redirect us to the compute creation
2:29page. Okay, so this is the compute name
2:33, which by default has taken "Madhu's
2:35Cluster" along with the date and time.
2:39So I will remove that and give it a
2:44name like "Databricks Tutorial Compute.
2:49" Okay, so here is the policy...What
2:54actually is a policy? Like, this is
2:56called a cluster policy here. There are
2:59different types of cluster policies
3:01available. Okay, so for unrestricted,
3:03all options are available here; there
3:06are no restrictions on any options.
3:09Okay, we have these options below,
3:11whether in performance, tags, or
3:13advanced; there are no restrictions.
3:16There are three sections down here;
3:18just ignore the general ones. There are
3:22no restrictions in performance, tags,
3:24or advanced; we can go ahead and edit
3:26them. So, if I select Personal Compute,
3:31look, machine learning wasn't ticked,
3:35but if I select Personal Compute,
3:38machine learning gets ticked
3:40automatically, and the single node
3:43option appears. So that went away as
3:47well. Some of these things have
3:50appeared by default. So, let's see what
3:53these 'terminate after 4320' settings
3:55are about. I will explain to you what
3:58this cluster policy is. I will open a
4:02new workbook, or rather, a new notebook
4:06. Okay, so it is necessary for us to
4:11know about cluster policies. As data
4:16engineers, we work on Databricks with
4:19PySpark. Okay, it is necessary to know
4:21all of this while working. I could just
4:24pick any cluster policy and move
4:26forward. But as a data engineer, we
4:29should know what is happening, what a
4:31cluster policy is, and why we are
4:32creating one. We need to know that.
4:35Okay, so for cluster policy, let's just
4:37skip this specific policy for a moment.
4:40First, imagine I have a team of one,
4:43two, three, or four members, okay. And
4:46I have a project head, okay, he is the
4:50head. So, Team member one and Team
4:53member two have just joined. Okay, when
4:57they joined, they sat down to create a
5:00cluster. They sat down to create a
5:03cluster, and maybe due to a lack of
5:06knowledge or accidentally, since they
5:08are just learning Databricks, they
5:11might make mistakes. So, they are
5:14trying to create and configure a
5:17cluster for project work. Since they
5:22aren't aware of some configurations,
5:25they might set the cluster size too
5:29high, and in a real-time industry
5:32environment, that leads to us paying
5:35high costs. This is a free model we are
5:40using now, but going forward, once the
5:43free subscription ends, we have to pay
5:46Azure based on how much we use the
5:48cluster. If trainees create clusters
5:53however they want, we have to end up
5:57paying that cost. When I say we, I mean
6:01the company has to pay. So, to handle
6:04this type of scenario, we have cluster
6:06policies. Okay, in Cluster Policy, I
6:12can create, like, a custom policy, and
6:16I can pre-configure the VM size here.
6:22I'll explain VM size shortly. VM size
6:27is nothing but the node type or node
6:30size. Okay. Nodes are nothing but VMs
6:33here. Okay, so I will configure the VM
6:37size to a specific value. And next,
6:40what options do we have here? So, there
6:44is Databricks Runtime. Databricks
6:46Runtime is nothing but the cluster
6:48runtime. I will set the cluster runtime
6:53to a specific value for the juniors—
6:59the new joiners—as well. And also,
7:05there is an option for "Terminate after
7:09X minutes," which I will set to 20. So,
7:14I have created a policy where all these
7:17are set by default. Once the policy is
7:21created, it will be added to the
7:23Cluster Policy dropdown. Okay, so let's
7:27say I created a custom policy called "
7:29Madhu's Policy" or something similar.
7:33I'll call it "Trainee's Policy" or "
7:35Trainee's Compute," so when they click
7:38it, all these values will automatically
7:41populate here. So, what I'm doing here
7:46is, I'm not giving the trainees the
7:49chance to choose whatever they want.
7:54Because, keeping cost in consideration,
7:57I have already created a runtime and
7:59node type within that "Trainee's
8:02Compute." So, I am able to optimize the
8:07costs. Okay! Now, let's say I haven't
8:10created any policies. If nothing is
8:13created, it defaults to "Unrestricted,"
8:14meaning there are no restrictions. We
8:16can go and edit the values ourselves.
8:20Okay! So, okay, I hope you understood
8:24what a cluster policy is. It is one of
8:26the important policies and an important
8:28topic. Okay! Now, for performance—the
8:33cluster performance—we will design
8:36the values for that now. Imagine you
8:42are configuring a cluster yourself,
8:46designing it according to your own
8:49requirements. Okay! Databricks Runtime
8:55is nothing but the cluster runtime—
8:57like Scala 2.12 and Spark 3.5.2. So,
9:01this is a cluster that runs on an
9:03Apache Spark engine. Okay! We have
9:05already discussed what the Apache Spark
9:07engine is in previous videos. So, it is
9:09one of the best and one of the most
9:12preferred data processing engines. Okay
9:15, it uses distributed computing. Okay,
9:17so the topic of distributed computing
9:19has come up again here. We have already
9:21discussed what distributed computing is
9:24. So for those who are unaware of
9:27distributed computing, I will explain
9:29it in a nutshell. In distributed
9:31computing, what happens is that the
9:33workload gets divided. Okay, we have a
9:35workload, let's say a 500 GB workload.
9:38Okay, so to compute that 500 GB, the
9:41driver node...the driver node first
9:44takes the 500 GB input. So, it takes it
9:47, and this is called the driver node.
9:49Or a node is nothing but a virtual
9:51machine here. Okay, I mentioned a
9:53virtual machine in the previous example
9:55, I explained it using a computer
9:56example. This is nothing but a virtual
9:58machine. So the driver node takes this
10:01500 GB and distributes it to the worker
10:03nodes. Okay, worker nodes...worker
10:07nodes...okay, worker nodes...I can have
10:10multiple worker nodes. Okay, there can
10:13be multiple worker nodes. Okay, so this
10:16is called a multi-node setup.
10:19Multi-node setup, multi-node setup.
10:22Okay, where I have a low workload, I
10:24don't need different work nodes, worker
10:28nodes. When the driver node can manage
10:31the workload by itself, creating worker
10:33nodes is unnecessary, right? So we call
10:36that...we call it single node. A single
10:39node means...there is only one driver
10:42node. There is only one driver node. So
10:45it can process the data, so when the
10:47workloads are low, like very low
10:49workloads, okay, when there are low
10:51workloads, single node, like when there
10:54are high workloads, multi-node, okay,
10:56so this processing, this computation,
10:58since it happens without being
11:00distributed, we call it distributed
11:02computing. So distributed computing is
11:06very important for fast processing, we
11:09see this in the Apache Spark engine, we
11:11have already discussed this in the
11:14Apache Spark engine. Okay, so the
11:17cluster we configure here, a cluster is
11:20nothing but where the actual processing
11:22happens with the help of this cluster,
11:25okay, so here Apache Spark...here we
11:27have Spark 3.5.2. So here I gave 6.4 (
11:32LTS). If we look here, we have many
11:35options. So, we can see a tag here, LTS
11:38. Which means Long Term Service, they
11:40provide us with service for around
11:42three years. So, that is why we opt for
11:45LTS, like in real-time in industrial
11:47environments, we also select those with
11:48the LTS tag. Okay, so I selected 16.4
11:53LTS, and for our workload, we selected
11:5716.4 2.13. So, for our training
12:00workload, this is enough, this Spark
12:02runtime, sorry, this cluster runtime is
12:05sufficient. And here there is an option
12:09called Photon Acceleration, okay, a
12:11checkbox. Photon Acceleration means, in
12:14our Apache Spark, to accelerate the
12:17computation a bit and to push it faster
12:20, we use Photon Acceleration. Okay,
12:22acceleration means we are accelerating
12:25the process, we are speeding up the
12:27process. So, when there are high
12:29workloads, Photon Acceleration is
12:30important, meaning it's necessary.
12:32Since we are currently training, or
12:34like learning, we won't have such heavy
12:37workloads. So, we don't need to check
12:39this, it's not needed for us. Okay, so
12:42next, node type. I mentioned node type
12:44earlier, so single node and multi-node.
12:47Since we are actually learning here, we
12:49will have less data, okay, and low
12:51workloads. So, for us, a single node is
12:54fine, a single node is fine. And node
12:56type, this is one of the important node
12:59types. Meaning, this is our VM size.
13:02Node means nothing but it's a virtual
13:03machine, right? So, how that virtual
13:06machine should be, like how many cores
13:09it should have, how much CPU memory,
13:11sorry, how much memory it should have,
13:14we can choose that in this. Okay, if
13:18you look carefully, like if you observe
13:20closely, there are different types here
13:23. So, there is one called general
13:26purpose, there is one called general
13:28purpose. Next, within general purpose,
13:30there are also different types, there
13:32is HD, okay, there is memory optimized,
13:35and in memory optimized, there are
13:37different types, and there are compute
13:39optimized, and again, general purpose
13:41has appeared. So, there are GPU
13:43accelerated ones here. So, first of all
13:46, what is general purpose? Whenever we
13:50process workloads, whenever we process
13:52any data, when we think we have more
13:55in-memory computations, and when we
13:57take in-memory computations as a key
14:00point, and for enterprise applications,
14:03we generally opt for general purpose.
14:07Okay. So, in general purpose, we can
14:09also choose how much memory we should
14:11process with and how many cores we need
14:13. Okay. So, that is general purpose.
14:16And another one is memory optimized.
14:19Memory optimized generally means we use
14:22memory optimized in machine learning
14:24workloads. Okay. So, next. So, compute
14:28optimized. When do we use compute
14:30optimized? When we have heavy workloads
14:33, like when there is a lot of data
14:34processing, and when data processing
14:36workloads are high, the processing time
14:39becomes critical. We have to focus very
14:42much on processing times. Generally, we
14:45use this in data science workloads.
14:48Taking processing time as critical
14:50there, like as a key component, we
14:52design this node type for data science
14:54workloads. Generally, keep in mind that
14:59we use compute optimized for data
15:01science workloads, and next we have GPU
15:04accelerated. We use GPU accelerators
15:08when working on deep learning models
15:10with GPU-accelerated general purposes.
15:13Okay. Even if you don't understand all
15:16this, I'm explaining each keyword. This
15:18is used for this, and that is used for
15:20that. When I talk about general-purpose
15:23, I mean how we use it for enterprise
15:26applications or as a key component for
15:28in-memory computations, you know. It's
15:32enough if you just get an idea of why
15:35we use these. So, I can proceed
15:37directly. In general-purpose, since we
15:41are learning Databricks, we should
15:43choose minimal cores. Because if we
15:48choose more cores, the charges will be
15:51much higher. Okay, so I will choose
15:55minimal cores. So, I'll go ahead and
15:58opt for 16 GB, four cores for
16:00general-purpose usage here. But you
16:04should know what we are doing; as a
16:06data engineer, I am explaining
16:09everything so you know it all. So, four
16:11cores and 16 GB are enough for us; I
16:13will keep it as standard. And for "
16:16Terminate after," after we configure
16:19the cluster, we will run that cluster.
16:23We write code in a workspace, configure
16:25the cluster, and then connect that
16:27cluster to the workspace or my notebook
16:30. Once done, we run the code and work
16:34on the cluster. Even after our work is
16:38finished, the cluster keeps running. If
16:42the cluster keeps running, we have to
16:47pay for the time it stays active, so to
16:52avoid that, we set "Terminate after" to
16:57shut down the cluster if it hasn't been
17:02in use for a while. Okay, it's set to
17:07120 here. By default, 20 minutes is
17:09enough for us. 20 minutes, let's keep
17:11it at 20 minutes for now. Because when
17:15we work on a cluster, if we configure
17:19the cluster to a Databricks workspace
17:23and don't work with it for 20 minutes,
17:27we don't need the cluster running. In
17:32that case, the cluster turns off
17:34automatically after 20 minutes; that's
17:36what this setting is for. So, this is
17:38also for cost optimization. Tags are
17:41useful during billing. This is not
17:43relevant for you right now. For
17:46Advanced, if you open this tab, you
17:48will find a few options. For Spark, you
17:52can perform any Spark configurations
17:55you need here. Okay, you can set Spark
17:58configurations and environmental
18:00variables here as well if needed. So,
18:02in the next two videos, we are going to
18:05work on Spark configurations. You will
18:08get clarity on what exactly it is then.
18:11I'm just telling you generally that
18:14Spark configs can be done here, and if
18:17you want to save any logs in DBFS—the
18:20Databricks File System—you can
18:23configure that here, along with init
18:26scripts. If you want to configure any
18:29scripts, you can do that here. And for
18:33instance, if you are working with
18:36Python and want to install some
18:38external packages as soon as the
18:40cluster starts, you can configure those
18:44packages here. Okay, I hope you
18:48understood this. I will verify these
18:51one more time to see if I've set them
18:53up correctly. Databricks is okay, this
18:56is okay, no type is okay, single node
19:00is okay, and I've removed photo
19:02acceleration. Now we will create it;
19:05there is a 'Create' button here, let's
19:07click it to create. Okay, this cluster
19:11creation will take some time, about
19:13three to four minutes. So look here, we
19:16can see this in the event logs. In the
19:18event logs, the cluster is currently in
19:20a creating status. Once the cluster is
19:23configured, the cluster name will
19:26appear here. We can see that. So, when
19:30we click on 'Compute' again, the
19:32Databricks tutorial compute that we
19:35created will appear here. Meaning, in
19:39this compute section. Since our compute
19:42has a green symbol, it means it is
19:45running. Okay, so if you want to
19:49terminate this compute, there is a
19:51terminate option here. There is a
19:54square button here; if you press it, it
19:56will terminate. And next, if you want
19:59to restart the compute, we can restart
20:03it. If you want to delete it, you can
20:06delete it, or edit permissions; we can
20:07edit anything. This is how we create a
20:10compute. Okay, I hope you found this
20:13video useful. Thank you so much.

About this transcript

This page contains the full transcript of #3 Creating Compute (Cluster) Step-by-Step|Azure Databricks Full Course in Telugu #telugu #course by Code chasers, generated from the public captions YouTube serves with the video. The transcript has 2,781 words across 418 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.