Learn Azure Databricks in 10 Minutes- Explained simple | Azure Databricks Tutorials for Beginners — Transcript
Full transcript
- 0:01[Music]
- 0:03hello everyone welcome back to my
- 0:06Channel today we are going to say about
- 0:08assure data breaks I have created a
- 0:10separate playlist for issue data breaks
- 0:12and I'll be using that playlist to
- 0:14upload all the videos related to Azure
- 0:16databricks so this is the first video in
- 0:18that playlist okay so now let's get
- 0:20started so firstly what is azure
- 0:23databricks it is an Apache spark based
- 0:26analytics platform which is mainly used
- 0:27for doing big data workloads this tool
- 0:30is very popular and it is used by data
- 0:32Engineers data scientists and also data
- 0:34analyst so pretty much all the data
- 0:37professionals use this tool in Azure so
- 0:39if you ask me why this tool is really
- 0:41popular I would say there are a lot of
- 0:43interesting features within Azure
- 0:45databricks so let's see all these
- 0:47individual features within data breaks
- 0:49one by one with an example so firstly
- 0:52the most important feature is Apache
- 0:54spark so what is Apache spark it is an
- 0:58open source distributor processing
- 0:59system which is made in the use for
- 1:01doing big data workloads so this Apache
- 1:05spark is inbuilt to assure data bricks
- 1:07and it uses the Spark engine for all the
- 1:10data processing
- 1:12so when we use Su data breaks to process
- 1:15any data The inbuilt Spark engine take
- 1:18care of all the partitioning and other
- 1:20optimization by itself which makes it
- 1:23extremely powerful and that's the reason
- 1:25databricks is widely used for data
- 1:27transformation in our show so we can
- 1:30create high performance spark compute in
- 1:32Azure databricks for doing the data
- 1:34processing there are wide varieties of
- 1:36spark compute available in data breaks
- 1:38with the different CPU and memory size
- 1:40it is very simple to configure based on
- 1:43the requirements and it is highly
- 1:44scalable
- 1:45say for example if you create a cluster
- 1:48using a smaller compute and in the later
- 1:50case for some reason you want to
- 1:52increase the cluster size it is very
- 1:54easy to do it in a click of a button
- 1:56which makes it extremely scalable so one
- 1:58thing to note here is the cause for this
- 2:00cluster is charged only when we are
- 2:02actually using it so we can turn off
- 2:04this cluster if we are not using it and
- 2:06we are not going to pay for it which is
- 2:08really cool right the next big feature
- 2:10in Azure data brexit notebooks in data
- 2:13breaks all the development work like
- 2:15data processing or model training is
- 2:17done using the notebooks a notebook is a
- 2:20kind of code editor which is in the
- 2:22format of different cell based structure
- 2:24you could type in the code to see the
- 2:27immediate results inside the notebook
- 2:28itself which is very easy and convenient
- 2:31to work with
- 2:32since it is a cell based editor you can
- 2:35use each cell to perform different
- 2:37functionalities which is very easy to
- 2:39understand and maintain as well having
- 2:42the notebook option inbuilt is another
- 2:43great feature of azure databricks the
- 2:46next feature is it is very easy to
- 2:48switch programming languages in
- 2:49databricks
- 2:50databricks support different programming
- 2:52languages like Pi spark spark SQL Scala
- 2:56and r
- 2:57so when you create your notebook in data
- 2:59bricks the default programming language
- 3:01you can type in the notebook is python
- 3:04you can easily switch to other
- 3:05programming languages in the same
- 3:07notebook say for example if you are very
- 3:09good at SQL then you can use a feature
- 3:12called Magic command in data breaks if
- 3:15you could type percentage SQL in the
- 3:16cell the cell automatically converts to
- 3:18your SQL cell where you could type SQL
- 3:21code to perform the data processing
- 3:23similarly you could use the magic
- 3:25command to convert the cell to different
- 3:27language cell such as spice Park spark
- 3:30SQL Scala and R this is extremely useful
- 3:34since you could use multiple language in
- 3:36the same notebook for processing the
- 3:38data
- 3:39these are some of the features of Apache
- 3:41spark in RC data bricks the next
- 3:43important feature in Azure databricks is
- 3:46ETL
- 3:47we all know ETL stands for extract
- 3:49transform and load although issue
- 3:52databricks is mainly used for the
- 3:54transformation part in the ATL framework
- 3:56we could use Ashu data bricks as an
- 3:59individual tool to perform the complete
- 4:02ETL process
- 4:03let's see how we could do that firstly
- 4:06we could use Spar connector for doing
- 4:08the extraction part
- 4:10with spark connector we could use
- 4:12databricks notebook to connect to
- 4:14multiple sources and ingest the data
- 4:17from source to destination
- 4:19the spark character is supported in many
- 4:21data sources for example SQL server
- 4:24issues SQL database mongodb and several
- 4:28other data sources
- 4:30for interesting the data from these data
- 4:32sources we don't want tools like Azure
- 4:35data Factory or Russia synapse analytics
- 4:37for performing the injection instead we
- 4:40could also use spark connector to
- 4:42connect to these data sources to extract
- 4:44the data using issue databricks itself
- 4:47and the next one is the transformation
- 4:48part we all know that databricks is the
- 4:51most preferred tool in Azure for doing
- 4:53the data transformation due to its
- 4:56inbuilt Apache spark option
- 4:59and the final one is the loading part we
- 5:01could also use a shoe data bricks for
- 5:03loading the data into the database using
- 5:06the high metastore feature
- 5:08with high metastore we can create
- 5:10databases and tables where we can load
- 5:13the transform data to it
- 5:15the tables can also be queried using the
- 5:18new SQL editor feature inside the
- 5:20databricks or can also be consumed by
- 5:23different reporting tools like power bi
- 5:25or Tableau so due to all of these
- 5:27different features we can use ASU
- 5:30databricks to do the complete ATL
- 5:32process also more than actually
- 5:34performing the ETL process we can also
- 5:37use issue databricks to orchestrate the
- 5:40ETL so what I mean by this is
- 5:43inertia data breaks we have an option
- 5:45called workflows using workflows we can
- 5:49run the notebook that we create using
- 5:51Azure databricks as a job which means
- 5:53that we can schedule the notebooks to
- 5:55run it on a regular basis
- 5:58due to this future using IC data breaks
- 6:01we can orchestrate the complete ETL
- 6:03process by configuring when and what
- 6:06notebook to run with the different
- 6:07triggers and stuff this makes issue data
- 6:10breaks as a complete ETL tool and hence
- 6:13its most popular tool among the data
- 6:15Engineers the other major feature in
- 6:17Azure databricks is data science and
- 6:19machine learning
- 6:21we can use data bricks to train a
- 6:23machine learning model
- 6:24also we can serve or deploy the model
- 6:27that has been trained databricks is well
- 6:30suited for doing data science and
- 6:31machine learning stuff as it has inbuilt
- 6:34features such as option to train automl
- 6:37experiment and also we have dedicated
- 6:39spark clusters available for doing the
- 6:42machine learning stuff with higher gpus
- 6:44suitable for model training we all know
- 6:46that for training a machine learning
- 6:48model with huge data sets requires
- 6:50higher GPU and we could configure it
- 6:53easily in databricks also there is a
- 6:56dedicated tab called serving to deploy
- 6:58the train model so once the model is
- 7:00deployed we can make use of the model
- 7:03endpoint for doing any kind of
- 7:05predictions
- 7:06so due to these features similar to how
- 7:09databricks is very popular among the
- 7:10data Engineers it is also popular among
- 7:13data scientists and thereby it is widely
- 7:16used to tool for doing machine learning
- 7:17and data science work in Azure the next
- 7:20important feature is streaming
- 7:22we can say that Ashu databricks is a
- 7:24perfect tool to work with streaming data
- 7:27there are different features in data
- 7:28bricks that allows us to work seamlessly
- 7:31with streaming data
- 7:32for example we have structure streaming
- 7:35which is an Apache power based streaming
- 7:37solution because of this we can perform
- 7:40near real-time data ingestion and also
- 7:43can perform near real-time machine
- 7:44learning and predictions
- 7:46due to this streaming capabilities we
- 7:49could ingest the data faster from source
- 7:51to destination in the streaming fashion
- 7:53instead of doing in a batch load on a
- 7:56scheduled basis
- 7:57in the future videos we can explore
- 8:00these streaming functions and how we
- 8:01could work with that the next feature is
- 8:03the SQL editor we have seen how data
- 8:06engineers and data scientists can work
- 8:08with databricks and this time this is
- 8:11all about the data analyst in Azure data
- 8:14breaks we have an option called SQL
- 8:15editor which is kind of similar to SQL
- 8:18Server management Studio
- 8:20you could use the SQL Editor to query
- 8:22the tables that we have in the database
- 8:24and Analysis can be done on the data in
- 8:28most cases a data analyst worked with a
- 8:30cleaner data that would be querying it
- 8:33to find insights from it they can make
- 8:35use of this Editor to write SQL commands
- 8:37to query the data we need to use
- 8:40databrick SQL query language to query
- 8:42the data of in databricks which is kind
- 8:44of similar to spark SQL
- 8:46so due to these features a data analyst
- 8:49can also work with azure databricks
- 8:52after this we have an interesting
- 8:53feature which is easy collaboration
- 8:56Azure databricks can be used by multiple
- 8:58people at the same time and can work
- 9:00collaboratively for performing any kinds
- 9:03of tasks this is possible due to its git
- 9:05integration functionality the popular
- 9:07git repos such as GitHub and ushi devops
- 9:10is supported in RC data bricks we have a
- 9:13separate tab called repos in Azure
- 9:15databricks workspace where we could use
- 9:17that to integrate with the different
- 9:18repos such as GitHub and acid devops
- 9:21this feature leads to EC collaboration
- 9:23and working collaboratively in doing any
- 9:26kinds of data engineering or data
- 9:27science work is important and we can do
- 9:30that very well using ashy data bricks so
- 9:32these are some of the main features of
- 9:34RC data breaks I think by this time you
- 9:37should be more excited to work with data
- 9:39bricks if you haven't worked with it
- 9:41before trust me this is a great tool to
- 9:43work with for any data Engineers or data
- 9:46scientist or data analyst I guess now
- 9:49you have a clear understanding about
- 9:50what exactly data is and its important
- 9:54features I'll be adding a lot of videos
- 9:56later to data breaks in this playlist so
- 9:58please subscribe to my channel if you
- 10:00want to learn more about it also give a
- 10:03like to this video If you find this
- 10:04video useful
- 10:06that's it for today see you in another
- 10:07great video Until Then cheers bye
About this transcript
This page contains the full transcript of Learn Azure Databricks in 10 Minutes- Explained simple | Azure Databricks Tutorials for Beginners by Mr. K Talks Tech, generated from the public captions YouTube serves with the video. The transcript has 1,750 words across 268 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.