YouTube2Text

Learn Azure Databricks in 10 Minutes- Explained simple | Azure Databricks Tutorials for Beginners — Transcript

by Mr. K Talks Tech · 1,750 words · 268 segments · language en · Watch on YouTube

Full transcript

  1. 0:01[Music]
  2. 0:03hello everyone welcome back to my
  3. 0:06Channel today we are going to say about
  4. 0:08assure data breaks I have created a
  5. 0:10separate playlist for issue data breaks
  6. 0:12and I'll be using that playlist to
  7. 0:14upload all the videos related to Azure
  8. 0:16databricks so this is the first video in
  9. 0:18that playlist okay so now let's get
  10. 0:20started so firstly what is azure
  11. 0:23databricks it is an Apache spark based
  12. 0:26analytics platform which is mainly used
  13. 0:27for doing big data workloads this tool
  14. 0:30is very popular and it is used by data
  15. 0:32Engineers data scientists and also data
  16. 0:34analyst so pretty much all the data
  17. 0:37professionals use this tool in Azure so
  18. 0:39if you ask me why this tool is really
  19. 0:41popular I would say there are a lot of
  20. 0:43interesting features within Azure
  21. 0:45databricks so let's see all these
  22. 0:47individual features within data breaks
  23. 0:49one by one with an example so firstly
  24. 0:52the most important feature is Apache
  25. 0:54spark so what is Apache spark it is an
  26. 0:58open source distributor processing
  27. 0:59system which is made in the use for
  28. 1:01doing big data workloads so this Apache
  29. 1:05spark is inbuilt to assure data bricks
  30. 1:07and it uses the Spark engine for all the
  31. 1:10data processing
  32. 1:12so when we use Su data breaks to process
  33. 1:15any data The inbuilt Spark engine take
  34. 1:18care of all the partitioning and other
  35. 1:20optimization by itself which makes it
  36. 1:23extremely powerful and that's the reason
  37. 1:25databricks is widely used for data
  38. 1:27transformation in our show so we can
  39. 1:30create high performance spark compute in
  40. 1:32Azure databricks for doing the data
  41. 1:34processing there are wide varieties of
  42. 1:36spark compute available in data breaks
  43. 1:38with the different CPU and memory size
  44. 1:40it is very simple to configure based on
  45. 1:43the requirements and it is highly
  46. 1:44scalable
  47. 1:45say for example if you create a cluster
  48. 1:48using a smaller compute and in the later
  49. 1:50case for some reason you want to
  50. 1:52increase the cluster size it is very
  51. 1:54easy to do it in a click of a button
  52. 1:56which makes it extremely scalable so one
  53. 1:58thing to note here is the cause for this
  54. 2:00cluster is charged only when we are
  55. 2:02actually using it so we can turn off
  56. 2:04this cluster if we are not using it and
  57. 2:06we are not going to pay for it which is
  58. 2:08really cool right the next big feature
  59. 2:10in Azure data brexit notebooks in data
  60. 2:13breaks all the development work like
  61. 2:15data processing or model training is
  62. 2:17done using the notebooks a notebook is a
  63. 2:20kind of code editor which is in the
  64. 2:22format of different cell based structure
  65. 2:24you could type in the code to see the
  66. 2:27immediate results inside the notebook
  67. 2:28itself which is very easy and convenient
  68. 2:31to work with
  69. 2:32since it is a cell based editor you can
  70. 2:35use each cell to perform different
  71. 2:37functionalities which is very easy to
  72. 2:39understand and maintain as well having
  73. 2:42the notebook option inbuilt is another
  74. 2:43great feature of azure databricks the
  75. 2:46next feature is it is very easy to
  76. 2:48switch programming languages in
  77. 2:49databricks
  78. 2:50databricks support different programming
  79. 2:52languages like Pi spark spark SQL Scala
  80. 2:56and r
  81. 2:57so when you create your notebook in data
  82. 2:59bricks the default programming language
  83. 3:01you can type in the notebook is python
  84. 3:04you can easily switch to other
  85. 3:05programming languages in the same
  86. 3:07notebook say for example if you are very
  87. 3:09good at SQL then you can use a feature
  88. 3:12called Magic command in data breaks if
  89. 3:15you could type percentage SQL in the
  90. 3:16cell the cell automatically converts to
  91. 3:18your SQL cell where you could type SQL
  92. 3:21code to perform the data processing
  93. 3:23similarly you could use the magic
  94. 3:25command to convert the cell to different
  95. 3:27language cell such as spice Park spark
  96. 3:30SQL Scala and R this is extremely useful
  97. 3:34since you could use multiple language in
  98. 3:36the same notebook for processing the
  99. 3:38data
  100. 3:39these are some of the features of Apache
  101. 3:41spark in RC data bricks the next
  102. 3:43important feature in Azure databricks is
  103. 3:46ETL
  104. 3:47we all know ETL stands for extract
  105. 3:49transform and load although issue
  106. 3:52databricks is mainly used for the
  107. 3:54transformation part in the ATL framework
  108. 3:56we could use Ashu data bricks as an
  109. 3:59individual tool to perform the complete
  110. 4:02ETL process
  111. 4:03let's see how we could do that firstly
  112. 4:06we could use Spar connector for doing
  113. 4:08the extraction part
  114. 4:10with spark connector we could use
  115. 4:12databricks notebook to connect to
  116. 4:14multiple sources and ingest the data
  117. 4:17from source to destination
  118. 4:19the spark character is supported in many
  119. 4:21data sources for example SQL server
  120. 4:24issues SQL database mongodb and several
  121. 4:28other data sources
  122. 4:30for interesting the data from these data
  123. 4:32sources we don't want tools like Azure
  124. 4:35data Factory or Russia synapse analytics
  125. 4:37for performing the injection instead we
  126. 4:40could also use spark connector to
  127. 4:42connect to these data sources to extract
  128. 4:44the data using issue databricks itself
  129. 4:47and the next one is the transformation
  130. 4:48part we all know that databricks is the
  131. 4:51most preferred tool in Azure for doing
  132. 4:53the data transformation due to its
  133. 4:56inbuilt Apache spark option
  134. 4:59and the final one is the loading part we
  135. 5:01could also use a shoe data bricks for
  136. 5:03loading the data into the database using
  137. 5:06the high metastore feature
  138. 5:08with high metastore we can create
  139. 5:10databases and tables where we can load
  140. 5:13the transform data to it
  141. 5:15the tables can also be queried using the
  142. 5:18new SQL editor feature inside the
  143. 5:20databricks or can also be consumed by
  144. 5:23different reporting tools like power bi
  145. 5:25or Tableau so due to all of these
  146. 5:27different features we can use ASU
  147. 5:30databricks to do the complete ATL
  148. 5:32process also more than actually
  149. 5:34performing the ETL process we can also
  150. 5:37use issue databricks to orchestrate the
  151. 5:40ETL so what I mean by this is
  152. 5:43inertia data breaks we have an option
  153. 5:45called workflows using workflows we can
  154. 5:49run the notebook that we create using
  155. 5:51Azure databricks as a job which means
  156. 5:53that we can schedule the notebooks to
  157. 5:55run it on a regular basis
  158. 5:58due to this future using IC data breaks
  159. 6:01we can orchestrate the complete ETL
  160. 6:03process by configuring when and what
  161. 6:06notebook to run with the different
  162. 6:07triggers and stuff this makes issue data
  163. 6:10breaks as a complete ETL tool and hence
  164. 6:13its most popular tool among the data
  165. 6:15Engineers the other major feature in
  166. 6:17Azure databricks is data science and
  167. 6:19machine learning
  168. 6:21we can use data bricks to train a
  169. 6:23machine learning model
  170. 6:24also we can serve or deploy the model
  171. 6:27that has been trained databricks is well
  172. 6:30suited for doing data science and
  173. 6:31machine learning stuff as it has inbuilt
  174. 6:34features such as option to train automl
  175. 6:37experiment and also we have dedicated
  176. 6:39spark clusters available for doing the
  177. 6:42machine learning stuff with higher gpus
  178. 6:44suitable for model training we all know
  179. 6:46that for training a machine learning
  180. 6:48model with huge data sets requires
  181. 6:50higher GPU and we could configure it
  182. 6:53easily in databricks also there is a
  183. 6:56dedicated tab called serving to deploy
  184. 6:58the train model so once the model is
  185. 7:00deployed we can make use of the model
  186. 7:03endpoint for doing any kind of
  187. 7:05predictions
  188. 7:06so due to these features similar to how
  189. 7:09databricks is very popular among the
  190. 7:10data Engineers it is also popular among
  191. 7:13data scientists and thereby it is widely
  192. 7:16used to tool for doing machine learning
  193. 7:17and data science work in Azure the next
  194. 7:20important feature is streaming
  195. 7:22we can say that Ashu databricks is a
  196. 7:24perfect tool to work with streaming data
  197. 7:27there are different features in data
  198. 7:28bricks that allows us to work seamlessly
  199. 7:31with streaming data
  200. 7:32for example we have structure streaming
  201. 7:35which is an Apache power based streaming
  202. 7:37solution because of this we can perform
  203. 7:40near real-time data ingestion and also
  204. 7:43can perform near real-time machine
  205. 7:44learning and predictions
  206. 7:46due to this streaming capabilities we
  207. 7:49could ingest the data faster from source
  208. 7:51to destination in the streaming fashion
  209. 7:53instead of doing in a batch load on a
  210. 7:56scheduled basis
  211. 7:57in the future videos we can explore
  212. 8:00these streaming functions and how we
  213. 8:01could work with that the next feature is
  214. 8:03the SQL editor we have seen how data
  215. 8:06engineers and data scientists can work
  216. 8:08with databricks and this time this is
  217. 8:11all about the data analyst in Azure data
  218. 8:14breaks we have an option called SQL
  219. 8:15editor which is kind of similar to SQL
  220. 8:18Server management Studio
  221. 8:20you could use the SQL Editor to query
  222. 8:22the tables that we have in the database
  223. 8:24and Analysis can be done on the data in
  224. 8:28most cases a data analyst worked with a
  225. 8:30cleaner data that would be querying it
  226. 8:33to find insights from it they can make
  227. 8:35use of this Editor to write SQL commands
  228. 8:37to query the data we need to use
  229. 8:40databrick SQL query language to query
  230. 8:42the data of in databricks which is kind
  231. 8:44of similar to spark SQL
  232. 8:46so due to these features a data analyst
  233. 8:49can also work with azure databricks
  234. 8:52after this we have an interesting
  235. 8:53feature which is easy collaboration
  236. 8:56Azure databricks can be used by multiple
  237. 8:58people at the same time and can work
  238. 9:00collaboratively for performing any kinds
  239. 9:03of tasks this is possible due to its git
  240. 9:05integration functionality the popular
  241. 9:07git repos such as GitHub and ushi devops
  242. 9:10is supported in RC data bricks we have a
  243. 9:13separate tab called repos in Azure
  244. 9:15databricks workspace where we could use
  245. 9:17that to integrate with the different
  246. 9:18repos such as GitHub and acid devops
  247. 9:21this feature leads to EC collaboration
  248. 9:23and working collaboratively in doing any
  249. 9:26kinds of data engineering or data
  250. 9:27science work is important and we can do
  251. 9:30that very well using ashy data bricks so
  252. 9:32these are some of the main features of
  253. 9:34RC data breaks I think by this time you
  254. 9:37should be more excited to work with data
  255. 9:39bricks if you haven't worked with it
  256. 9:41before trust me this is a great tool to
  257. 9:43work with for any data Engineers or data
  258. 9:46scientist or data analyst I guess now
  259. 9:49you have a clear understanding about
  260. 9:50what exactly data is and its important
  261. 9:54features I'll be adding a lot of videos
  262. 9:56later to data breaks in this playlist so
  263. 9:58please subscribe to my channel if you
  264. 10:00want to learn more about it also give a
  265. 10:03like to this video If you find this
  266. 10:04video useful
  267. 10:06that's it for today see you in another
  268. 10:07great video Until Then cheers bye

About this transcript

This page contains the full transcript of Learn Azure Databricks in 10 Minutes- Explained simple | Azure Databricks Tutorials for Beginners by Mr. K Talks Tech, generated from the public captions YouTube serves with the video. The transcript has 1,750 words across 268 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.