01 PySpark Tutorial | PySpark Training | Learn from Basics to Advanced Performance Optimization — Transcript

Full transcript

1. 0:00hello and welcome back to a brand new
2. 0:03course Pi spark Zero to Hero
3. 0:05now before you think what is different
4. 0:07in this course let me give you three
5. 0:09reasons why you should look into this
6. 0:11course
7. 0:12first park is in huge demand for data
8. 0:15Engineers not only for data Engineers
9. 0:17even data analysts and data scientists
10. 0:19are also using Spark
11. 0:21second we are in threat generative AI is
12. 0:25on rise it can write any type of code
13. 0:28that you need
14. 0:29so we must need to have a difference
15. 0:32from writing code understand how and why
16. 0:36the things are important and how the
17. 0:38things work in the background
18. 0:40third
19. 0:41I will be sharing many tips with you and
20. 0:43tricks that will use in general
21. 0:45production scenario when we work with
22. 0:48spark
23. 0:48so to begin with hi I am shubham and I
24. 0:52will be your instructor throughout this
25. 0:54course believe me by the end of this
26. 0:56course you will be a hero in pi Spark
27. 0:59to start the course let's see what is
28. 1:02Spark
29. 1:03spark is nothing but a open source
30. 1:05unified Computing engine for parallel
31. 1:07data processing
32. 1:09distributed computing is on Rise and
33. 1:11Spark is one of them it supports major
34. 1:13programming languages such as Java Scala
35. 1:15python R and Spark is built up on Scala
36. 1:19and the important reason why we should
37. 1:21use spark if our data processing is
38. 1:23spark is 100 times faster than map
39. 1:26reduce of traditional Hadoop spark
40. 1:29processes data using graph which is in
41. 1:31memory and which is very fast what are
42. 1:34the spark important components now I
43. 1:37will not dig much more into spark
44. 1:38components because you have seen a lot
45. 1:41of documents lot of videos about it I'll
46. 1:43just tell you what is important these
47. 1:45are the spark components on High level
48. 1:47structure spark consists of three
49. 1:49components low level API structured API
50. 1:52libraries and ecosystem
51. 1:55the low level API is the bottom
52. 1:57structure of spark component and the
53. 1:59structure API is built upon low level
54. 2:00apis low level apis are generally
55. 2:03consist of rdd which is resilient
56. 2:05distributed data and distributed
57. 2:07variables structured apis such as spark
58. 2:10SQL data Frame data sets are built on
59. 2:12top of rdd and are optimized for use
60. 2:16the top level consists of libraries and
61. 2:19ecosystem such as structured streaming
62. 2:21which is built up upon structured apis
63. 2:23Advanced analytics graph query languages
64. 2:26and other such huge libraries that spark
65. 2:28support this is all as an introduction
66. 2:31for spark I'll not bore you with too
67. 2:33much of introduction for spark you can
68. 2:35go and learn a lot about spark in
69. 2:37Internet if this is all we need to know
70. 2:39for now in the next video we will see
71. 2:42how spark Works under the hood we will
72. 2:44see how spark distributed data and how
73. 2:47spark Works in a parallel Computing way
74. 2:50keep learning keep growing keep sharing