How Uber Handles TRILLIONS of Transactions — Transcript

Full transcript

1. 0:00How can software handle billions of payments without missing
2. 0:03a single transaction?
3. 0:08Today
4. 0:08we're going to talk about the Uber engineers solution to.
5. 0:12Billions and billions of payments.
6. 0:13and easily explain how it all works.
7. 0:17So let's go.
8. 0:24Now we all know how Uber works.
9. 0:27Click a button that pays adjusted money for a leaf from one place to another.
10. 0:31Wait for your driver.
11. 0:32Get in the car and arrive at your destination.
12. 0:35Leave and then Uber pays the money the driver is owed.
13. 0:38Pretty simple right?
14. 0:39add 100 million users.
15. 0:42Billions of rides and hundreds of billions worth of money
16. 0:45being passed around later.
17. 0:47And then our easy solution is no longer an easy solution.
18. 0:51As Uber grew and grew in popularity, they would evolve their tech stack,
19. 0:55going from a monolith application aka just one big application.
20. 1:00They deployed to a microservice.
21. 1:01One which is like thousands of applications all over the place.
22. 1:05They'd switch to Postgres and MySQL.
23. 1:07And just build technology everywhere around it.
24. 1:10However, if there's one thing that you just
25. 1:12can't mess up in your text hack, it's what's dealing with money.
26. 1:16their system was messing up big time.
27. 1:18first it was just beyond unstable.
28. 1:20The system was fragmented and multiple different areas
29. 1:23and services transactions were really, really slow.
30. 1:27then reconciling transactions was just an absolute nightmare.
31. 1:30Payment discrepancies and just major delays were everywhere.
32. 1:33Imagine being an Uber driver, driving and giving rise for two days straight.
33. 1:37Then a cyberattack happens and then all your payments
34. 1:42that you've been expecting for the last few days, it's just gone.
35. 1:45Nowhere to be seen, nowhere collected.
36. 1:47So they rebuilt their payment system from scratch.
37. 1:50First, they switched their system to a job order system.
38. 1:55The job in this case represents the trip that you go on.
39. 1:57Or an Uber Eats order.
40. 1:59This job system allowed flexibility in having multiple orders
41. 2:02assigned to the same job.
42. 2:04So, for example, asking to stop at McDonald's before getting home.
43. 2:08We've all been there.
44. 2:09It's it's okay to admit it.
45. 2:11Inside these orders are entries, which represents money moving from a payer
46. 2:15to app.
47. 2:16The sum always zero.
48. 2:18No creating or destroying money.
49. 2:20So if I paid $20 for an Uber ride, it would come up in two different items.
50. 2:25trip fare, which is $18, and the service fee, which is $2.
51. 2:29This then goes into the Uber drivers account and then Uber's account.
52. 2:32Trip fare 18. Service fee $2.
53. 2:36This essentially is how double entry accounting works.
54. 2:39Credit and debit.
55. 2:40I've covered so many different types of databases on my channel
56. 2:44in the past, but none have specifically been designed
57. 2:47to tailor towards the need of money or finance.
58. 2:51So like, how would you even do that in the first place?
59. 2:54A ledger.
60. 2:55A line by line table of your transactions.
61. 2:58Don't ask the crypto people what a ledger means.
62. 3:00What makes ledger still popular to this day
63. 3:02is that it's the only way to truly know where you're financially.
64. 3:05If my bank account starts at $100 and ends at $1,000,
65. 3:09but the purchases and depositors don't end in $1,000,
66. 3:13well, some suss stuff is happening in my bank account.
67. 3:16doing two seconds of research on this, I can see that, you know, Postgres
68. 3:21or MySQL is clearly capable of doing something like this.
69. 3:25So why isn't it suitable for finance?
70. 3:28Well, first, most databases aren't immutable.
71. 3:32if I wanted to change my name on any website,
72. 3:34once I change it on the form, the database goes to the existing entry
73. 3:38and then updates it, removing the previous value.
74. 3:41This is because it is mutable.
75. 3:43if I were to try this on
76. 3:44an immutable database, it would have to create a brand new row
77. 3:48with the new information and duplicate existing information.
78. 3:52Then in the UI, I would have to grab the latest version of this.
79. 3:54So obviously this is just not good and so wasteful.
80. 3:59If you had a database of transactions and then they all equal to zero,
81. 4:03and then some dingus deleted a row out of that database.
82. 4:08you forever screwed up the integrity of your books.
83. 4:11This transaction is just gone forever.
84. 4:13So an immutable
85. 4:14system would force an append only system, meaning that if a mistake happens,
86. 4:18the correction is appended at the bottom of the database.
87. 4:21Rather than deleting the record completely.
88. 4:23For example, if I were to document a transaction, I could buy lunch for $10.
89. 4:27That's -$10 from my account.
90. 4:29However, the deli accidentally charged me $15.
91. 4:32So am I. Can't you see -$15?
92. 4:35Well, rather than just deleting the records and trying again,
93. 4:38the deli has to refund me $5, which then turns into a credit on my account.
94. 4:43You fix things by adding or subtracting to the end, not by replacing.
95. 4:48popular open source options exist like IMDb, but at scale.
96. 4:52Uber made the classic programing decision to make their own version
97. 4:55from scratch, so they created ledger store.
98. 5:03ledger store as an application layer
99. 5:04that sits above existing technologies like NoSQL databases.
100. 5:08It specifically adds three core concepts.
101. 5:11Ceiling closes a time range of changes.
102. 5:13This means that transactions that happened last month, you can look at
103. 5:16and validate that they are correct, making them now read only.
104. 5:20Uber specifically mentioned that they have 30 minute spans of seals.
105. 5:23Manifest are files generated and the sealing process.
106. 5:26This is a security measure
107. 5:27to show that only ledger store can access seal transactions in the past.
108. 5:30Just remember though, transactions are happening every second, so they use
109. 5:34Apache Kafka, which helps the streaming of this data in real time to calculate
110. 5:38and revisions.
111. 5:38As for when,
112. 5:39sometimes the data needs to be corrected in an already closed time range.
113. 5:43Revision is like a table
114. 5:44that shows the corrections that have happened
115. 5:46while also making the previous additions Queryable.
116. 5:49And leisure store.
117. 5:49It was great.
118. 5:50but Uber had a really, really big problem.
119. 5:57You see, Leisure
120. 5:57Store was built to be an opinionated about its underlying technology.
121. 6:01So when they moved to their new internal database,
122. 6:04they had to migrate over 250 billion records.
123. 6:09And so they only had two options
124. 6:12process all of this 300 terabyte of data in one go and then just dump it
125. 6:16into their new database, or break down historical data
126. 6:19into parts and process them individually and create checkpoints throughout.
127. 6:23The issue with doing number one is that with such a big data,
128. 6:27at any point that it goes wrong, you have to restart the entire process.
129. 6:32Think about installing a software for a long time and then it just fails.
130. 6:35On the last step.
131. 6:36So they went with option two and here's how it worked.
132. 6:43A worker boots up and reads the checkpoint
133. 6:45of where the process needs to be started from.
134. 6:48it then goes to cold storage where the data lives and processes
135. 6:51this data and moves it to the new database, and then updates the checkpoint
136. 6:55to say that the partition it filled out is successful so it doesn't get repeated.
137. 6:59And it then stores this in a metadata database so that other
138. 7:03simultaneous workers don't work on the same ones.
139. 7:06And this process only allowed that data to take a couple of weeks
140. 7:10and saved on a lot of money.
141. 7:11have our historical data taken care of on a separate database.
142. 7:15But how do we put this over live onto one of the busiest
143. 7:19applications of all time?
144. 7:25This is when Uber introduced the Shadow Rider.
145. 7:27The Shadow Rider sits in between two databases,
146. 7:30the current one and the one they wanted to migrate over to.
147. 7:34The Shadow Rider pushes data to the main database as usual,
148. 7:37synchronously, everything executed in order to keep things fast.
149. 7:41The Shadow Rider pushes data to the new database asynchronously,
150. 7:45meaning our system isn't dependent on if it saves right.
151. 7:48The second.
152. 7:48but here's another issue.
153. 7:50What happens if the items fail?
154. 7:55I'm. because it's asynchronous.
155. 7:57We don't know if it's going to work or not work.
156. 8:04To prevent this, they deployed a spot
157. 8:05repair job to look at any issues and fix them on the spot.
158. 8:08They also created a dual read and merge system.
159. 8:12This reads both databases at the same time and compares the two requests.
160. 8:16If the primary database had five records and the secondary had three,
161. 8:19it would merge the objects together and say that to the secondary database
162. 8:23and return to the user.
163. 8:25This put the new database into production gradually to see
164. 8:28how it handles under the pressure of a production system, but eventually
165. 8:32that leap of faith was needed to be done.
166. 8:38So they finally decided
167. 8:39to commit to the new databases and made the switch.
168. 8:43dock store being the primary and then Dynamo being the secondary.
169. 8:50Gradually, less requests were being read
170. 8:52from Dynamo and Doc Store continue to grow and grow.
171. 9:01Oh, and thank God for that.
172. 9:03But why all of this work?
173. 9:06Just to change databases.
174. 9:08When we talk about the ledger, we can think of it like a piece of paper.
175. 9:10With each row dedicated to a new entry.
176. 9:13I mean, makes sense for like 50 transactions.
177. 9:16But what about trillions of transactions
178. 9:20over billions of ledgers?
179. 9:27Indexes.
180. 9:29Now imagine this scenario.
181. 9:31You walk into a movie store and you want a copy of the movie jaws.
182. 9:36The cashier says, one moment and one by one,
183. 9:38make sure that each movie in a gigantic pile of movies is jaws.
184. 9:42I mean, you would be there forever.
185. 9:44It's just stupid.
186. 9:47An index in a database
187. 9:48would be categorizing each movie by the first letter or by genre.
188. 9:52So if I ask for jaws, you already know to look at aisle J,
189. 9:56But in this case 26 I think 26 indexes are just not going to cut it.
190. 10:01And this is why Uber has trillions of indexes
191. 10:05that they have managed in their database systems.
192. 10:07And there's different types.
193. 10:08A strongly consistent indexes used to handle credit card authorization.
194. 10:12When you use Uber, your credit card has a hold on it until the end of your trip.
195. 10:16If the database isn't able to look up the whole fast,
196. 10:19It risks charging you again and continuing this hold.
197. 10:22Yikes.
198. 10:23This is done with an intricate write and read process.
199. 10:25First, ledger store preps the indexes that will be inserting a database record.
200. 10:30Once it's prepped, it writes the record and if it's successful, will commit
201. 10:33the indexes needed.
202. 10:34They found that their indexes get stuck in the intent phase.
203. 10:37So to counteract this, if they read that there's a record in the database,
204. 10:41they commit the index.
205. 10:43Or if it doesn't it rolls back the index.
206. 10:45A lot goes into making this extremely reliable.
207. 10:48eventual consistent indexes.
208. 10:49As for parts of the application that doesn't require extreme precision.
209. 10:53So these could be places like previous rides greens where
210. 10:56waiting a little bit is fine as long as you get the end result.
211. 10:59Think about a ledger for a second.
212. 11:01One of the biggest key columns you could probably think of,
213. 11:05if not the most important one, is date and time.
214. 11:08Time range indexes does exactly that.
215. 11:10is able to index within a certain time frame of ledger entries.
216. 11:13When working with data, this is a great way to offload storage
217. 11:16that is really old.
218. 11:17In this case you can move data from hot storage.
219. 11:20Think like OneDrive.
220. 11:22You know easily accessible files all the way to cold storage, which is
221. 11:25just, you know, really cheap storage that takes a while to get your files.
222. 11:30So we now know that these types of indexes exist.
223. 11:33Great.
224. 11:33But how do we manage 2 trillion of them?
225. 11:37After the index is created, it back fills itself with historical data.
226. 11:42This is done in batches to save on data download speeds
227. 11:45then is validated for completeness.
228. 11:47This is done by using a checksum
229. 11:49comparison on the data and then decommission this index
230. 11:52so that the new index can get started and move along.
231. 11:57the idea that
232. 11:58so much technology goes into what we developers
233. 12:01or even users take for granted is just absolutely insane.
234. 12:05It just takes another level of engineering in order to figure all this stuff out.
235. 12:09There's a lot of cloud services that exist that specialize in
236. 12:13transitioning your database system over from one cloud provider or on premise
237. 12:17to another cloud or on premise database.
238. 12:22But at the scale of Uber, you have to figure out your own solutions,
239. 12:25because what works for you isn't something that's going to work for everybody.
240. 12:30Unless like you're left or something.
241. 12:31A big shout out to the three engineers from the Uber engineering team
242. 12:35that have written the articles that I've used to reference for this video.
243. 12:39literally without that, I wouldn't be able to make this video.
244. 12:42But also, I just think it's so cool that they can be vulnerable about
245. 12:46the different processes they've tried failed and just, you know, done.
246. 12:50What's the biggest transition you ever had to make?
247. 12:53Let me know in the comments below and don't forget to subscribe to the channel!
248. 12:55Peace out. Coders.