Optimization 2.0: faster, fasterrr! — Transcript

Full transcript

1. 0:00Hey guys, Sal. I just released a new
2. 0:01optimization mode and it is
3. 0:03significantly faster. Not only that, it
4. 0:05also gives us new options and features
5. 0:07which are super handy. I've been working
6. 0:08on this for months, so I cannot wait to
7. 0:11show it to you. So, let's get right into
8. 0:12it. So, in front of me, I have two
9. 0:14terminals open. On the left side, I have
10. 0:16the new version of Jessie, which has a
11. 0:18new optimization mode. And on the right
12. 0:19side, I have the old one. Now, I did
13. 0:21this because I want to run the same
14. 0:22strategy using the optimization mode so
15. 0:25we can see the differences to compare
16. 0:26the two together. Now, one way to
17. 0:28recognize the new Jessie is using this
18. 0:30new welcoming page, which will print out
19. 0:32the Jesse logo really big. Now, of
20. 0:33course, this is not going to affect your
21. 0:35trading results, but I just thought it's
22. 0:36really cool to have. But anyways, let's
23. 0:38go to the Jesse dashboard. And notice
24. 0:39the one with the port 9,0001 is the new
25. 0:41Jesse and the one with the port 9,0002
26. 0:44is the old one. So, let's generate a new
27. 0:45strategy and we're going to call it
28. 0:49optimization
29. 0:52benchmark. I will also do the same on
30. 0:54the new version of Jesse.
31. 0:57Instead of writing a strategy from
32. 0:59scratch, I'm going to copy the code for
33. 1:01this one from our
34. 1:03website. Go back to Jesse and paste it
35. 1:05here. I will also do the same thing one
36. 1:08more time here. Now, before I forget, we
37. 1:09also need to copy the name of the
38. 1:11strategy to be this new one. So, let's
39. 1:14save it and also do the same one here.
40. 1:17All right. So, now we need to update the
41. 1:19strategies so that it should be prepared
42. 1:21for running the optimization. Now, now
43. 1:23instead of preparing a strategy for
44. 1:24optimization manually, I'm going to do
45. 1:26the lazy way. So, I will just copy it
46. 1:29and open JGPT and I will say this one.
47. 1:32Prepare my strategy for optimization and
48. 1:34I will paste it right here. All right.
49. 1:36So, it gave me this one. Let's copy it.
50. 1:37Go here and paste it
51. 1:40here. All right. So, notice that now
52. 1:42instead of hard coding the values for
53. 1:44the period of this SMA and this one, we
54. 1:46are using the HP which stands for
55. 1:48hyperparameters. And here it has defined
56. 1:51the hyperparameters function for us
57. 1:53which has these two values with the type
58. 1:56of int the minimum value being 100 and
59. 1:58here being 10 the max being 300 and 100
60. 2:01and here are the default values which
61. 2:03are the ones that we had hardcoded in
62. 2:05the previous strategy code. Now by the
63. 2:07way in this one we are only trying to
64. 2:10optimize two integer values but we also
65. 2:12have support for float values and in
66. 2:14fact with the new optimization mode we
67. 2:16also have support for categorical type.
68. 2:17Now, if you don't know what that is,
69. 2:19that's fine. I will show it to you in
70. 2:20the end of the video. All right. So,
71. 2:21let's pick the optimization mode from
72. 2:23within the dashboard. So, again, this is
73. 2:25the new dashboard. And the way I know
74. 2:27this is because in the previous one,
75. 2:29let's actually open it. We only had one
76. 2:31duration, but with the new one, we have
77. 2:33durations for training period and the
78. 2:35testing period. Now, in the previous
79. 2:37version, the training and testing
80. 2:38periods were being generated behind the
81. 2:40scenes for you. So, you only had control
82. 2:42over the total duration of the
83. 2:44optimization run. But with the new one,
84. 2:46you can specify whichever period you
85. 2:49want for training and for testing.
86. 2:50Another difference is that you can also
87. 2:52specify the CPU course to use right from
88. 2:54here instead of from the settings page
89. 2:56which was how we used it in the past. So
90. 2:59in fact, let's update this one to be
91. 3:01another number such as 12. And I will
92. 3:03leave all the other options to be as
93. 3:05they are. All right. So let's also pick
94. 3:07this strategy. It is simply called
95. 3:09optimization benchmark. And I'm going to
96. 3:11choose the 4 hours as a time frame. the
97. 3:14route is correct. As for the duration, I
98. 3:17will pick until the end of 2024 and I
99. 3:21will leave the number of trades as 50. I
100. 3:23will also turn on the fast
101. 3:25mode. All right, this looks good. Now,
102. 3:27let's also go to the new one and pick
103. 3:30the
104. 3:31same strategy. I will change the time
105. 3:34frame to 4
106. 3:36hours. We don't need any data routes.
107. 3:39As for training period, I will pick 2024
108. 3:43until the end of it. But for my testing
109. 3:46period, actually just to keep them the
110. 3:48same, I want to make sure the testing
111. 3:50period of the new dashboard is the one
112. 3:52we had for the previous one. So in the
113. 3:54previous one, we were using 85% of the
114. 3:57data for training and 15% for testing.
115. 4:00So approximately that makes this like
116. 4:03two months less. So again, let's make
117. 4:06sure this one is correct.
118. 4:11And this one should also be turn 24
119. 4:14since the beginning and until the end of
120. 4:17it which in other words would be the
121. 4:19beginning of 2025. As for optimal number
122. 4:21of trays I will pick the same number 50
123. 4:24and as for the CPU course we are using
124. 4:28it is set to 12. The fast mode is
125. 4:30already on. All right. So first I'm
126. 4:32going to run the optimization using the
127. 4:33previous version. So let's click on the
128. 4:35start
129. 4:36button. All right. All right. So, I'm
130. 4:37getting this error because I implemented
131. 4:40the hobby parameters method on the
132. 4:43strategy of the new optimization mode,
133. 4:44but I forgot to do the same for the
134. 4:46previous one. So, let's also open this
135. 4:48one and paste it
136. 4:49here. Go back and run it again. All
137. 4:52right. So, it began working. And as you
138. 4:54can see in the previous version, we had
139. 4:56to run the optimization in two phases.
140. 4:59The first phase which this one is in
141. 5:01right now, we had to generate the
142. 5:04initial population and then use that
143. 5:06initial population to evolve it and find
144. 5:09good parameters. But with the new one,
145. 5:11that's not the case. We just begin
146. 5:12optimization and begin evolving
147. 5:14immediately. And this is because the
148. 5:16algorithm behind them. The previous
149. 5:17version was written by me and it was
150. 5:19using the genetic algorithm. But the new
151. 5:21version is using the Optuna library and
152. 5:23their algorithms which have been
153. 5:26optimized through the years and they
154. 5:28work really fast and really good. All
155. 5:29right, so it's been about 45 minutes
156. 5:31since I started this and we are at
157. 5:34nearly 80% and we cannot see the sharp
158. 5:37ratio in the previous dashboard, but we
159. 5:39can see the training and testing P&L and
160. 5:42the total number of trades and the win
161. 5:44rate. All right, so let's cancel this
162. 5:45one and go to the new dashboard. And
163. 5:48again, notice that everything is the
164. 5:50same. So, and using again 12 cores of
165. 5:53CPU. Let's run this. All right. So,
166. 5:56first of all, it's going so fast that I
167. 5:59don't think it's going to take more than
168. 6:00just a few minutes. It's been 23 seconds
169. 6:02and we're already at 34%. We also have
170. 6:05this chart here which shows us the
171. 6:07objective progress. We also have the
172. 6:09ability to change the metrics value
173. 6:10here. By default, it is set to whatever
174. 6:12you choose in the settings for the
175. 6:15fitness function. Now, by default, it is
176. 6:17set to sharp and that's the value here.
177. 6:20But I could change it into other things
178. 6:21such as calma ratio, omega or even the
179. 6:24piano. Also, while this is going, we can
180. 6:27see here that in the new table, we have
181. 6:30the ability to click on this to get more
182. 6:32info. So, we can see the parameters, the
183. 6:34translated values of it. We can copy the
184. 6:37DNA from here. We can also see the full
185. 6:39list of the metrics for both the
186. 6:41training and the testing period, which
187. 6:43allows us to compare all the values that
188. 6:45we actually care about. All right, so
189. 6:46it's been 59 seconds and we are at 92%.
190. 6:49So that one took nearly an hour, but
191. 6:52this one finished in about 1 minute. So
192. 6:55that's how much faster this new
193. 6:56optimization mode is. And you don't even
194. 6:58have to copy the DNA, go to the back
195. 7:00test, paste it there, just so you can
196. 7:02see the translated values or the other
197. 7:04metrics. You can simply click on this
198. 7:06button and see all the metrics that you
199. 7:08actually care about. And again, you can
200. 7:09also see the parameters. Now, if you
201. 7:11like the values, just like before, copy
202. 7:13the DNA value from here. go to the
203. 7:15strategy and add a
204. 7:17function called DNA which simply returns
205. 7:22this string value. Now notice that the
206. 7:24new DNAs are a bit longer than the
207. 7:26previous ones, but that's perfectly
208. 7:28fine. And also in case you're wondering
209. 7:30whether you have to update your existing
210. 7:32strategies to work with the new one,
211. 7:34well, you don't. I made sure that if you
212. 7:36enter a DNA with the previous format, it
213. 7:38will continue to work so that you guys
214. 7:40don't have to update your strategies
215. 7:42immediately. But if you use the new
216. 7:43optimization mode and get a DNA with the
217. 7:45new format, this one will also work. The
218. 7:47reason the new optimization mode is so
219. 7:49much faster is because we are using
220. 7:50first the optuna library which is this
221. 7:53one. And second, for doing
222. 7:55multipprocessing, we are using the ray
223. 7:57library which is super optimized. But
224. 8:00there's a caveat. The ray library at the
225. 8:03moment of recording this video is not
226. 8:04supported on Python 3.13 which is the
227. 8:07latest version of Python that we support
228. 8:09Jesse on. So if you want to use the new
229. 8:11optimization mode, make sure your Python
230. 8:13version is nothing more than 3.12. All
231. 8:15right. So now that you can see how much
232. 8:17faster the new optimization mode is,
233. 8:19let's continue so that I can show you
234. 8:21the new options that it supports. So one
235. 8:23thing is that previously we had the
236. 8:26minimum and the maximum values for let's
237. 8:28say an integer type that we wanted to
238. 8:30optimize. But with the new version, we
239. 8:32also have another option called step.
240. 8:39So for example we can specify 10 and now
241. 8:42it will not try for example the number
242. 8:44101 102 it will only increase by 10. So
243. 8:49for example 110 120 etc. This by the way
244. 8:53was one of the most requested features
245. 8:54of JC for the optimization mode. The
246. 8:57other thing that's new is the support
247. 8:59for categorical type. Now let me show
248. 9:01you what that is and how you can use it.
249. 9:03So for example here in this strategy we
250. 9:06are using the SMA indicator. So what if
251. 9:08I wasn't sure whether to use SMA or EMA?
252. 9:12I could simply come here and say this MA
253. 9:15standing for moving average equals
254. 9:19HP MA type. Sorry, this should be
255. 9:24self.Hp. All right. And I'm going to say
256. 9:27if MA equals SMA, then return this. If
257. 9:31it equals EMA, then return this. Now, we
258. 9:34don't need all the rest. Now, this one
259. 9:36is good practice. So maybe I just keep
260. 9:38it here, but you don't need it. All
261. 9:39right. So if the MA equals SMA, we're
262. 9:42going to return TA. SMA, but if it
263. 9:45equals EMA, we will return TA. EMA. So
264. 9:48let's copy this one. Come down to the
265. 9:51hyperparameters. Now here now I have the
266. 9:54option to define a new parameter called
267. 9:57MA type. And the type of it well, it's
268. 10:00not CR, it should be
269. 10:03categorical. And the options are SMA,
270. 10:06EMA. Now, they could be as many as you
271. 10:08like, of course, but we only care about
272. 10:10SMA and EMA. And we also define a
273. 10:13default value, which is SMA. All right.
274. 10:15So, now let's go back to
275. 10:18Jesse, pause, and start a new session.
276. 10:21And start it
277. 10:25again. All right. So, let's click on the
278. 10:27info button. And now, if you check out
279. 10:30the parameters, you can see we also have
280. 10:32the MA type. And for example with these
281. 10:34results we are using the EMA. But if I
282. 10:36check another one for this one is also
283. 10:40EMA. This one is also
284. 10:45EMA. This one for example is SMA. You
285. 10:48see? So with this new type, you can use
286. 10:50the optimization mode to choose the
287. 10:52right indicator for your strategy. Now
288. 10:54this can be super helpful and I'm going
289. 10:56to leave it to you to come up with all
290. 10:58sorts of ideas that you can use it with.
291. 10:59All right, so let's pause this one.
292. 11:01Again, notice how fast it is going. It's
293. 11:03crazy. So, we also have other options.
294. 11:05For example, how many trials you want to
295. 11:08run per each hyperparameter. Well, that
296. 11:10one is now a settings. Now, go to the
297. 11:12settings page to the optimization tab.
298. 11:14And here you can modify the number of
299. 11:16trials you want to run per each
300. 11:18hyperparameter. So, for example, if you
301. 11:20run optimization and you see that it
302. 11:22hasn't yet found a good enough parameter
303. 11:24for you, you can come here and increase
304. 11:26this number. Or for example, if you
305. 11:28found that it has already found a good
306. 11:30number and it is just working too hard
307. 11:32or maybe overfitting your strategy, you
308. 11:35can come here and lower this number. Or
309. 11:36if you don't care about it, just simply
310. 11:38use the default option. As you just saw,
311. 11:40the new optimization mode is
312. 11:42significantly faster, which allows us to
313. 11:44run the optimization mode whenever we
314. 11:46want. We don't have to wait for hours or
315. 11:48days for it to finish. Instead, it's
316. 11:50going to finish in a few minutes or
317. 11:52hours at the maximum. Now, I think this
318. 11:54changes everything and how we tackle our
319. 11:56strategies altogether. I'm super excited
320. 11:58to see what you guys build with it. So,
321. 12:00make sure to post a comment and let me
322. 12:01know about it. Thank you guys for
323. 12:02watching and I'll see you in the next
324. 12:04one.
325. 12:09[Music]
326. 12:10[Applause]
327. 12:12[Music]