The Scientific Way to Test Technical Analysis — Transcript

Full transcript

1. 0:00Does technical analysis actually work?
2. 0:02Some traders swear by it. Others think
3. 0:05it's a complete nonsense. The problem is
4. 0:07that technical analysis isn't one single
5. 0:10method. Every indicator, pattern, and
6. 0:12trading rule makes a different claim
7. 0:14about the market. David Erensson's book
8. 0:16called Evidence-based Technical Analysis
9. 0:18demands a stricter approach. Define a
10. 0:21rule clearly enough to be tested or
11. 0:23accept that it isn't useful at all. The
12. 0:25rule must then be evaluated
13. 0:27statistically to determine whether its
14. 0:29historical performance shows real
15. 0:31predictive power or is merely the result
16. 0:33of luck. This is called rule
17. 0:36significance testing. In this video,
18. 0:37we're going to understand how rule
19. 0:39significance testing works, why a
20. 0:42profitable back test isn't enough, and
21. 0:44then use Python to define a trading rule
22. 0:47and run a significance test ourselves.
23. 0:49Before we can test a trading rule, it
24. 0:51needs to be objective. For example,
25. 0:53imagine someone says buy when the price
26. 0:55bounces off the trend line. And the
27. 0:57reason is different traders may draw
28. 0:59that trend line differently. They may
29. 1:01also disagree about what counts as a
30. 1:03bounce unless those decisions are
31. 1:05defined precisely. We cannot test the
32. 1:07rule consistently. Now compare that with
33. 1:10a rule that says go long when the 50
34. 1:12period EMA crosses above the 100 period
35. 1:15EMA. We can write it in Python, run it
36. 1:18on historical data and get the same
37. 1:20signal every single time. This does not
38. 1:22mean the EMA rule is profitable. It
39. 1:24simply means that we have a clear claim
40. 1:27that can be tested with evidence. A
41. 1:29profitable back test does not
42. 1:31necessarily mean that the rule can
43. 1:33predict the market. Imagine a strategy
44. 1:35that randomly generates long only
45. 1:37signals throughout a powerful bull
46. 1:39market. It may produce an impressive
47. 1:41back simply because prices are broadly
48. 1:43rising, not because its signals contain
49. 1:46useful information. It simply benefited
50. 1:48from the direction of the market, not
51. 1:50from its ability to predict it. There's
52. 1:53also another problem called data mining
53. 1:55bias. If we test hundreds of different
54. 1:57trading rules and keep only the most
55. 2:00profitable one, the winner may simply be
56. 2:02the luckiest one, not the most
57. 2:04predictive. So instead of asking only
58. 2:06whether the backis was profitable, we
59. 2:09need to ask a better question. Is the
60. 2:11rule actually useful or did it just get
61. 2:14lucky? We begin the assumption that the
62. 2:16rule has no predictive power. This is
63. 2:18called the null hypothesis. The rule
64. 2:20produces one of three signals on each
65. 2:22candle. Long, short, or neutral. A long
66. 2:25signal is represented by + one, a short
67. 2:28signal by minus one, and a neutral
68. 2:30signal by zero. Each signal is then
69. 2:32compared with the market return that
70. 2:34follows it. Before evaluating the
71. 2:36signal, the market's overall trend is
72. 2:38removed. This is called dtrending. D
73. 2:40trading prevents a strategy from
74. 2:42receiving free credit simply because it
75. 2:44stayed long during a rising market or
76. 2:47short during a falling market. Next, we
77. 2:49use bootstrap simulations to generate
78. 2:51thousands of result representing what
79. 2:53chance could produce if the rule had no
80. 2:56real edge. Finally, we compare the
81. 2:58actual rule with those simulated
82. 2:59results. The result is expressed as a p
83. 3:02value. For example, a p value of 0.03 03
84. 3:06means that if the rule had no real edge,
85. 3:08luck alone would produce a result this
86. 3:11good only about 3% of the time. The
87. 3:13lower the p value, the harder it is to
88. 3:15explain the result through luck alone.
89. 3:18But this does not mean the strategy has
90. 3:20a 97% probability of being profitable.
91. 3:23The p value only tells us how unusual
92. 3:26this historical result would be if the
93. 3:28result had no edge. For example, let's
94. 3:30get started with an actual back test
95. 3:32result. So take a look at this. It is
96. 3:35generating more than 400 in P&L. The
97. 3:38annual return is more than 2,000. The
98. 3:40win mean rate is 65% and the sharp ratio
99. 3:42is 3.17. So looks awesome, right? And
100. 3:45this is also the equity curve of the
101. 3:48actual portfolio when we compare to the
102. 3:50benchmark which is the price of BTC USDT
103. 3:53over this period. So you might think
104. 3:55this result actually looks good. But
105. 3:56before we jump into conclusion, we have
106. 3:58to consider the fact that price of BTC
107. 4:00itself was also going up. So even a buy
108. 4:03and hold a strategy would have made
109. 4:05money. So that begs the question, does
110. 4:06the strategist code have some real edge
111. 4:09or is it just pure luck? And in fact, if
112. 4:11I show you the code for the strategy,
113. 4:14you can see in the shoot long method, we
114. 4:16have this. So it's not exactly a random
115. 4:18function, but it is making decisions
116. 4:20based on the time stamp of the candle.
117. 4:23We're not even considering the price at
118. 4:25all. So of course, this strategy does
119. 4:28not have any edge. But again, when I
120. 4:30look at the result of the back test, it
121. 4:32looks good. It's going up. So, how do we
122. 4:35make a final decision? Well, that's when
123. 4:37the rule significance testing comes in
124. 4:39hand. So, if I show you the next result,
125. 4:42this is for a rule significance test
126. 4:44over Jess's dashboard. Now, by the way,
127. 4:46as always, I'm using the Jesse framework
128. 4:47to do my analysis. And specifically for
129. 4:50this use case, I'm using the UI
130. 4:52dashboard version to show you the result
131. 4:54of the test, but you do not have to use
132. 4:56this. You can also use the research
133. 4:58module of Jessie which provides the rule
134. 5:01testing completely for free. And on this
135. 5:03page, you can also see some example code
136. 5:05in Python. Now, while that's going, I
137. 5:07want to quickly remind you guys about
138. 5:09our Telegram. It's the fastest way to
139. 5:10get notified about my future work,
140. 5:12whether it's a new tutorial or a tool
141. 5:14that I create. Also, don't forget to
142. 5:16check out our free Discord where more
143. 5:18than 5,000 members like you and I are
144. 5:20hanging out there and helping out each
145. 5:21other with algo trading so we can all
146. 5:23succeed together. The links for both are
147. 5:25down in the description. And you can
148. 5:27also ask Jess's MCP to run the tests for
149. 5:30you. In fact, that's what I did. So, I
150. 5:31just asked it to write both the
151. 5:33strategies and run the rule tests for
152. 5:35me. And at the end, it just gave me the
153. 5:37results. So, I just clicked on them and
154. 5:39then I was inside the dashboard seeing
155. 5:41the results as I am right now. So,
156. 5:43anyways, looking at the result of the
157. 5:44test, we can clearly see that it is
158. 5:47failing. So, it says so right here. So,
159. 5:49the result is not significant. And if
160. 5:51you take a look, the observed mean of
161. 5:53the actual back test is sitting almost
162. 5:56in the middle. So it definitely is not
163. 5:58beating these simulations at all. To be
164. 6:00more precise, we want a p value to be
165. 6:03less than 0.10. But even that may not be
166. 6:06great. So preferably we want it to be
167. 6:09below 0.05.
168. 6:11And the smaller this number is, we will
169. 6:13have more confident in the result that
170. 6:15we are seeing. Now let's compare the
171. 6:16same exact period with this one. So this
172. 6:19one also has an equity curve that is
173. 6:21going up. It is the exact same time
174. 6:23period. So the market was going up
175. 6:25during this time as well. So again,
176. 6:27we're going to have to ask the question,
177. 6:29how do we ensure the result is not luck
178. 6:32and that the strategy actually has some
179. 6:34edge. So before I show you the result of
180. 6:36the rule significance test, let's take a
181. 6:38look at the strategies code. So this one
182. 6:40is actually defining some indicator. So
183. 6:43a fast, a slow one, the ADX indicator,
184. 6:46and the dungeon channel. So the inter
185. 6:48rule of the strategy is actually
186. 6:51meaningful. So it is saying that
187. 6:52whenever a crossover happens here and
188. 6:54the ADX is above good threshold and the
189. 6:57current price is above the previous
190. 6:59dungeon channel upper. So basically we
191. 7:01are also checking to ensure whether or
192. 7:03not a breakout is happening. That's only
193. 7:06the time that we actually want to go
194. 7:08long. So this one actually makes sense.
195. 7:10And these are some strategies that
196. 7:12researched in the past by so many
197. 7:14people. And when we take a look at the
198. 7:16results of the rule testing for this
199. 7:18one, we can see that it says it is
200. 7:20statistically significant. The p value
201. 7:23is below 0.05.
202. 7:25And if we take a look at this chart, we
203. 7:27can see the observed the mean of the
204. 7:30strategies return is indeed an outlier
205. 7:33and it is not sitting like in the middle
206. 7:35or something like the other one. So
207. 7:37again, this was the previous one, the
208. 7:39one that isn't passing and this is the
209. 7:42one that is indeed passing. But if you
210. 7:44are using Jess's dashboard, you can just
211. 7:46look take a look at this part and it
212. 7:48will tell you whether or not it is
213. 7:49statistically significant or not. And by
214. 7:52the way, in case you want to actually
215. 7:54run this test yourself, this is the form
216. 7:56of this is dashboard. So you choose an
217. 7:58exchange a time period, you pass a
218. 8:01trading route which asks for the symbol,
219. 8:03the time frame and the strategies name
220. 8:05and then we give it the number of
221. 8:06simulations and that's it. Then you just
222. 8:08press start and it will begin. Now these
223. 8:11days if you have watched my previous
224. 8:12videos you know that I'm doing almost
225. 8:15all of my research using Jesse MCP. So I
226. 8:18just ask it and give it a prompt and it
227. 8:20start the research for me and at the end
228. 8:22it just gives me the result inside just
229. 8:24dashboard. So I click on these URLs then
230. 8:26I'm able to just take a look at them and
231. 8:28whenever I'm asking it to write a
232. 8:31strategy I always say that hey like this
233. 8:33is the type of strategy that I want but
234. 8:35before moving on I need you to run a
235. 8:37rule test and then move on with the back
236. 8:40testing and then optimization and then
237. 8:42Monte Carlo simulations. So basically
238. 8:44the rule testing is the first thing that
239. 8:45I need the agent to do for me right
240. 8:47after writing the strategy because if
241. 8:49the rule test is failing there's not
242. 8:51even a point in running a back test for
243. 8:54the strategy in the first place. And
244. 8:55lastly I know that guys these days we
245. 8:58are not used to reading books anymore
246. 9:00especially if it's such a long book such
247. 9:02as this one. But I genuinely enjoyed
248. 9:04reading this book and it changed how I
249. 9:06looked at trading indicators altogether
250. 9:09because you hear all the time that
251. 9:10whether or not a specific trading
252. 9:12indicator works or not. But in fact, all
253. 9:15we need to do actually is to just run
254. 9:17these tests to get an answer ourselves.
255. 9:20So I definitely recommend reading this
256. 9:22book if you haven't already and
257. 9:24definitely begin writing rule tests by
258. 9:26yourself and run them. They are very
259. 9:28easy but very effective. If you enjoyed
260. 9:30the video, please give it a like and
261. 9:32subscribe to the channel if you haven't
262. 9:33already. I plan to create more tutorials
263. 9:35just like this one. Thanks for watching.
264. 9:37I'll see you in the next one.