Using an LLM to Trade Based on Reddit Posts — Transcript

Full transcript

1. 0:00I built the world's stupidest trading
2. 0:01bot that uses Reddit posts to predict
3. 0:03which stocks are about to explode before
4. 0:06they actually do. And it failed
5. 0:07miserably because Oxford beat me to the
6. 0:09punch and has already developed an
7. 0:11algorithm to do exactly this, which is
8. 0:13now publicized. Last time I built a bot
9. 0:16to trade off Wall Street bets and it
10. 0:18lost to the S&P baseline. Then you guys
11. 0:20left 600 comments detailing every
12. 0:22important improvement I should make. And
13. 0:24I went through every single one of them
14. 0:25and have assembled a list that I will
15. 0:27use to create the best Wall Street Bets
16. 0:29trading algorithm implementing your
17. 0:31suggestions to beat these brainiac
18. 0:33Oxford quants with comment section
19. 0:35creativity. So about the Oxford paper,
20. 0:38they didn't actually build a trading
21. 0:40algorithm. Instead, they built three
22. 0:42specific places where a signal might
23. 0:44exist [music] and then three big
24. 0:45warnings about where it doesn't. Today,
25. 0:47the comment section beats Oxford
26. 0:49accuracy with five minutes of coding.
27. 0:51Then, I want to go straight into every
28. 0:53single warning that Oxford specifically
29. 0:55told us to avoid. And by the end, I'm
30. 0:57going to show you two doors that I still
31. 0:58haven't opened yet, and which ones we're
32. 1:00going to be opening next. But before we
33. 1:02beat Oxford, I owe the comment section
34. 1:04one test, [music] the one everybody
35. 1:05asked for, the one that got a,000 up
36. 1:08votes. All right, so quick recap in case
37. 1:10you missed my last video, by the way.
38. 1:12Definitely go check that out. We had
39. 1:13scanned through Wall Street Bets posts
40. 1:15counting the number of mentions for a
41. 1:17particular stock ticker every single
42. 1:19day. We then bought the hype, basically
43. 1:21the most popular stock for a given day,
44. 1:22and then we sold it when it lost its
45. 1:24popularity. And we lost to the S&P every
46. 1:27single time, as expected. We then tried
47. 1:30to buy the fastest risers, maybe stocks
48. 1:32where mentions were spiking suddenly,
49. 1:34and we also lost. So sadly, these two
50. 1:36different strategies did not work. But
51. 1:38you guys pointed out exactly why, and
52. 1:40Oxford actually knew it, too. You can't
53. 1:42just buy the most mentioned stock with
54. 1:44the fastest riser because the sentiment
55. 1:46could actually be negative. Maybe the
56. 1:48company is about to declare bankruptcy
57. 1:50and that's why everybody's talking about
58. 1:51it. So, it's probably not a good time to
59. 1:53buy. Oxford actually went through the
60. 1:55effort of handlabeling 4,000 posts
61. 1:57either bullish, bearish, or neutral.
62. 1:59They then fine-tuned an AI called
63. 2:01Finnbert and hit an accuracy of 69%. And
64. 2:04then they tested investing based on
65. 2:06sentiment for every Wall Street Bets
66. 2:08post and they lost money. Fantastic.
67. 2:11>> [music]
68. 2:11>> So, we do need sentiment. And boy oh
69. 2:13boy, do I have a plan for this. But
70. 2:15before we get to that, I have to address
71. 2:17your most upvoted comment. Did you try
72. 2:20doing the opposite?
73. 2:22>> So, if the crowd is reliably wrong, then
74. 2:24inverting them should be reliable,
75. 2:26right? I mean, come on. It's basically
76. 2:27free money. Let's bet against Wall
77. 2:29Street bets, which means we will short
78. 2:31the most mentioned stocks. [music] So
79. 2:33when the company we bet on crashes, we
80. 2:35see massive returns because surely the
81. 2:38most mentioned stocks are overhyped
82. 2:40right now. At first I chose the GameStop
83. 2:43year running the back test from January
84. 2:45to February of 2021. I know it's an
85. 2:48interesting [music] time period to
86. 2:49choose. Now this quickly resulted in all
87. 2:51of us losing our money together as
88. 2:53GameStop exploded in value and we lost
89. 2:56everything during our short. We could
90. 2:58run a bunch of individual random
91. 3:00portfolios where we short stocks on the
92. 3:02date they are mentioned in the
93. 3:04subreddit. This could generally tell us
94. 3:06if Wall Street Bets is on average
95. 3:08consistently wrong. And maybe we can
96. 3:10make a profit by just betting against
97. 3:12the [music] whole forum in general. And
98. 3:14unfortunately, this too does not work
99. 3:16with our portfolio coming in just below
100. 3:18average market return. But running this
101. 3:21during the GameStop year would be
102. 3:23misleading. It was a massive outlier.
103. 3:26So, let's pull data for the entirety of
104. 3:282025. Now, what you're seeing is our
105. 3:30simulated portfolio in green compared to
106. 3:32the S&P in red. Even by shorting the
107. 3:35most mentioned stock throughout this
108. 3:36entire year, there's only a single
109. 3:38fleeting moment where the strategy was
110. 3:40beating the S&P. And how did we do by
111. 3:42shorting random stocks on the Wall
112. 3:44Street Bets forum? Well, also not great.
113. 3:47These uncapped downsides mean that our
114. 3:49risk is so much higher than simply
115. 3:50buying and holding a stock, which
116. 3:52results in these rather poor returns.
117. 3:54The median return for our shorting
118. 3:56strategy was approximately minus 1.09%
119. 4:00over 2025 to 2026 as compared to the S&P
120. 4:03return of about 18%. So do the opposite
121. 4:07isn't a cheat code. [music] And we just
122. 4:09confirmed Oxford's second warning that
123. 4:11shorting the crowd does not work. But
124. 4:14the comment section had a much better
125. 4:16idea. The idea that 40 of you pointed
126. 4:18out. Okay, quick thing. All of this goes
127. 4:21up on GitHub for my sponsors. every
128. 4:23script, every back test, everything. 600
129. 4:25of you already run it and you can cancel
130. 4:28any time, but every month I merge your
131. 4:30ideas into the code. So, the people who
132. 4:32stay get something that keeps getting
133. 4:33smarter. Plus, you get to support a
134. 4:35struggling PhD grad. Link in the
135. 4:37description. I should maybe stop
136. 4:39publishing this stuff because, as one of
137. 4:41you have already pointed out,
138. 4:43>> don't publish all [music] your secrets
139. 4:45in a YouTube video. Tsunoo.
140. 4:48>> Okay, back to the Oxford thing. Roughly
141. 4:5040 of you said the same thing. Use an
142. 4:52LLM that classifies sentiment for each
143. 4:54post. That way we know if people are
144. 4:56bullish or bearish on a stock. [music]
145. 4:58We buy the ones we like and short the
146. 5:00ones we hate. Okay. Oxford did this too
147. 5:02by labeling 4,000 posts by hand
148. 5:05fine-tuned in a Finnbert model and hit
149. 5:08an accuracy of 69%. [music] So that's
150. 5:10the number that we have to beat. They
151. 5:12did it with two PhDs hand labeling. And
152. 5:14I'm going to use Finnbert for the entire
153. 5:15[music]
154. 5:16thing. Finnbert is just Google's
155. 5:18language model that is specifically
156. 5:19trained on financial sentiment data.
157. 5:21It's able to generate an indicator for
158. 5:23positive or negative sentiment based
159. 5:24[music] on text alone. And yes, I know a
160. 5:27lot of posts are super sarcastic, but
161. 5:29I'm hopeful we can beat this thing
162. 5:30[music] regardless. So, I used Finn to
163. 5:32classify posts from January to February
164. 5:34of 2025 into positive or negative
165. 5:37sentiment. Then, I validated them by
166. 5:40hand to get a sense for its accuracy. I
167. 5:42ended up checking about 200 posts with
168. 5:44confidence levels above 90%. [music] And
169. 5:47yeah, I know that this is like way less
170. 5:48than the 4,000 posts manually validated
171. 5:51by the Oxford level PhDs, but I am lazy
172. 5:54and pucker up for this. In those 200
173. 5:57posts, the prediction accuracy that I
174. 5:59found was 94%. [music]
175. 6:02Which is pretty damn good. Now, full
176. 6:04disclosure, I am cheating a little bit
177. 6:06because the Oxford 69% accuracy was on
178. 6:09three different classes, both bullish,
179. 6:11bearish, and neutral. [music]
180. 6:13And across every post, which is very
181. 6:15difficult, some are sarcastic or not
182. 6:17really obvious. I'm doing two classes
183. 6:19only, both bearish and bullish on the
184. 6:22easiest high confidence posts only,
185. 6:24which I think makes [music] much more
186. 6:25sense for creating a trading algorithm.
187. 6:27The advantage in this case is that our
188. 6:28algorithm only uses data that has high
189. 6:30confidence [music] rather than the
190. 6:32sarcastic posts that are uncertain. So
191. 6:34with better accuracy, can we beat the
192. 6:36Oxford model? That is the question.
193. 6:39Okay, this is a recap on the algorithm
194. 6:40in case you're taking notes. Basically,
195. 6:42every day we're going to scan through
196. 6:44Wall Street Bets posts and label it as
197. 6:46bullish or bearish using our LLM. Then
198. 6:48each stock ticker mentioned, we
199. 6:50calculate a score. I want the score to
200. 6:52make use of positive and negative
201. 6:54sentiment. And I also want it to be dead
202. 6:56simple because I'm not that smart. So
203. 6:58the score is going to be the number of
204. 7:00bullish posts mentioning that stock
205. 7:02minus the number of bearish posts
206. 7:04mentioning that stock. Whichever stock
207. 7:06has the highest score that day is the
208. 7:08one that we're going to buy. We then
209. 7:10hold it until a different stock takes
210. 7:12the top spot. Then we rotate to that new
211. 7:14one instead. I decided not to have any
212. 7:16diversification because come on, live a
213. 7:19little. In my short back test, we see
214. 7:21that buying the stock with the highest
215. 7:22positive sentiment does really well
216. 7:25compared to buying and holding the S&P.
217. 7:27This worked super well, and we even had
218. 7:29a day give plus 25% returns. The total
219. 7:33return over these two months was 51%.
220. 7:36Which amounts to a compound annual
221. 7:38growth rate of over a,000%.
222. 7:42H the spy over the same time period
223. 7:44[music]
224. 7:45did a compound annual growth rate of
225. 7:4710%. So, is this thing actually working?
226. 7:50Am I making a massive mistake by
227. 7:51uploading this video to YouTube? Is this
228. 7:53commenter right? I genuinely don't know.
229. 7:56But if my physics background taught me
230. 7:58anything, it's that a good scientist
231. 8:00always seeks to disprove himself. So,
232. 8:02let's try to investigate this further.
233. 8:04My first thought was that I got lucky. I
234. 8:06mean, we only looked at January to
235. 8:08February of 2025, so maybe let's look at
236. 8:10all of 2025. I then pulled the data from
237. 8:13January of 2025 to January of 2026 and
238. 8:17let Finnbert get to work. And there are
239. 8:19a lot of posts to classify, so this took
240. 8:22a couple of hours. Okay. I mean, this is
241. 8:24what I'd expect. It has no real edge. Uh
242. 8:27oh. Oh gosh. Okay. Okay. This is working
243. 8:31really well. All right. So, I went back
244. 8:33to my code to investigate because if
245. 8:34something looks too good to be true, it
246. 8:36probably is. And I ended up finding
247. 8:39this. This simple less than symbol is
248. 8:42the difference between the code working
249. 8:43or not. It's basically allowing us to
250. 8:46use the current-day sentiment for
251. 8:47tomorrow's prediction. But the way I had
252. 8:49wrote it allowed the program to look at
253. 8:51the community reaction and then buy it
254. 8:54before the news has spread into the
255. 8:56market value. Basically, I had really
256. 8:58bad look ahead bias. The model was
257. 9:00cheating off of tomorrow's knowledge.
258. 9:02So, now that we fixed our look ahead
259. 9:04bias, we can rerun the code and see the
260. 9:06results. Here we're going long on stocks
261. 9:08based on high scores from our sentiment.
262. 9:10We can see that the model unfortunately
263. 9:13does not work. So simply following the
264. 9:15crowd is definitely a bad idea. But
265. 9:17stick around for a bit because we're not
266. 9:19done yet. We still need to short Wall
267. 9:21Street Bets based on sentiment.
268. 9:22Following Wall Street Bets positive
269. 9:24sentiment gave a 1-year return of minus
270. 9:2649%.
271. 9:28Which is uh not good. Maybe you don't
272. 9:31trust Wall Street bets with your next
273. 9:32stock trade. So, what if we bet against
274. 9:35stocks that have highc scoring positive
275. 9:37sentiment? In my opinion, this is what
276. 9:39would give the highest probability of
277. 9:41working because we're purposely betting
278. 9:43against a degenerate crowd of gamblers.
279. 9:45So, we should have a pretty good shot at
280. 9:47winning, right? Also, quick note, the
281. 9:49algorithm flags words like tech, bull,
282. 9:51and meme as tickers because they are
283. 9:54real ETFs. So, when Wall Street Bets
284. 9:56says this is a meme stock, my bot thinks
285. 9:59they're hyping the meme ETF, which is a
286. 10:01cute bug, but it doesn't really change
287. 10:03the results by much. So, here we go.
288. 10:06Finally, let's short Wall Street Bets
289. 10:08with sentiment. Now, we can see that the
290. 10:10volatility is super high because of the
291. 10:12uncapped downsides. And while the market
292. 10:15in general took a downswing in February
293. 10:17of 2025, which helped our short
294. 10:19positions, we quickly lost all of our
295. 10:21gains. and the S&P quickly recovers back
296. 10:24to beating the strategy. So, no, it did
297. 10:27not work. We walked into every warning
298. 10:30Oxford left for us. Don't follow the
299. 10:31hype. We did and lost 49%. Don't trade
300. 10:35on aggregate forum sentiment. We did. No
301. 10:38edge. But here's what we didn't do.
302. 10:40Oxford's paper has a few findings I just
303. 10:42glossed over, and I thought I'd save the
304. 10:44best for last. Oxford built a network
305. 10:46map of which assets get discussed
306. 10:48together and most of the big clusters
307. 10:50lost money. But small niche clusters did
308. 10:53have positive returns. So maybe small
309. 10:56groups of weirdos like me and you
310. 10:58talking about boring stocks might
311. 10:59actually work. Now we have a lot more
312. 11:02work to do and I need your help. So drop
313. 11:04a comment with your ideas and I will
314. 11:06explore them in the next [music] video.
315. 11:08The code for this is on GitHub. Have fun
316. 11:10playing around with it. But if you solve
317. 11:11something, you are obligated to tell me
318. 11:13about it. I'll see you guys in the next
319. 11:15one.