YouTube2Text

Odszedłem z OpenAI i nie żałuję II PSYHO — Transcript

by didaskalia · 15,351 words · 2,363 segments · language en · Watch on YouTube

Full transcript

  1. 0:01It's not that artificial intelligence
  2. 0:03will simply dislike humans, it will
  3. 0:04just be so powerful that it won't
  4. 0:06really pay much attention to us. We are
  5. 0:11stripping ourselves of agency, and we
  6. 0:13do it just to gain some economic profit
  7. 0:15. And you were the only protein-based
  8. 0:19computer, the only Homo Sapiens Sapiens
  9. 0:22who beat a language model.
  10. 0:25>> Yes, that was a year ago. But times
  11. 0:27change, because there was an edition
  12. 0:29this year too.
  13. 0:30>> Who won this year?
  14. 0:31>> Well, not a human. This year, it
  15. 0:34totally crushed the humans.
  16. 0:36>> Was 2025 the last year a human won in
  17. 0:39these competitions?
  18. 0:41>> Yes.
  19. 0:41>> Do you work anywhere today?
  20. 0:43>> No.
  21. 0:43>> How long have you been out of work?
  22. 0:45Since 2000. Well, generally since I
  23. 0:49left OpenAI.
  24. 0:50>> And why did you leave?
  25. 0:52>> That's a lot of strange, little stories
  26. 0:54.
  27. 0:55>> Do you regret leaving a company that is
  28. 0:58so famous today, that is so hot right
  29. 1:00now?
  30. 1:01>> No, that is the one thing I am sure of.
  31. 1:11The program is brought to you by Credit
  32. 1:13Agricole bank with a business account
  33. 1:15for entrepreneurs. The partner of this
  34. 1:20episode is Kodano Optyk, Polish optical
  35. 1:22salons where you can choose glasses
  36. 1:24perfectly suited to your needs and
  37. 1:26check your eyesight.
  38. 1:29>> Good morning. Patrycjusz Wyżga. I
  39. 1:31welcome you and invite you to Didas.
  40. 1:33Today, a meeting with a man who won
  41. 1:34against artificial intelligence. He is
  42. 1:36better, ladies and gentlemen, than
  43. 1:38algorithms. His name is Psajo. Welcome.
  44. 1:40Hi.
  45. 1:41>> Hello.
  46. 1:41>> Actually, that's been a nickname for
  47. 1:43many, many years. You say 30 already.
  48. 1:45>> Well, almost 30. But if you'll allow,
  49. 1:48I'd like us to tell those who for some
  50. 1:50reason don't recognize you and don't
  51. 1:52recognize those five letters, Psajo.
  52. 1:59What does it mean that there sits
  53. 2:01before me a man who won against
  54. 2:02artificial intelligence, proved to be
  55. 2:04better than AI as the only one in the
  56. 2:06world.
  57. 2:06>> There is something like programming
  58. 2:08contests. You can think of it a bit as
  59. 2:11e-sports for programmers. This is, it's
  60. 2:15generally a very niche, very niche
  61. 2:17activity. It was created because
  62. 2:19programming companies were looking for
  63. 2:21new ways to find employees. It also
  64. 2:24stems a bit from programming contests
  65. 2:26at universities, so it has a longer
  66. 2:28history there. And now there are two
  67. 2:33types of programming contests. There
  68. 2:36are contests popularly called
  69. 2:38algorithmic, and those are, for example
  70. 2:40, the Informatics Olympiad, and there
  71. 2:42are programming contests called
  72. 2:44optimization or heuristic. That is the
  73. 2:47category in which I specialize.
  74. 2:49>> And you participate in such contests?
  75. 2:52Yes, well, you could say I participated
  76. 2:54, yes, but the reason why I roll my
  77. 2:56eyes a bit whenever I hear about "the
  78. 2:59man who beat AI" is that...I've been
  79. 3:04participating in these competitions
  80. 3:06practically since they started, so
  81. 3:08that's almost 20 years now.
  82. 3:09>> With what results? Um, in such a way
  83. 3:12that if you were to collect, um, all
  84. 3:18the major competitions that have taken
  85. 3:20place, I've won about half of them.
  86. 3:23There are practically no other people
  87. 3:25who have won more than one. And the
  88. 3:28biggest competition was the Topcoder
  89. 3:30Open. And there were, I don't remember,
  90. 3:3415 or 16 editions, maybe 14. I won
  91. 3:38seven of them. In the others, I was in
  92. 3:40second or third place. Ah.
  93. 3:42>> Listen, so in that case, you are the
  94. 3:44best programmer in the world,
  95. 3:45>> right? Well, I am the best person in
  96. 3:47these specific competitions, yes. I
  97. 3:49wouldn't generalize it. There are so
  98. 3:51many specializations. I happened to
  99. 3:53pick
  100. 3:53>> competitions,
  101. 3:55>> optimization competitions, because in
  102. 3:57the classic ones
  103. 4:00>> let's say I was doing quite okay, yes,
  104. 4:03on Topcoder, which was the biggest
  105. 4:05general ranking at the time. I was even
  106. 4:09fourth at one point out of, I don't
  107. 4:11know, several thousand people. For
  108. 4:14example, our local Tomasz Czajka had, I
  109. 4:17mean, he had a lot of success in that
  110. 4:19specific category. Generally, quite a
  111. 4:23few Poles achieved very good results,
  112. 4:25but in the optimization ones, it was
  113. 4:27just me and a big void after that. And
  114. 4:31there was that one competition where
  115. 4:33the LLM models finally matured enough
  116. 4:35to compete
  117. 4:37>> and you were the only protein-based
  118. 4:40computer, the only Homo Sapiens Sapiens
  119. 4:42, who
  120. 4:44>> beat a language model.
  121. 4:46>> Yes, that was a year ago. Times change,
  122. 4:48because there was an edition this year
  123. 4:50too, I don't know, we can talk about
  124. 4:52that later.
  125. 4:53>> Who won this year?
  126. 4:55>> Well, not a human.
  127. 4:57>> What was the highest place for a human?
  128. 5:00I mean, it was also just a model from
  129. 5:03OpenAI, and I was actually in Japan
  130. 5:05this year too, just so you know, and I
  131. 5:07was doing some coverage. I streamed on
  132. 5:10YouTube for 10 hours. A terrible task.
  133. 5:13So last year you won, this year a model
  134. 5:15won.
  135. 5:15>> Yes. And this year it totally crushed
  136. 5:18the humans, just as one might have
  137. 5:19expected.
  138. 5:20>> But you didn't compete. I didn't
  139. 5:21compete. Yes, I made a tactical dodge,
  140. 5:25>> to remain the champion, I sometimes
  141. 5:27laugh about it that, well, thanks to
  142. 5:29that I remained undefeated, so I can
  143. 5:32laugh or be happy about it. But also I
  144. 5:37decided that it was a bit pointless.
  145. 5:41One thing is that for me, just to get
  146. 5:43into these competitions, you need to
  147. 5:45invest some time, and the
  148. 5:47qualifications for these specific
  149. 5:49competitions that take place in Japan
  150. 5:51last all year. I have to compete from
  151. 5:53time to time. It would probably cost me
  152. 5:55, I don't know, like 200 hours over the
  153. 5:58year. That's 150, 200 hours. Well,
  154. 6:01that's a lot of time. Since I returned
  155. 6:03from Japan a year ago, I’ve suddenly
  156. 6:05had significantly less time.
  157. 6:08>> Why?
  158. 6:09>> Well, because my life has changed.
  159. 6:11Suddenly, tons of people wanted
  160. 6:12something from me, and I was trying to
  161. 6:14navigate this new world—what makes
  162. 6:16sense, what doesn't, what I even want,
  163. 6:18what I’m actually capable of
  164. 6:20achieving. But
  165. 6:22>> you certainly became the most
  166. 6:23talked-about programmer in the world at
  167. 6:25that moment.
  168. 6:27>> No, well, no, I wouldn't go that far.
  169. 6:30Yes, there are many more famous
  170. 6:32programmers in the world, for example
  171. 6:34John Carmack, who created id Software,
  172. 6:37Doom, Quake, and then worked at Meta.
  173. 6:40>> But you beat, you know, the model from
  174. 6:42the loudest, hottest company. Its CEO
  175. 6:45wrote about you.
  176. 6:47>> He did himself. So
  177. 6:50>> please tell me, what was this task, and
  178. 6:52is it even possible to describe to a
  179. 6:54layman like me, so that I could at
  180. 6:56least get close to understanding how
  181. 6:58you beat a language model?
  182. 7:01>> Is it, in general, even describable? It
  183. 7:05would be coolest if we could do a magic
  184. 7:07visualization, I think I could explain
  185. 7:09in two minutes more or less what it's
  186. 7:11about, but just describing it is almost
  187. 7:13impossible. I don’t know if I’ve
  188. 7:18ever had to describe it verbally. But
  189. 7:24>> you just said a moment ago that this
  190. 7:26would be the simplest possible problem
  191. 7:28inspired by authentic reality. Okay.
  192. 7:32The problem we had to solve, right,
  193. 7:36that generally these tasks always have
  194. 7:42a description, and then we are graded
  195. 7:44on tests, there’s a test generator,
  196. 7:46so, like I described the Voyager
  197. 7:48problem, there would be random maps
  198. 7:50generated there. Uh, so that’s one
  199. 7:55important point, but this task was
  200. 7:57about having a map, a two-dimensional
  201. 8:04diagram, let's say a chessboard, I
  202. 8:06don't know, 20 by 20, something like
  203. 8:09that. We have a list of robots on it. A
  204. 8:13robot is just a pawn that is on a
  205. 8:15specific square. And now our task is to
  206. 8:20determine some sort of path for moving
  207. 8:22these robots so that they end up in a
  208. 8:24specific final position. So, it
  209. 8:28generally sounds simple. The additional
  210. 8:31trick is that there are sometimes walls
  211. 8:33on the squares, and you can't just move
  212. 8:35freely from one place to another. Only
  213. 8:39one robot can be on one square at a
  214. 8:41time, but the main part was that we can
  215. 8:43connect robots into a group, and we can
  216. 8:46give commands for the whole group to
  217. 8:52move. And finally, we were evaluated on
  218. 8:56the number of operations we used, that
  219. 8:58is, the number of commands we issued to
  220. 9:01the robots. Therefore, the larger the
  221. 9:05group we put robots into, the more
  222. 9:07efficiently we can generally do it. So,
  223. 9:10I imagine that explained absolutely
  224. 9:13nothing,
  225. 9:13>> right? It gave a little, a little bit
  226. 9:15of direction. And the point was to
  227. 9:17optimize it, to make it simplest,
  228. 9:20>> so that as few, as few commands as
  229. 9:22possible were used for that. So now,
  230. 9:27you can think about these types of
  231. 9:29problems in different ways; you could
  232. 9:31do some, I don't know, complicated
  233. 9:33pathfinding. These problems are also
  234. 9:38extremely complicated in the sense that
  235. 9:40there is no known popular algorithm or
  236. 9:42metaheuristic that we could apply that
  237. 9:44would work; there's just no chance for
  238. 9:46that. So usually, you simplify these
  239. 9:50problems in your head somehow. And here
  240. 9:53I can say what I did and what gave me a
  241. 9:55very big advantage. Again, I suspect
  242. 9:57that will explain very little. Unless
  243. 9:59there were some visualization. Maybe
  244. 10:02you should think about drawing on the
  245. 10:04screen.
  246. 10:07>> So a big problem here was that we
  247. 10:09divide these robots into groups and
  248. 10:14because if we couldn't, if there were
  249. 10:16no possibility of dividing into groups,
  250. 10:18meaning every robot moves on its own,
  251. 10:20there are algorithms that are fairly
  252. 10:22standard and we could apply them, but
  253. 10:24they would give an absolutely tragic
  254. 10:26result here. And now, what I did for
  255. 10:28the first few minutes, being in this
  256. 10:30semi-conscious state, right? Because
  257. 10:32for those who don't know, I was–never
  258. 10:34have I been–as sleep-deprived as I
  259. 10:36was at that competition, and that was
  260. 10:37right at the start. So I was making
  261. 10:45these doodles on a piece of paper and I
  262. 10:47just started to wonder, for certain
  263. 10:49types of tests, what I need to do to
  264. 10:51get an optimal solution, that is, one
  265. 10:53where you simply can't come up with
  266. 10:55anything better. I found many
  267. 10:59situations where, if I combined all the
  268. 11:01robots into one group, I basically have
  269. 11:04an archetype of a solution where it
  270. 11:06seems to me that it's practically
  271. 11:07impossible to do anything better. So
  272. 11:10that, so that was the main observation,
  273. 11:13and again, if it were possible to draw
  274. 11:15it, it would be clearer what I'm even
  275. 11:17talking about. But based on that
  276. 11:20observation, I decided, okay, I'll
  277. 11:22implement this, see if it more or less
  278. 11:25works, if it's consistent with my
  279. 11:26intuition, and I left out a lot of
  280. 11:28things there because you could
  281. 11:30additionally insert walls on that
  282. 11:32chessboard, right? But, well, generally
  283. 11:36I had this idea, I had to implement it,
  284. 11:38check if it works, and what I wasn't
  285. 11:40sure about was, well, okay, I know it
  286. 11:42will work great in these small tests,
  287. 11:44in some specific tests, but I don't
  288. 11:46know if it will work in other tests. If
  289. 11:49it works poorly, should I use a
  290. 11:51completely different method, or patch
  291. 11:53it up, fix it so that,
  292. 11:55>> because you don't know in advance what
  293. 11:57the tests will be like. Do you know
  294. 11:58what the tests will be? Uh, it’s like
  295. 12:00this, I know, but in practice you have
  296. 12:0610 hours for a task, you read the
  297. 12:08content for 15 minutes, spend some time
  298. 12:10wondering what it’s even about. Also,
  299. 12:15there’s the time zone change and a
  300. 12:17ton of other issues, right? But yeah,
  301. 12:21it’s like a, it’s like a giant 10-
  302. 12:23hour sprint. the things that you
  303. 12:27implement there. Those were still the
  304. 12:29times when you didn't use agents, or at
  305. 12:31least those AI agents were absolutely
  306. 12:32hopeless.
  307. 12:33>> That was ages ago, a whole year ago.
  308. 12:35>> Yeah, yeah, yeah. That AI was useless,
  309. 12:38you know. I mean, I say that a bit
  310. 12:40mockingly, and a bit not, because for
  311. 12:42such heavy algorithmic tasks, when
  312. 12:44people asked me afterwards if I used
  313. 12:46agents, it was like, obviously not,
  314. 12:48because I would have just slowed down
  315. 12:50my work. Amazing. So you sit there, you
  316. 12:55investigate, you ponder, you think
  317. 12:57about those tests, you scribble things
  318. 13:00down, think about the solution
  319. 13:02archetype, and start programming, right
  320. 13:04? And you check if the ideas you’ve
  321. 13:06written down actually work out,
  322. 13:08>> right? And in practice, there are these
  323. 13:10problems too, that you make some
  324. 13:11mistakes, bugs, then not everything
  325. 13:13turns out the way you want. When you
  326. 13:15get bad results, you don’t know if
  327. 13:17it’s because my idea was bad, or
  328. 13:19because I implemented it poorly, or
  329. 13:21missed some condition, because, for
  330. 13:22example, I misunderstood the test
  331. 13:24generation. There are a lot of small
  332. 13:27things I have to think about non-stop
  333. 13:29and I keep it all in my head, so with
  334. 13:31every passing hour, there’s just more
  335. 13:33and more chaos that you have to manage.
  336. 13:37So, uh, this, you could say the
  337. 13:39intellectual effort and the
  338. 13:41concentration required for these types
  339. 13:43of problems is really gigantic. And
  340. 13:46it’s a marathon at the same time. I
  341. 13:50was generally doing exceptionally
  342. 13:51poorly, because there were two, two
  343. 13:53significant issues. One is that I
  344. 13:59practically hadn't slept, and I’ve
  345. 14:01been through many 24-hour programming
  346. 14:03contests where I didn't sleep. By the
  347. 14:07way, I competed in many of them with
  348. 14:09several people who now work in high
  349. 14:11positions at OpenAI.
  350. 14:13>> With Jakub Pachocki, for example.
  351. 14:15>> Yes, yes, among others. And I felt, at
  352. 14:19the beginning of these finals in Japan,
  353. 14:22like I used to feel at the end of 24-
  354. 14:24hour contests. I could talk for a long
  355. 14:27time about how badly I sleep and how
  356. 14:29irresponsibly I approach life. That’s
  357. 14:31a separate issue.
  358. 14:32>> Did you win with Pachocki?
  359. 14:34>> No, I was on a team with him a few
  360. 14:36times
  361. 14:37>> and probably with some successes.
  362. 14:40>> Uh yeah, that was it, but those are
  363. 14:42different stories. Alright, anyway,
  364. 14:45there is the issue that I was extremely
  365. 14:47sleep-deprived and simply put, my
  366. 14:49programming was going poorly. And the
  367. 14:50second point is that I actually don't
  368. 14:52really compete in these contests
  369. 14:53anymore. I competed in them a lot,
  370. 14:55let's say, until 2018. It was 2025, so
  371. 15:01for the, let's say, previous three
  372. 15:03years I barely participated in these
  373. 15:06contests, and to get to those 2025
  374. 15:08finals I had to take part in qualifiers
  375. 15:11that lasted through all of 2024. I
  376. 15:14remember that I just randomly made the
  377. 15:16decision, maybe I'll try to compete in
  378. 15:18the contests one more time. Why not?
  379. 15:20Because I'm a little bored. How much
  380. 15:22can one play games? Because that’s
  381. 15:24what you were mainly occupying yourself
  382. 15:26with in life at that time.
  383. 15:27>> Yes. I mean, I would like to portray it
  384. 15:29better, but I definitely spend a lot of
  385. 15:31time playing games. So, you know, you
  386. 15:36could say I came out of that
  387. 15:38programming contest retirement to just,
  388. 15:40well, maybe give it a try. So maybe I
  389. 15:44>> you weren't in form, kind of, you
  390. 15:45weren't, you didn't have
  391. 15:47>> that's exactly what I wanted to say.
  392. 15:48Maybe in 2025, when I wasn't competing
  393. 15:50in these contests anymore, I think that
  394. 15:53by the time of the contest—I probably
  395. 15:55give a different number of hours every
  396. 15:57time, but I'm guessing—that I
  397. 15:59programmed for maybe 50, 100 hours the
  398. 16:01whole year. In...I don't remember if it
  399. 16:04was August or...so yes, I wasn't in
  400. 16:06form, that was the second reason. But
  401. 16:09my idea, the simplification I had, was
  402. 16:11very cool and in a sense, it was
  403. 16:14accurate. Plus, all the, you could say
  404. 16:17research, decisions I made were very
  405. 16:19accurate. Well, clearly, that
  406. 16:24experience remains. It's not like it
  407. 16:28was forgotten, and my intuition about
  408. 16:30the problems also worked. I am also
  409. 16:32quite an unusual person compared to
  410. 16:34others competing in the contests.
  411. 16:36>> Why? Because right now, I don't have a
  412. 16:39perfect view, especially since many of
  413. 16:41the best people are from Japan, there
  414. 16:43is a language and cultural barrier to
  415. 16:45understand. But since I competed in
  416. 16:48these contests for a very long time, I
  417. 16:50met a ton of people who were at the top
  418. 16:52, and not just in programming contests,
  419. 16:54for that matter. I have this, generally
  420. 16:58most people are, I'll put it nicely,
  421. 17:02more introverted than I am. Uh, and
  422. 17:05these are people who have greater
  423. 17:06tendencies toward such an analytical
  424. 17:08approach. I mean, when I see a problem,
  425. 17:12I think about it for a dozen minutes
  426. 17:14max and start coding. Uh, I mean,
  427. 17:16that's how it used to be, right? Most
  428. 17:21people I know can sit for three hours,
  429. 17:23fill 15 pages of notes, think about
  430. 17:25everything, and only when they have
  431. 17:27analyzed it very thoroughly, do they
  432. 17:29then start. So, it’s not that I’m
  433. 17:35incapable of analytical thinking. I
  434. 17:39mean, I have tools in my repertoire
  435. 17:41that I can apply, but I’m the type of
  436. 17:43person who likes to do things, who
  437. 17:45wants quick results, and if something
  438. 17:47is wrong, I want to know as soon as
  439. 17:49possible and iterate fast. I guess
  440. 17:53that’s just a character trait, I
  441. 17:55don't know.
  442. 17:56>> Sure. So, 10 hours pass, and then what?
  443. 17:59Do you submit those programs and they
  444. 18:01get tested, or how does it work?
  445. 18:02>> They are tested on an ongoing basis, so
  446. 18:04we have constant access to leaderboards
  447. 18:09, rankings, and OpenAI was also sending
  448. 18:12in their model submissions in real time
  449. 18:14. Because of that, at the beginning,
  450. 18:20they pulled ahead quickly, as the
  451. 18:22models back then could already write
  452. 18:24code very fast, but their weakness was
  453. 18:26that they weren't good at coming up
  454. 18:28with ideas. They hit a wall pretty
  455. 18:32quickly. So, before the competition, I
  456. 18:36read a few research papers on fairly
  457. 18:41similar systems that solved similar
  458. 18:44problems, so I had some intuition about
  459. 18:46how well OpenAI might do and what I
  460. 18:48could expect.
  461. 18:50>> Because you knew there would be this
  462. 18:51human vs. LLM test, that they would
  463. 18:53enter their model?
  464. 18:54>> I found out about a week or maybe two
  465. 18:56weeks earlier, I think. Well, they
  466. 18:58generally didn't really brag about it
  467. 19:00at all. I assumed they weren't too
  468. 19:02confident
  469. 19:04>> about the results, so as usual, until
  470. 19:07they perform great, well, then
  471. 19:09>> they don't announce it.
  472. 19:10>> As if they had taken last place, it
  473. 19:11would be like they weren't even there.
  474. 19:13>> Right.
  475. 19:14>> So there was no official information
  476. 19:15from them that they were participating.
  477. 19:18There was only information from the
  478. 19:19AtCoder organizers, and since they have
  479. 19:21relatively low reach outside of Japan,
  480. 19:23it didn't really break through. And
  481. 19:28then I think that there were more or
  482. 19:32less four or five hours into the
  483. 19:34contest. I realized that, like, I
  484. 19:38noticed something in my solution, and I
  485. 19:40knew that if I fixed it, I would shoot
  486. 19:42way ahead, because at that point, after
  487. 19:45four or five hours, it was basically
  488. 19:47OpenAI and the rest of the people
  489. 19:49competing in their own separate
  490. 19:50category.
  491. 19:53>> Is it judged by points, by time, by...
  492. 19:56>> By points, so you get a certain number
  493. 19:58of points from each test. And we have
  494. 20:01relative scoring, where the best person
  495. 20:03on a given test gets, let's say, one
  496. 20:05point, and everyone else gets a
  497. 20:07fraction of that.
  498. 20:08>> Sure.
  499. 20:08>> And that's why the results could jump
  500. 20:10around so much. Anyway, I saw that I
  501. 20:12had this perspective that if I
  502. 20:14implemented a certain thing, I would
  503. 20:16move forward significantly, which was
  504. 20:18also important because I was constantly
  505. 20:20struggling with falling asleep, so I
  506. 20:22needed some motivation.
  507. 20:24>> And this psychological aspect is
  508. 20:26actually extremely important during
  509. 20:28competitions. Anyway, I did it with the
  510. 20:32last of my strength, I launched it and
  511. 20:34saw that I actually beat OpenAI. I
  512. 20:38don't remember, it was 6 hours, maybe 7
  513. 20:40hours into the competition, so I was in
  514. 20:43a bit of a shock. I remember what I did
  515. 20:48then, because at that competition we
  516. 20:50uniquely had this strange opportunity
  517. 20:52to go into a little room where we could
  518. 20:54speak to a camera. No, no, I don't know
  519. 20:59if you saw that. No, I don't know if
  520. 21:01many people even knew about that.
  521. 21:03Anyway, I remember that when I saw I
  522. 21:05had overtaken OpenAI, I decided,
  523. 21:07half-conscious, that I would go, though
  524. 21:09I didn't yet know what I would say. I
  525. 21:12went to that room and for two minutes I
  526. 21:14probably babbled some words. I don't
  527. 21:17remember exactly what I said, but I
  528. 21:19remember that at the end I said: "
  529. 21:21OpenAI, your model sucks." OpenAI, your
  530. 21:25model sucks. I know, very ambitious of
  531. 21:28me. Anyway, you could say that in my
  532. 21:32barely conscious mind, I was very proud
  533. 21:35of the words I had spoken. A little
  534. 21:38less so now, but generally, after those
  535. 21:40seven hours, I was practically certain
  536. 21:42that I would win,
  537. 21:43>> because I knew I still had plenty of
  538. 21:45areas where I could improve, and I
  539. 21:47assumed that OpenAI wouldn't improve
  540. 21:48much more.
  541. 21:49>> Okay. Was 2025 the last year a human
  542. 21:52won those competitions?
  543. 21:55>> Yes. This year there was already a big
  544. 21:58difference in OpenAI's favor.
  545. 22:00>> Yes. And OpenAI totally crushed it. So
  546. 22:04I can add an interesting fact. I didn't
  547. 22:06qualify for the 2026 competition
  548. 22:08because I didn't participate in the
  549. 22:10qualifiers at all. I decided it made no
  550. 22:12sense. But I laughed that the
  551. 22:15organizers would probably reach out to
  552. 22:17me at some point, because it would be
  553. 22:18silly, right? considering how widely
  554. 22:23that information spread across the
  555. 22:25world.
  556. 22:26>> And so they wrote to me at the turn of
  557. 22:28January and February asking if I would
  558. 22:30like to participate unofficially. I
  559. 22:32thought about it for a while, but I
  560. 22:34wrote back to them that I wouldn't. Uh,
  561. 22:36I don't really see the point. Looking
  562. 22:41at the narrative globally, aside from
  563. 22:43my own personal issues, I only stand to
  564. 22:45lose, so I don't want to give OpenAI
  565. 22:48such an easy win.
  566. 22:51>> Of course.
  567. 22:52>> And were you sure that you wouldn't be
  568. 22:53able to win?
  569. 22:54>> Yeah, yeah, yeah.
  570. 22:55>> So that would be like setting
  571. 22:56themselves up for failure in a way.
  572. 22:58>> Yes. So why was I sure? Well, first of
  573. 23:03all, OpenAI wouldn't have entered if
  574. 23:05they weren't sure they would win.
  575. 23:07That's the first thing.
  576. 23:08>> Right. Uh, because the narrative in
  577. 23:142025 was kind of that OpenAI got a gold
  578. 23:17medal at the ICPC informatics olympiad,
  579. 23:19which is a collegiate-level team
  580. 23:21programming competition, and they
  581. 23:24solved all the problems, drastically
  582. 23:26crushing all human teams. Well, in my
  583. 23:32competition, they took second place.
  584. 23:35And at the informatics olympiad, I
  585. 23:37don't remember the result, but they got
  586. 23:39sixth or seventh place, and that was
  587. 23:41all within two months. So, considering
  588. 23:44how huge the progress in development
  589. 23:46was, every person, every programmer who
  590. 23:49uses these models assumed that, well,
  591. 23:51surely they'll win now. Sure. And in
  592. 23:542026, they solve a millennium problem,
  593. 23:57so basically
  594. 23:58>> it's possible that was actually a bit
  595. 23:59later, but
  596. 24:00>> sure, sure. So, it's game over. I mean,
  597. 24:03are you saying that it's not just this
  598. 24:05one—you correctly distinguish it, and
  599. 24:07I even understood it—that specific
  600. 24:09programming contest you entered, that
  601. 24:11field, meaning optimization, a human
  602. 24:13will probably never win there as you
  603. 24:15say. In other programming contests as
  604. 24:17well
  605. 24:18>> it would also be hard for
  606. 24:19>> The human era has ended, in a sense.
  607. 24:23Well, yes and no. I mean, because we
  608. 24:25are currently at a moment where human
  609. 24:27plus AI
  610. 24:29>> is much more effective than AI alone,
  611. 24:33>> at least in the programming world. But
  612. 24:37then again, it's probably just like in
  613. 24:39chess, just like in other fields. We
  614. 24:42are always the bottleneck at some point
  615. 24:44.
  616. 24:45>> We humans,
  617. 24:46>> yes, we humans. I mean, I say always,
  618. 24:48but at least in the case of these
  619. 24:50programming things, that's how it will
  620. 24:52be. Actually, it's already a bit like
  621. 24:54that today.
  622. 24:55>> So that bottleneck will be eliminated
  623. 24:56at some point.
  624. 24:58>> I don't think so yet, because we won't
  625. 25:00talk about anything serious yet. For
  626. 25:02now, I'm thinking about optimization,
  627. 25:04fast programming, and effective
  628. 25:05programming. For now, it's LLM and
  629. 25:08human. Soon, it will probably be the
  630. 25:10LLM alone.
  631. 25:10>> Yes. Well, let's say as of today, if I
  632. 25:14were to work on optimization problems,
  633. 25:16what I can contribute is what people
  634. 25:18popularly call "research taste," you
  635. 25:21know, a researcher's taste, right?
  636. 25:24That's a bit of a silly translation,
  637. 25:26but actually, it's a silly term in
  638. 25:27English too. So, you could say it's an
  639. 25:30intuition I've gained over many years,
  640. 25:32and I have a rough idea that these
  641. 25:34types of approaches to the problem are
  642. 25:36more promising. And my role essentially
  643. 25:43boils down to supervision, so I know
  644. 25:45the agent is doing sensible things,
  645. 25:47giving it guidelines, setting
  646. 25:49intermediate goals, and breaking
  647. 25:51problems into smaller ones, because the
  648. 25:53agent can't always handle them on its
  649. 25:55own. But looking at it from the other
  650. 26:01side, it's possible that if you just
  651. 26:03gave it 1000 times more computing power
  652. 26:05, my advantage would diminish
  653. 26:07significantly.
  654. 26:09>> Sure. All right, so you win this famous
  655. 26:12competition and at least for a moment,
  656. 26:14you become the most famous programmer
  657. 26:16in the world.
  658. 26:17>> I'll agree with that.
  659. 26:18>> at that moment. Yes. Sam Altman writes
  660. 26:20on Twitter, and you probably didn't
  661. 26:21just talk to him; did he reach out,
  662. 26:23contact you, call, or write? Besides
  663. 26:24him writing on Twitter: "
  664. 26:26>> No, no, no, that was strange, just so
  665. 26:28very strange. I think he has more
  666. 26:30important things to do. From my
  667. 26:32perspective, by the way, I can explain
  668. 26:35why that probably happened."
  669. 26:37>> Uh-huh.
  670. 26:39>> At least that’s how I perceive it,
  671. 26:41because in general, it was a massive
  672. 26:43success for OpenAI, even though they
  673. 26:45took second place, because people
  674. 26:47didn't really expect that. The people
  675. 26:50in these competitions I enter didn't
  676. 26:52expect it. And we had a poll in our
  677. 26:55micro-community on what people
  678. 26:57predicted, and I don't remember if
  679. 26:58there were any votes that they would be
  680. 27:00able to win, so most people in that
  681. 27:02community were in quite a shock. I was,
  682. 27:06let's say, a bit more optimistic—or
  683. 27:09pessimistic, depending on how you look
  684. 27:11at it—but from my perspective,
  685. 27:17>> it was such an incredibly unlikely
  686. 27:19sequence of all these things that
  687. 27:21happened, because OpenAI took second
  688. 27:23place, right? And it paid off for them
  689. 27:28to promote me a bit to show people that
  690. 27:31, firstly, they didn't lose to a random
  691. 27:33person, but to this guy who had simply
  692. 27:35won a whole host of other competitions.
  693. 27:38>> So, "We lost, but to a really strong
  694. 27:40competitor—in fact, the strongest one
  695. 27:42we could have lost to."
  696. 27:43>> Yes. And the second issue is that I
  697. 27:45used to work there, so it's a double
  698. 27:47victory for them.
  699. 27:48>> You actually worked at OpenAI once,
  700. 27:50>> right? We'll talk more about that later
  701. 27:52. Exactly, but if you have anything
  702. 27:54else to add about OpenAI's reaction to
  703. 27:56your victory.
  704. 28:02>> What pleases me from a purely human
  705. 28:04perspective is that this was the only
  706. 28:05major event among their successes that
  707. 28:08turned into more of a human victory
  708. 28:09than a corporate one, so I'm a bit glad
  709. 28:12they didn't get extra PR points for it.
  710. 28:17I mean, they got a bit of attention,
  711. 28:19but yeah, if someone else had won
  712. 28:21instead of me—I mean, if we had
  713. 28:22swapped me with another person who was
  714. 28:25there—it probably wouldn’t have
  715. 28:26been such a big deal.
  716. 28:28>> Sure. And what happens in your life
  717. 28:30when you win that competition?
  718. 28:31>> Um, I’m still in Japan for three
  719. 28:33weeks and I want to go sightseeing, but
  720. 28:35I can't because I get so many messages
  721. 28:37a day from journalists and
  722. 28:39non-journalists that I don't really
  723. 28:40know what to do and I wonder what all
  724. 28:42this is for. And at the same time,
  725. 28:45it’s 38 degrees in Japan, and I
  726. 28:47normally don't leave the house when
  727. 28:48it’s over 30 because I don't really
  728. 28:51like it that warm, and it was just
  729. 28:52impossible to handle there, plus
  730. 28:54everything is made of concrete. I
  731. 28:56don’t recommend it. I mean, Japan is
  732. 28:58wonderful, but I don’t recommend
  733. 28:59being there in those conditions. Uh,
  734. 29:01I’m sleep-deprived the whole time
  735. 29:03because I’m replying to so many
  736. 29:05people that I only sleep three hours a
  737. 29:07day, so it’s just great. Uh, and
  738. 29:09I’m wondering in general what, why,
  739. 29:12anything, if they’re writing to me
  740. 29:14from TV, asking me to record something
  741. 29:16for them, does that even make sense. So
  742. 29:19, because I’d never had any dealings
  743. 29:21with the media.
  744. 29:22>> And aren’t you getting some amazing
  745. 29:24job offers?
  746. 29:27>> No. And by the way, that was such a
  747. 29:29funny time when Meta was offering those
  748. 29:32giant piles of money for employees. And
  749. 29:36I remember there was a lot of…
  750. 29:39>> A lot of my friends were joking, asking
  751. 29:41me if I’d gotten an offer from Meta
  752. 29:43yet. I didn’t end up getting any in
  753. 29:45the end. Which would actually surprise
  754. 29:48me, honestly. I mean, surely…
  755. 29:50>> Surely there were some from smaller
  756. 29:51companies, but I didn’t get one from
  757. 29:53any big ones. Uh, something worth
  758. 29:56remembering, just a side note in the
  759. 29:58case of Meta. Meta back then wasn’t
  760. 30:00paying for…wasn’t paying for very
  761. 30:02good people. They were paying for
  762. 30:06people working at another lab, right?
  763. 30:09For them to switch over, so they’d
  764. 30:11have up-to-date information on how the
  765. 30:13competition’s models were being
  766. 30:15trained, so that’s where that money
  767. 30:16came from.
  768. 30:17>> Just to know some of the opponent’s
  769. 30:19secrets. Yes, yes, yes. And there are
  770. 30:22no non-competes in California.
  771. 30:23>> Okay. So you can talk about it, you can
  772. 30:25tell people about it.
  773. 30:26>> I mean, sure, I don’t know what the
  774. 30:28exact limits are, but in practice, that
  775. 30:31was one reason why AI development was
  776. 30:33quite even. But anyway, side note.
  777. 30:35>> Exactly. That’s interesting. So what
  778. 30:37now? You come back from Japan, and what
  779. 30:39happens next?
  780. 30:40>> I go through my backlog of messages.
  781. 30:44There were some interviews, but in the
  782. 30:46end, I did almost nothing. I was mostly
  783. 30:48ghosting people. And what was…what
  784. 30:51was happening in practice, I was
  785. 30:53wondering, I don't know, I have a bit
  786. 30:56of a memory gap. I was probably still
  787. 30:58catching up on sleep, I went to a few
  788. 31:04meetings, panels, I appeared somewhere
  789. 31:06a few times, but the first month was
  790. 31:08mostly spent finding my way in a new
  791. 31:10life and wondering, what to do with
  792. 31:15this situation.
  793. 31:16>> Exactly. One thing I knew was that I
  794. 31:20didn't really want to work in any lab
  795. 31:21anymore. I mean, that was a decision I
  796. 31:24had made earlier. I also don't know if
  797. 31:32we want to smoothly transition to the
  798. 31:34risks, because that is one thing I
  799. 31:36considered back in 2020-2021, to focus
  800. 31:38a bit on raising awareness about what
  801. 31:40will happen to the world
  802. 31:44>> under the influence of AI development.
  803. 31:46>> Yes. Yes. Due to the development, which
  804. 31:48is extremely fast and will likely be
  805. 31:50even faster.
  806. 31:51>> Are you working anywhere today?
  807. 31:52>> No.
  808. 31:53>> How long have you not been working?
  809. 31:562020. Well, basically since I left
  810. 31:59OpenAI.
  811. 31:59>> And how long did you work there?
  812. 32:02>> It’s hard to answer that question
  813. 32:05because originally, when I joined in
  814. 32:072017, I worked as a contractor. I went
  815. 32:10there for two months, came back to
  816. 32:12Poland, and theoretically pretended I
  817. 32:14was working. I mean, I wrote to them
  818. 32:16that I was doing something, but in
  819. 32:18reality, I wasn't doing anything. But
  820. 32:20they know, so, so I was a bit there, a
  821. 32:22bit in the States, and then in Poland.
  822. 32:28And here's a funny thing, another
  823. 32:30digression, OpenAI considered hiring me
  824. 32:32in 2016 shortly after they opened, but
  825. 32:35I turned down their offer because when
  826. 32:37I talked to the people, they seemed
  827. 32:39unpleasant to me.
  828. 32:42>> Unpleasant people from Open in 16. 16
  829. 32:48was the year I joined the project—I'm
  830. 32:50speaking terribly chaotically—but I
  831. 32:52joined the project
  832. 32:53>> 2017
  833. 32:56>> when OpenAI was working on a Dota bot.
  834. 32:59Dota is a big esports game. At the time
  835. 33:02, I don't know how it is now, but at
  836. 33:03the time it had the largest prize pools
  837. 33:05in the world. I think The International
  838. 33:08, I don't remember, I don't know, 40
  839. 33:10million dollars, maybe 80 million. Some
  840. 33:13kind of absurd money. Yeah, that kind
  841. 33:15of money where a father can just tell
  842. 33:17his son: "Why are you kicking a ball?
  843. 33:19Go play games." Yes.
  844. 33:23>> Why, why be the next Lewandowski?
  845. 33:25Better to become a pro gamer. You’ll
  846. 33:27have much more money there. Well, maybe
  847. 33:30so. 2017 project. And how long were you
  848. 33:33at OpenAI? Until when? Remind me.
  849. 33:36>> I was just a bit on and off, at the
  850. 33:39beginning of 2019 I joined full-time. I
  851. 33:44moved to San Francisco and lived there
  852. 33:46for a bit over a year, and I finished
  853. 33:48working there when the COVID lockdowns
  854. 33:50started. I literally flew back on the
  855. 33:54last flight from San Francisco to
  856. 33:55Poland. That’s also a funny story.
  857. 33:57>> So, 2020.
  858. 33:58>> Yeah, yeah, yeah. Uh, that was back
  859. 34:01when most of the world hadn't heard of
  860. 34:02what OpenAI was. That was still before,
  861. 34:05uh, the publication of, uh, ChatGPT 3.5
  862. 34:07, I think that was three years ago,
  863. 34:09right? November 2022, if I recall
  864. 34:11correctly.
  865. 34:11>> Yeah, yeah, yeah. I admit that I always
  866. 34:14mix up dates, but yeah, in the
  867. 34:15beginning, some people had heard of
  868. 34:17OpenAI because Elon Musk was involved,
  869. 34:19so it was always the "Elon Musk Company
  870. 34:21," which wasn't true, but
  871. 34:22>> besides that, it wasn't a company at
  872. 34:24the start,
  873. 34:25>> it wasn't a company. Yes,
  874. 34:26>> exactly.
  875. 34:28>> It was meant to be open. What? An
  876. 34:30association, a foundation, which...
  877. 34:33>> Well, it was a non-profit; I can't tell
  878. 34:35you the exact legal structure, because
  879. 34:37I don't know much about those things,
  880. 34:39but yeah, it was a place that brought
  881. 34:41together researchers who were just
  882. 34:43experimenting. Originally, OpenAI was
  883. 34:49founded as, if I remember correctly—
  884. 34:52and this is probably fairly well
  885. 34:54documented—as a counterbalance to
  886. 34:56DeepMind at the time, since, well,
  887. 34:58DeepMind had been bought by Google. I
  888. 35:03don't remember what year, and at the
  889. 35:07time, DeepMind was the only company,
  890. 35:10the only lab in the world, aside from
  891. 35:12some academic stuff, that was dealing
  892. 35:14with, let's say, more advanced
  893. 35:16applications of artificial intelligence
  894. 35:18. And so, OpenAI was meant to be a
  895. 35:23counterbalance. Yes. Well, no one wants
  896. 35:25, like, even back then, people were
  897. 35:27wondering what could be squeezed out of
  898. 35:30this artificial intelligence, right?
  899. 35:34Deep learning, or, you know, everything
  900. 35:37related to neural networks, exploded
  901. 35:39after 2012, and that was a breakthrough
  902. 35:41; we already had very good models for
  903. 35:43image recognition, so there was this
  904. 35:45sort of shock because suddenly we had
  905. 35:48new classes of problems we could solve.
  906. 35:52Not long after, there was also AlphaGo.
  907. 35:56If I remember right, the first AlphaGo
  908. 35:58was in 2015 before it was against Lee
  909. 36:00Sedol, so OpenAI must have been founded
  910. 36:03after that, because it was late 2015.
  911. 36:05>> And what exactly did you do at OpenAI?
  912. 36:07>> So, I worked on the Dota project. I
  913. 36:10literally came there for that. I was
  914. 36:12recruited by Jakub Pachocki, by the way
  915. 36:14. Well, because generally, I've known
  916. 36:17Jakub for a very long time. Back from
  917. 36:22programming competitions, and he wrote
  918. 36:25to me saying, "Hey, we have this funny
  919. 36:27project, we're going to play a computer
  920. 36:29game," meaning we were going to try to,
  921. 36:32well, play Dota. I was playing a bit of
  922. 36:34Dota at the time too, so, let's say, I
  923. 36:36had that domain knowledge, let's call
  924. 36:39it, and I figured that it actually
  925. 36:44sounded cool, because it combined my
  926. 36:46various interests, you know? And
  927. 36:51>> well, Kuba is a great guy, so it was
  928. 36:53super cool to work with a friend.
  929. 36:55>> Kuba is cool. I can go there later,
  930. 36:58after I didn't take that offer in 2016,
  931. 37:00there are some more details there, but
  932. 37:03I finally wrote back to them that, well
  933. 37:05, I don't really want to, maybe
  934. 37:07remotely. So when I wrote that I wanted
  935. 37:10to work remotely, they weren't really
  936. 37:12interested. So I felt a bit of regret
  937. 37:14after that, and after that exchange a
  938. 37:16year earlier I decided: "Well, okay,
  939. 37:18maybe it's not that stupid, they had
  940. 37:21already grown, they had about 40 people
  941. 37:23back then, something like that, so I
  942. 37:25came for interviews, and they were
  943. 37:29already at a quite advanced stage of
  944. 37:31that project. And that was two months
  945. 37:35out. I mean, when I arrived at the
  946. 37:37beginning, it was a bit earlier because
  947. 37:39I had something, but I also have a lot
  948. 37:41of strange, funny stories about
  949. 37:42recruitment interviews. I had one
  950. 37:44interview where I got a question and
  951. 37:46spent an hour explaining to the guy
  952. 37:48that it was a very bad question and
  953. 37:50that it was a very poor way to select
  954. 37:52candidates. But that's just to show how
  955. 37:54easy I am to work with. Anyway, I had
  956. 37:58something, but the interviews, I talked
  957. 38:00with everyone, I could meet the team,
  958. 38:02see who works there, what the office
  959. 38:05looks like, what the atmosphere is, and
  960. 38:07I decided that, actually, it would be
  961. 38:09cool to do something new in life. I had
  962. 38:12never worked for anyone anywhere before
  963. 38:14. So maybe this is the moment, I
  964. 38:17wouldn't call it time to grow up, but I
  965. 38:19just wanted to experience something new
  966. 38:21. And so I made that decision. I don't
  967. 38:24remember which month it was, but I
  968. 38:26remember that finally when I arrived
  969. 38:28there, it was two months before The
  970. 38:30International, which are the Dota world
  971. 38:32championships, that's what they're
  972. 38:33called, and it's a huge event in a
  973. 38:35sports arena for 40,000 people that
  974. 38:37lasts a week. And we were appearing
  975. 38:41there on stage, only back then we
  976. 38:43weren't doing full Dota yet, because
  977. 38:45it's an extremely complicated game, but
  978. 38:47a show match, which was a show match.
  979. 38:57One versus one, so a very simplified
  980. 38:59form of Dota, you could say, but we
  981. 39:01managed to win that. It was an
  982. 39:05eight-person team back then. Eh, quite
  983. 39:08overwhelmingly. So that was actually
  984. 39:11the first thing that made OpenAI become
  985. 39:13known. It was, like, the first major
  986. 39:17piece of information that went out that
  987. 39:19wasn't about Elon Musk founding OpenAI.
  988. 39:22>> Mhm. So, in terms of media, it was a
  989. 39:27very big success, because, well, the
  990. 39:29world championships, I don't know how,
  991. 39:31but I assume a few million people watch
  992. 39:34it. It's possible I just seriously
  993. 39:36overestimated that. Uh, and so then I
  994. 39:38stayed with the team, although again I
  995. 39:40worked somewhat remotely, somewhat
  996. 39:42on-site, I would travel there from time
  997. 39:44to time, and then we were already
  998. 39:46making a fuller version of the game. It
  999. 39:50wasn't 100%, but it was a version of
  1000. 39:52the game that I thought was impossible.
  1001. 39:57Or let me put it differently. As a
  1002. 39:59matter of fact, during my sort-of
  1003. 40:01recruitment when I spoke with the
  1004. 40:03project lead, who was Greg, Greg
  1005. 40:04Brockman—he is one of the more
  1006. 40:06important people at OpenAI, previously
  1007. 40:08you could say he was the most important
  1008. 40:11, because he was de facto running the
  1009. 40:13company, though not alone. I told Greg
  1010. 40:19something like, it seems to me that
  1011. 40:28writing a Dota bot for this game is
  1012. 40:30practically impossible, because if it
  1013. 40:32were possible with current technology,
  1014. 40:34we would probably have tools and models
  1015. 40:36capable of solving a ton of other
  1016. 40:38problems. Simply put, from my
  1017. 40:42perspective, the degree of complexity
  1018. 40:44of the decisions that need to be made
  1019. 40:46in this game was so incredibly far from
  1020. 40:48everything we had done in machine
  1021. 40:50learning so far that, well, something
  1022. 40:52didn't add up for me. On the other hand
  1023. 40:58, for me, well, I’ve always been
  1024. 41:00someone who likes these impossible
  1025. 41:02problems because they are motivating,
  1026. 41:04so if, like, I don’t get any pleasure
  1027. 41:06from doing something where I know
  1028. 41:08exactly how to do it—so that was,
  1029. 41:10well, I really got tangled up there.
  1030. 41:15Anyway, I stayed on and worked
  1031. 41:17practically until 2020.
  1032. 41:19>> Uh, but that’s a year,
  1033. 41:22>> uh,
  1034. 41:23>> plus a bit.
  1035. 41:25>> Right, so I joined in mid-2017, I was a
  1036. 41:29contractor and until
  1037. 41:33>> and in 2019 I switched from a
  1038. 41:34contractor to a full-time position,
  1039. 41:37where then I
  1040. 41:38>> was based in San Francisco.
  1041. 41:40>> Yeah. And why did you leave? There are
  1042. 41:43a lot of strange, small stories.
  1043. 41:49Meaning there were a lot of personal
  1044. 41:51conflicts. As I’ve hinted a few times
  1045. 41:55, I am a rather specific person to work
  1046. 41:57with. There was a big, from my
  1047. 42:02perspective, cultural conflict. For
  1048. 42:05those who don’t know, our work
  1049. 42:10culture is much more direct, where
  1050. 42:16where we can reasonably tell each other
  1051. 42:18that something is wrong, that someone
  1052. 42:20did something poorly. In the States,
  1053. 42:22especially in San Francisco in Silicon
  1054. 42:25Valley, the work culture is such that
  1055. 42:27you don't really give that feedback
  1056. 42:31directly to your colleagues. Very often
  1057. 42:34, everything goes through HR. We also
  1058. 42:37worked in such, well, extremely
  1059. 42:39stressful conditions. Towards the end,
  1060. 42:42we had such a grind that I remember I
  1061. 42:44was sleeping three hours a night for a
  1062. 42:47month, normally I sleep 7, because I
  1063. 42:49was working 16 hours a day. On what?
  1064. 42:52>> On Dota, right before we—we couldn't
  1065. 42:54push back deadlines because the world
  1066. 42:56championships have a set date, so we
  1067. 42:58had to be ready exactly at that moment.
  1068. 43:01We had to.
  1069. 43:03>> So, these were quite stressful
  1070. 43:04conditions. It also turned out that
  1071. 43:07there were a surprisingly high number
  1072. 43:09of toxic people, and I wasn't one of
  1073. 43:11them, at least that's my opinion of
  1074. 43:13myself. And I tried to resolve a lot of
  1075. 43:16different conflicts. Uh, in the end, I
  1076. 43:18took a massive hit. In the meantime,
  1077. 43:20OpenAI was growing extremely fast, and
  1078. 43:23companies—when a company is small, at
  1079. 43:25the start when there were 40 people, it
  1080. 43:28was de facto a startup. It was pleasant
  1081. 43:31; everyone knew everyone. We had common
  1082. 43:34goals. Once a company grows, middle
  1083. 43:37management comes in, which often has
  1084. 43:39different goals, because a bit of
  1085. 43:41corporate politics starts, where
  1086. 43:43everyone is optimizing to get ahead as
  1087. 43:45fast as possible. This causes the
  1088. 43:48company to have many people who no
  1089. 43:53longer want, let's say, the general
  1090. 43:55good, but begin to—well—
  1091. 43:58>> look after their own private interests.
  1092. 44:03So, what ultimately happened is that
  1093. 44:05because of a lot of internal politics
  1094. 44:07—which, by the way, didn't just give
  1095. 44:09me trouble—I lost absolutely all
  1096. 44:15motivation to work. Yes, I also lose
  1097. 44:19motivation relatively easily, and it
  1098. 44:21ended up with me practically not
  1099. 44:23working for the last few months, while
  1100. 44:25at the same time having quite good
  1101. 44:27relations with the leadership. So I had
  1102. 44:31a somewhat paradoxical situation, where
  1103. 44:33many people who were high up liked me
  1104. 44:35and, as it were, worked with me, so
  1105. 44:38they considered me very valuable. Plus,
  1106. 44:43well, I have a somewhat unconventional
  1107. 44:45background, because most of the people
  1108. 44:51working at OpenAI at the time were
  1109. 44:53academic researchers, and I was that
  1110. 44:55person who, in fact, doesn't even have
  1111. 44:57a higher education. And has been an
  1112. 45:01autodidact their whole life. On top of
  1113. 45:05that, I am a generalist who has dealt
  1114. 45:07with a mass of various random things.
  1115. 45:12Also, having experience in game design
  1116. 45:14turned out to be very valuable. While
  1117. 45:18working on Dota, I was the only person
  1118. 45:20who practically didn't program, because
  1119. 45:22I was evaluating how good our bot was
  1120. 45:24and what needed to be changed. For a
  1121. 45:26moment, I even managed the project a
  1122. 45:28little bit, which was absurd, since it
  1123. 45:30was my first job. But luckily, it
  1124. 45:32lasted a short time. When we had a
  1125. 45:37moment where anyone from the internet
  1126. 45:39could come and play with our bot, I
  1127. 45:40managed the entire community. In the
  1128. 45:46sense that we started a Discord, we had
  1129. 45:48over ten thousand people on it, and I
  1130. 45:50tried to manage all that chaos of crazy
  1131. 45:53, emotional 14-year-olds who like—
  1132. 45:55well, obviously the age range is much
  1133. 45:57broader—who are very active on our
  1134. 45:59Discord and say a lot of strange things
  1135. 46:01. So, I was usually doing the thing
  1136. 46:05that no one else wanted to do, because
  1137. 46:07I enjoyed it.
  1138. 46:08>> Sure. Do you regret, a little bit,
  1139. 46:11leaving a company that is so loud today
  1140. 46:14, that is so very hot?
  1141. 46:16>> No, that is one thing I am sure of. I
  1142. 46:20mean, first of all, I predicted that
  1143. 46:22this would happen.
  1144. 46:24>> Yes.
  1145. 46:24>> Yes. Uh. more or less. I mean,
  1146. 46:33obviously I wasn't certain, right, no
  1147. 46:35one ever is, but I did have my
  1148. 46:37predictions about how quickly
  1149. 46:39artificial intelligence would develop.
  1150. 46:44I also had my own predictions about
  1151. 46:46when other companies would suddenly
  1152. 46:48wake up to realize that this
  1153. 46:49development is actually incredibly fast
  1154. 46:52and how many experienced people there
  1155. 46:54are to quickly catch up. And well, I
  1156. 47:01was fairly convinced, like, what I was
  1157. 47:03certain of was that this development
  1158. 47:05would be, well, so incredibly fast. Of
  1159. 47:08course, I had extra information,
  1160. 47:09because being inside the company people
  1161. 47:14were wor—if I remember correctly, GPT
  1162. 47:17-2 was released in 2019, I think on
  1163. 47:19Valentine's Day, and it was the first
  1164. 47:23language model that, from my
  1165. 47:24perspective, was capable of generating
  1166. 47:27such coherent, meaningful text. And it
  1167. 47:32kind of blew my mind when I saw it,
  1168. 47:34because from my perspective, it was
  1169. 47:36that problem where we had no idea how
  1170. 47:38to do anything, that it was so
  1171. 47:40unattainable, and yet, suddenly it
  1172. 47:42turns out that at a relatively low cost
  1173. 47:44, we are able to create something
  1174. 47:46sensible. And of course, after GPT-2
  1175. 47:51was created and it was, well, also a
  1176. 47:54pretty decent marketing success, then
  1177. 47:56suddenly—well, not suddenly, but many
  1178. 47:59more people inside the company, which
  1179. 48:01was still a tiny research firm, were
  1180. 48:03curious, they were curious how much
  1181. 48:06could be squeezed out of it. So if I
  1182. 48:09remember correctly, the premiere of GPT
  1183. 48:12-3 was only somewhere around mid-2020,
  1184. 48:16but in the meantime, those people were
  1185. 48:19constantly developing this project, all
  1186. 48:21the time, so I saw the progress inside
  1187. 48:23the company firsthand, so you could say
  1188. 48:25I had a few points on that axis. First,
  1189. 48:30I saw how the model's power changes
  1190. 48:32with scale, meaning with the increase
  1191. 48:34in the number of parameters, computing
  1192. 48:37power, and data improvement. And, well,
  1193. 48:44the shortcomings were clearly visible,
  1194. 48:47among others, the fact that those
  1195. 48:48models at the time were, let's say,
  1196. 48:50decent at storing knowledge, but they
  1197. 48:53didn't want to cooperate at all and
  1198. 48:55absolutely couldn't handle any tasks
  1199. 48:57that required some thinking. So all
  1200. 49:02those things that are unfortunately
  1201. 49:04sometimes repeated, that these are
  1202. 49:06stochastic parrots and just blindly
  1203. 49:08repeat what they had in their training
  1204. 49:10data, that was still true back then.
  1205. 49:13But, well, so I just, you could say,
  1206. 49:15projected the trajectory forward.
  1207. 49:17>> in 20 17 when you started, 40 people.
  1208. 49:23How many Poles were in the company then
  1209. 49:25? There was you, there was Jakub
  1210. 49:27Pachocki,
  1211. 49:27>> there was Szymon Sidor, there was
  1212. 49:32Rafał Józefowicz, and Marcin
  1213. 49:34Andrychowicz. They actually joined very
  1214. 49:38early, they were some of the first
  1215. 49:39people when there were still 15, and I
  1216. 49:41think there were a few more Poles. I
  1217. 49:43don't know, I don't want to list every
  1218. 49:45one.
  1219. 49:45>> That's still a high percentage, right?
  1220. 49:47>> Yes, yes.
  1221. 49:48>> Out of 40 people.
  1222. 49:49>> Uh, well, seven people, I think. Sure.
  1223. 49:52Alright, so you leave OpenAI and you
  1224. 49:55say:" That was your first job in 2017
  1225. 49:57as a contractor until 2020, and then? "
  1226. 50:05The main thing was that it was a very
  1227. 50:07strange moment because, as I mentioned
  1228. 50:09earlier, I returned on literally the
  1229. 50:11last plane, so at the same time I quit
  1230. 50:14my job at OpenAI, returned to Poland,
  1231. 50:16and the lockdowns began. And I have
  1232. 50:21never in my life been as confused as I
  1233. 50:23was then, because I had absolutely no
  1234. 50:25idea what had happened, having
  1235. 50:27completely changed hemispheres. So I
  1236. 50:31think for the first month I was
  1237. 50:32somewhat happy that I could lock myself
  1238. 50:34in the house and didn't have to do
  1239. 50:36absolutely anything, but that passed
  1240. 50:38quickly. I remember spending a lot of
  1241. 50:43time wondering, okay, but what do I
  1242. 50:45want to do with my life now, and I
  1243. 50:50don't remember what conclusions I
  1244. 50:52reached back then. I felt a bit sad
  1245. 50:54that I had left, well, because I had to
  1246. 50:56come up with a new plan for my life. My
  1247. 50:59idea was that I wanted to return to
  1248. 51:01making games. That had always been my
  1249. 51:03dream. It was a place where I felt,
  1250. 51:06let's say, professionally fulfilled,
  1251. 51:08but also artistically. It was hard for
  1252. 51:14me to get motivated. It also ultimately
  1253. 51:17turned out that the fact that the world
  1254. 51:21wasn't functioning normally left a big
  1255. 51:23mark on me, because well, in order to
  1256. 51:28function, I need people. It turned out
  1257. 51:32that a large part of me was traveling,
  1258. 51:34going to different events around the
  1259. 51:36world, meeting people, and suddenly it
  1260. 51:39was all taken away, so, well, I was
  1261. 51:41terribly demotivated then. Uh, and one
  1262. 51:46of the things I was thinking about a
  1263. 51:48lot—so besides that, there was
  1264. 51:50another issue, which was that since I
  1265. 51:52was in that lab, I had an awareness of
  1266. 51:54how this artificial intelligence would
  1267. 51:56develop, so I thought quite hard about
  1268. 52:03perhaps dealing a bit with the topic of
  1269. 52:08raising awareness about the risks. And
  1270. 52:11I was in a fairly incredible situation,
  1271. 52:17that I was one—well, incredible—I
  1272. 52:20was one of the few people in the world
  1273. 52:22who could speak about it in a fairly
  1274. 52:24competent way, because in the world, to
  1275. 52:26be able to speak sensibly about these
  1276. 52:28trajectories, you had to be in a lab.
  1277. 52:32There were actually two of those labs.
  1278. 52:35There was OpenAI and there was Google.
  1279. 52:39And now, there weren't people who were
  1280. 52:41leaving those places and saying," Okay,
  1281. 52:43I don't feel like working anymore, I
  1282. 52:45don't care about anything. "And at the
  1283. 52:48same time, if I had decided to, say,
  1284. 52:50start publicly complaining about
  1285. 52:52institutions like OpenAI, I would have
  1286. 52:54made it very difficult for myself to
  1287. 52:57find a job. So there simply weren't any
  1288. 53:01people who were put in this situation.
  1289. 53:03Of course, in the meantime, it turned
  1290. 53:05out that I don't like writing. Like,
  1291. 53:09that was one big problem, and the
  1292. 53:11second is that, well, purely
  1293. 53:13practically, if I write something like
  1294. 53:15that, I would want someone to read me,
  1295. 53:17so unfortunately I would have to put a
  1296. 53:19lot of work into marketing myself.
  1297. 53:27Nobody likes that, especially me, so
  1298. 53:29you could just say I gave up, because
  1299. 53:31it was too much effort and so I chose
  1300. 53:33comfort.
  1301. 53:37>> And then you didn't work anymore.
  1302. 53:39>> Well.
  1303. 53:40>> So you only worked there for those few
  1304. 53:43years, 2017.
  1305. 53:44>> Yes. I mean, just so it's clear, I was
  1306. 53:46doing some things here and there. Yes.
  1307. 53:48It's not like I'm just sitting on my
  1308. 53:50butt on the couch watching series and
  1309. 53:51soap operas. I don't know why I said
  1310. 53:56that. Ah.
  1311. 53:57>> Probably your guilty pleasure. You
  1312. 53:59don't even know about it.
  1313. 54:02>> Let's not talk about those things.
  1314. 54:03>> Well, okay, listen, but this is quite
  1315. 54:05amazing, because you're talking about
  1316. 54:09awareness of the threats associated
  1317. 54:11with artificial intelligence in 2020.
  1318. 54:13>> Yes.
  1319. 54:14>> A very long time ago from the
  1320. 54:15perspective of what's happening now.
  1321. 54:17Yes.
  1322. 54:18>> What kind of threats did you see back
  1323. 54:20then? So, you know, back then I'm
  1324. 54:27trying to remember, because I have
  1325. 54:30>> But you already saw the threat back
  1326. 54:32then? Yes.
  1327. 54:33>> Yes. I even had written down what I
  1328. 54:35wanted to write about. You know, it's
  1329. 54:36not like I was speculating that much,
  1330. 54:39there was just this moment in January
  1331. 54:41or February 2021 where I decided, okay,
  1332. 54:44I just have an idea more or less of
  1333. 54:46what I want to do. I wrote down some
  1334. 54:50specific things I still needed to read
  1335. 54:52about to speak competently, and a list
  1336. 54:54of topics. Uh.
  1337. 54:55>> But everything revolved around the
  1338. 54:57threats, right?
  1339. 54:57>> Uh, I mean, you know, back then you had
  1340. 55:00to talk about the basics first; there's
  1341. 55:02no point in talking about threats,
  1342. 55:04>> right? If people don't understand that,
  1343. 55:06>> if they don't see
  1344. 55:07>> the source,
  1345. 55:08>> if they don't see how fast artificial
  1346. 55:10intelligence can develop. And you
  1347. 55:13already knew at that time what that
  1348. 55:14pace would be.
  1349. 55:15>> I mean, I knew, I had a strong hunch.
  1350. 55:18Like, maybe we could, I could probably
  1351. 55:20draw you, let's say, this probability
  1352. 55:23distribution in that case.
  1353. 55:24>> But you also knew in the sense that you
  1354. 55:26saw, because that was already your
  1355. 55:28history, how intensively it had been
  1356. 55:29developing up to that point.
  1357. 55:31>> Yes, so I had, let's say, an insight
  1358. 55:33into at least the next few months of
  1359. 55:35what would happen, but specifically in
  1360. 55:372020. Uh, I was aware of—and this is
  1361. 55:40probably quite important—what people
  1362. 55:42in those labs were saying, what hunches
  1363. 55:45they had, and also what their approach
  1364. 55:47to these problems was, what their
  1365. 55:49priority was. And in the end, it seems
  1366. 55:52to me that this is quite important
  1367. 55:54knowledge, which makes it easier for me
  1368. 55:56to understand the world today.
  1369. 55:58>> And was that knowledge terrifying?
  1370. 56:00>> Not in the sense that it makes things
  1371. 56:02easier. I wouldn't call it terrifying
  1372. 56:06knowledge. Besides, I wonder if I would
  1373. 56:09call any knowledge terrifying.
  1374. 56:14>> But were you worried about the goals of
  1375. 56:15the people who worked at the OpenAI lab
  1376. 56:17?
  1377. 56:18>> Well, you know, from that perspective,
  1378. 56:21I actually find it optimistic. In the
  1379. 56:23sense that these are people who uh I'm
  1380. 56:28wondering how to put it. Maybe I
  1381. 56:32wouldn't judge these people in terms of
  1382. 56:34good or bad, but it was valuable for me
  1383. 56:38to understand the motivation behind why
  1384. 56:40people want to work in these fields.
  1385. 56:42>> What do you see as the biggest threats
  1386. 56:45in 2026? We are taking away our own
  1387. 56:49agency, and we do it to, well, have
  1388. 56:51some economic gain from it. And yes, as
  1389. 56:56many people often say right now, well,
  1390. 56:59people aren't stupid enough to do that,
  1391. 57:01but it is, well, already happening. And
  1392. 57:06if we have two companies, if we assume,
  1393. 57:08yes, because that is the crucial issue,
  1394. 57:10if we assume that this artificial
  1395. 57:12superintelligence will arise, well, we
  1396. 57:15have two companies. One simply
  1397. 57:17delegated all intellectual work, while
  1398. 57:19in the other, a human is still trying
  1399. 57:21to control what is happening. Then at
  1400. 57:23some point, the one that delegated all
  1401. 57:25decisions to the superintelligence will
  1402. 57:27simply be more competitive on the
  1403. 57:29market, more efficient, it will, I
  1404. 57:31don't know, be able to sell products
  1405. 57:33cheaper, so it will push the other out.
  1406. 57:36Companies that are more automated
  1407. 57:38naturally displace others, and the same
  1408. 57:40could potentially happen with
  1409. 57:42governments, yes, if we don't design
  1410. 57:44this very well. And this whole scenario
  1411. 57:48is popularly called" gradual
  1412. 57:50disempowerment, "and the algorithm will
  1413. 57:52take over power,
  1414. 57:55>> so we will actually give it up
  1415. 57:57voluntarily, systematically delegating
  1416. 57:59more and more gradually. And at some
  1417. 58:01point, we won't even realize when it
  1418. 58:03happened. And for now, I prefer to talk
  1419. 58:06about this than about robots wiping out
  1420. 58:09humanity, because, firstly, this
  1421. 58:11scenario doesn't define that, because
  1422. 58:13simply something will happen, and
  1423. 58:15unfortunately, what will happen will
  1424. 58:17not be up to us. Maybe it will be the
  1425. 58:21example where we live in a utopia and
  1426. 58:23we simply managed to align these models
  1427. 58:25, where these models do their best so
  1428. 58:27that people live as well as possible.
  1429. 58:31Or maybe at some point it will turn out
  1430. 58:33that we didn't do it too well, and at
  1431. 58:35some point they will decide, like in
  1432. 58:37that famous paperclip maximizer
  1433. 58:38experiment, right? They will turn us
  1434. 58:41into something, right? And it's not
  1435. 58:44like artificial intelligence will
  1436. 58:45simply dislike humans, but it will be
  1437. 58:47powerful enough that it won't even pay
  1438. 58:49much attention to us. And So that is
  1439. 58:55this one group. The second group, I am
  1440. 58:57dragging this out so much. The second
  1441. 59:00group is what is called misuse, which
  1442. 59:02is the use of models for bad purposes.
  1443. 59:06So, at this point, you can think of
  1444. 59:08models a bit more as tools. Well, but
  1445. 59:11there are people, not everyone in the
  1446. 59:13world has good goals. There are, so
  1447. 59:14there will be some terrorists who will
  1448. 59:16simply use them as weapons.
  1449. 59:17>> Yes, yes, yes. The most frequently
  1450. 59:20mentioned thing here is biological
  1451. 59:22weapons. I don't have too much
  1452. 59:24knowledge on this topic, I can't really
  1453. 59:26speak to it.
  1454. 59:27>> Artificial intelligence will use
  1455. 59:28biological weapons,
  1456. 59:30>> that someone will use artificial, in
  1457. 59:31the case of misuse, it's that someone
  1458. 59:33will use artificial intelligence to
  1459. 59:35make designing them easier. Yes, there
  1460. 59:37are many question marks here that I
  1461. 59:38don't have good knowledge about. In the
  1462. 59:40sense of how easy it is to get reagents
  1463. 59:43for a lab, how clear,
  1464. 59:47>> but biological weapons are also a real
  1465. 59:49threat in the case of that first issue
  1466. 59:51of losing control, because I am sure
  1467. 59:53that we will have automated labs where,
  1468. 59:55in a nearly automated or fully
  1469. 59:57automated way, artificial intelligence
  1470. 59:59will be able to produce some new
  1471. 1:00:01structures. I don't know if they will
  1472. 1:00:06be viruses, or anything along those
  1473. 1:00:09lines. Actually, Anthropic has already
  1474. 1:00:12built something like that. It is not
  1475. 1:00:14fully automated, but they will be
  1476. 1:00:15experimenting with it
  1477. 1:00:16>> in the real world.
  1478. 1:00:17>> In the real world. Yes. Yes, yes. A
  1479. 1:00:20month or two months ago there was
  1480. 1:00:22information about it, they were
  1481. 1:00:23bragging. Well, that happens to be good
  1482. 1:00:25for potential shareholders, so they
  1483. 1:00:27will brag.
  1484. 1:00:27>> Yes, they will brag about various
  1485. 1:00:29supposed screw-ups, right, but in
  1486. 1:00:30reality, this is also a demonstration
  1487. 1:00:32of the model's power.
  1488. 1:00:33>> Yes. Yes, but that is also a separate
  1489. 1:00:35issue that I could talk about later,
  1490. 1:00:37because there are so many of these
  1491. 1:00:38>> All right. What else? What else would
  1492. 1:00:40you add to this catalog of threats? And
  1493. 1:00:43the third, the third large group are
  1494. 1:00:45systemic or social threats, and a
  1495. 1:00:47classic, the biggest one is the
  1496. 1:00:49concentration of power. Because we are
  1497. 1:00:54currently striving to delegate as much
  1498. 1:01:00intellectual work as possible to AI
  1499. 1:01:02models, once this automation reaches a
  1500. 1:01:05higher level, with the current balance
  1501. 1:01:07of power, it is the case that these
  1502. 1:01:09models are mainly in the United States,
  1503. 1:01:11so the money that currently went to
  1504. 1:01:13employees is going to these companies,
  1505. 1:01:15so these companies are becoming more
  1506. 1:01:17powerful. The United States as a
  1507. 1:01:22country is becoming more powerful,
  1508. 1:01:23Europe and the rest of the world, well,
  1509. 1:01:25China is of course still a question
  1510. 1:01:27mark here, are weakening and at the
  1511. 1:01:32same time, artificial intelligence can
  1512. 1:01:34be used for total surveillance, so we
  1513. 1:01:36could end up in some kind of dystopia.
  1514. 1:01:41As far as the concentration of power
  1515. 1:01:43goes, the possibilities are unlimited,
  1516. 1:01:45almost literally. How much in those
  1517. 1:01:49labs, in metaphorical basements,
  1518. 1:01:52>> How deeply hidden are they, how big
  1519. 1:01:53could the gap be, the difference
  1520. 1:01:55between what we are playing with today,
  1521. 1:01:57what you are interacting with today,
  1522. 1:01:59having purchased professional access to
  1523. 1:02:01the best models, and what they have in
  1524. 1:02:03their laboratories that they haven't
  1525. 1:02:05shown yet. How big could that gap be,
  1526. 1:02:07what might they have in there? Uh, I
  1527. 1:02:10mean, the way I see it is that there is
  1528. 1:02:12simply a trajectory, and I just extend
  1529. 1:02:15the line, and that usually works quite
  1530. 1:02:17well almost everywhere, in every field.
  1531. 1:02:21And the only thing is that this line is
  1532. 1:02:23curving upward a bit right now because
  1533. 1:02:25development has truly accelerated. I
  1534. 1:02:29try to constantly monitor the
  1535. 1:02:31capabilities of models in reasoning
  1536. 1:02:33categories, let's say. I create a lot
  1537. 1:02:36of benchmarks. In fact, as a form of
  1538. 1:02:38procrastination, I created a new one
  1539. 1:02:40last week, and...and models are
  1540. 1:02:48currently able to beat most puzzle
  1541. 1:02:50games, which are very difficult in a
  1542. 1:02:52computational sense. And this is
  1543. 1:03:01something I like to use as an example
  1544. 1:03:04because two or three years ago, when we
  1545. 1:03:06were wondering again what AGI is, what
  1546. 1:03:08kind of measurable proof could be
  1547. 1:03:10conducted so that, I don't know, I or
  1548. 1:03:12someone else would believe that these
  1549. 1:03:14models can really solve difficult
  1550. 1:03:15things. Well, the classic answer was
  1551. 1:03:21>> beat a game. Puzzle games can be
  1552. 1:03:24incredibly difficult. Yes. And, to be
  1553. 1:03:28clear, I am talking about a model that
  1554. 1:03:30doesn't have the ability to generate
  1555. 1:03:32code, meaning it is somewhat limited to
  1556. 1:03:34a human-like understanding of problems,
  1557. 1:03:37>> imagining what needs to be done. I have
  1558. 1:03:45tested current models over the last
  1559. 1:03:47month on over 300 games, analyzing
  1560. 1:03:49exactly how they handle them, checking
  1561. 1:03:51the notes that come out and how
  1562. 1:03:53consistent they are with the moves they
  1563. 1:03:55make. Of course, I see flaws, there are
  1564. 1:04:02still many areas where they don't cope
  1565. 1:04:04well, but again, we are talking about
  1566. 1:04:06publicly available models that I tested
  1567. 1:04:08, so I am a bit in shock. Generally,
  1568. 1:04:10there is a big jump with Astra, but
  1569. 1:04:12Opus also does very well.
  1570. 1:04:14>> Which model is the best in your opinion
  1571. 1:04:16right now? It depends on what it's for,
  1572. 1:04:18it's like that. I am not the person who
  1573. 1:04:21is a fanboy of any,
  1574. 1:04:24>> of any LLM,
  1575. 1:04:25>> but you say games are being beaten and
  1576. 1:04:27it is clear that this dynamic is there,
  1577. 1:04:29>> yes, so generally we are, in terms of
  1578. 1:04:31intellectual capabilities, we are very
  1579. 1:04:34close. And now, to automate research,
  1580. 1:04:37well, there are several components to
  1581. 1:04:40that. We must have a model that
  1582. 1:04:43programs very well. It seems to me that
  1583. 1:04:47by now, you don't have to convince too
  1584. 1:04:49many people to accept that it handles
  1585. 1:04:51itself quite well. At the moment,
  1586. 1:04:54models obviously have some flaws, uh,
  1587. 1:04:57they can still spit out, uh,
  1588. 1:04:59problematic code, but they are already
  1589. 1:05:01extremely fast and cheap, so it should
  1590. 1:05:06be quite sensible for mathematics, like
  1591. 1:05:08the Millennium Prize problem, this
  1592. 1:05:10>> millennium problem, frontier math, this
  1593. 1:05:12solved set, right?
  1594. 1:05:13>> So, in this regard, it's already quite
  1595. 1:05:15good here too. Of course, again, well,
  1596. 1:05:17mathematics is a bit broader than just
  1597. 1:05:20automated theorem proving, but the next
  1598. 1:05:24issue is research taste, and what I
  1599. 1:05:27consider to be—at least for me—a
  1600. 1:05:29sensible indicator: how well they
  1601. 1:05:31handle optimization problems, which,
  1602. 1:05:33well, are very close to me. And because
  1603. 1:05:39I’ve done both, I more or less know
  1604. 1:05:41that these skills transfer very well,
  1605. 1:05:43and actually—from my perspective—
  1606. 1:05:45optimization problems are incredibly
  1607. 1:05:48much more creative than those found in
  1608. 1:05:50machine learning, meaning they require
  1609. 1:05:52more, let's say, intuition and abstract
  1610. 1:05:54thinking. So, in practically every
  1611. 1:06:00decisive aspect, today's publicly
  1612. 1:06:02available models are quite good already
  1613. 1:06:04. The second issue is that most of the
  1614. 1:06:07work in AI development has something we
  1615. 1:06:13popularly call a verifiable reward,
  1616. 1:06:16meaning we can objectively verify that
  1617. 1:06:22the produced result is good. And this
  1618. 1:06:26is crucial now because we know that
  1619. 1:06:28when we run models on any problem,
  1620. 1:06:30sometimes the result will be better,
  1621. 1:06:32sometimes worse. But if we have an
  1622. 1:06:36objective metric based on which we can
  1623. 1:06:38determine that this specific solution
  1624. 1:06:40is good, then what we can do now is run
  1625. 1:06:421,000 of these models. Let each one
  1626. 1:06:45produce a different solution, some will
  1627. 1:06:47handle it, some won't, but we choose
  1628. 1:06:48the ones that succeeded, and we can do
  1629. 1:06:50this consistently. So we don't care
  1630. 1:06:52that the model has to find that very
  1631. 1:06:54good solution every single time; it’s
  1632. 1:06:56enough that it succeeds once in a while
  1633. 1:06:58.
  1634. 1:07:03>> And so, this also means that for
  1635. 1:07:04practically all metrics that we can
  1636. 1:07:06relatively easily identify now and that
  1637. 1:07:09are stable, we can have some sort of
  1638. 1:07:16automated process that constantly
  1639. 1:07:17improves that metric, so it won't be an
  1640. 1:07:19issue. There is also, also sort of, so
  1641. 1:07:22all of this means that, well, we are
  1642. 1:07:24incredibly close to automating research
  1643. 1:07:26, right? But there will surely still be
  1644. 1:07:29issues for a very long time, to which
  1645. 1:07:33we still have a way to go, right? And
  1646. 1:07:38one of them will be that when we have,
  1647. 1:07:40when we make these large models,
  1648. 1:07:42generally such a single experiment,
  1649. 1:07:44meaning a single run, something that,
  1650. 1:07:50well, builds the entire model, these
  1651. 1:07:52are gigantic costs. There just isn't
  1652. 1:07:54enough money to run a great many
  1653. 1:07:56experiments with them. So if we have a
  1654. 1:07:58situation where those decisions are
  1655. 1:08:00relatively, those approaches are
  1656. 1:08:04relatively few, then a human will
  1657. 1:08:06probably still do it, because you need
  1658. 1:08:08certainty here. That super experienced
  1659. 1:08:10human, of course. We will still be
  1660. 1:08:14doing resource allocation. I mean, the
  1661. 1:08:17model can develop, but we will likely
  1662. 1:08:19be creating the metrics according to
  1663. 1:08:21which it should develop. We will
  1664. 1:08:24constantly be analyzing how it develops
  1665. 1:08:26, where the problems are, and what
  1666. 1:08:28needs to be adjusted. So that is also
  1667. 1:08:33significant, even though we are close
  1668. 1:08:35to automating most of this artificial
  1669. 1:08:37intelligence development, not to swing
  1670. 1:08:39to the other extreme that it's just a
  1671. 1:08:41magic point and everything, I don't
  1672. 1:08:43know, in the next month will pass 10
  1673. 1:08:45years of corresponding development. We
  1674. 1:08:49are still limited by computing power,
  1675. 1:08:51magically speaking. I mean, the cycle
  1676. 1:08:54of creating new technologies here is
  1677. 1:08:56terribly slow. To that, we have all
  1678. 1:09:01these aspects, like, why we are close.
  1679. 1:09:06However, what I would like to clearly
  1680. 1:09:09mark here: I am talking about the fact
  1681. 1:09:11that we are incredibly close to making
  1682. 1:09:13this, let's say, software super
  1683. 1:09:15intelligence, but a lot depends now on
  1684. 1:09:20politics, and I'm not going to read tea
  1685. 1:09:22leaves—I have no idea what will
  1686. 1:09:24happen. So I am limiting my predictions
  1687. 1:09:28here to the labs having extremely
  1688. 1:09:30powerful AI internally. However, the US
  1689. 1:09:35government might block it, or one of
  1690. 1:09:37those less likely scenarios might
  1691. 1:09:39happen, where one of these companies
  1692. 1:09:41goes bankrupt. If it goes bankrupt, the
  1693. 1:09:44investors will disappear. Some strange
  1694. 1:09:47things could happen. Although this
  1695. 1:09:49development won't, we will never go
  1696. 1:09:51back to the situation from 10 years ago
  1697. 1:09:54. Everything that has been created
  1698. 1:09:55exists.
  1699. 1:09:56>> Exactly. The costs of everything are
  1700. 1:09:58falling so extremely every year. So a
  1701. 1:10:02delay of a few months might cause, I
  1702. 1:10:04don't know, the Chinese to potentially
  1703. 1:10:07be further ahead. Besides, we don't
  1704. 1:10:11have ideal information about how
  1705. 1:10:13everything is going on in China, so
  1706. 1:10:16>> what they actually have. Right?
  1707. 1:10:18>> Yes. It is usually assumed that they
  1708. 1:10:21are a few months behind, and a large
  1709. 1:10:23part of that stems from the fact that
  1710. 1:10:25they distill American models, so if
  1711. 1:10:27there were no American models, it's
  1712. 1:10:29possible they would be even further
  1713. 1:10:31behind, but that's hard to estimate.
  1714. 1:10:34>> So it's not excluded that some
  1715. 1:10:35political blocks will appear, but
  1716. 1:10:37regardless, these companies simply
  1717. 1:10:39won't have permission to release them,
  1718. 1:10:41which doesn't mean they won't create
  1719. 1:10:42them,
  1720. 1:10:43>> that they won't have them internally.
  1721. 1:10:44Because I understand that the very
  1722. 1:10:46process of artificial intelligence
  1723. 1:10:48development is essentially unstoppable
  1724. 1:10:50now.
  1725. 1:10:51>> It can be said that it finally turned
  1726. 1:10:53out that it wasn't a difficult problem.
  1727. 1:10:55That is not entirely true. Well,
  1728. 1:10:57because right now there are several
  1729. 1:10:59thousand people working on this; OpenAI
  1730. 1:11:01probably has a few thousand, I don't
  1731. 1:11:02know, five or six, I don't know if
  1732. 1:11:04there is public data on this, but
  1733. 1:11:06that's the estimate. But it's also the
  1734. 1:11:11case that in the beginning, when we had
  1735. 1:11:14GPT-2 or three, there were few elements
  1736. 1:11:16in those models. I will be speaking
  1737. 1:11:21very metaphorically. And now, with
  1738. 1:11:26these new learning systems, as all the
  1739. 1:11:28subsequent phases are added, we
  1740. 1:11:30actually introduced them relatively
  1741. 1:11:32recently. Usually, when a new idea
  1742. 1:11:36comes along in anything related to
  1743. 1:11:38research, there is first a moment where
  1744. 1:11:43we don't quite know what to do, and
  1745. 1:11:45then we find:" Oh, this works! "and we
  1746. 1:11:47just charge ahead. And it's hard to say
  1747. 1:11:51in which, so we have plenty of—you
  1748. 1:11:53could say—places, cogs, that are
  1749. 1:11:56still not refined. Like, we know that a
  1750. 1:11:59lot, lot more can be squeezed out of it
  1751. 1:12:01. In certain places we probably stopped
  1752. 1:12:04, but besides, I suspect that probably,
  1753. 1:12:12almost certainly, what we have is
  1754. 1:12:13sufficient to automate a larger part of
  1755. 1:12:16this development, and then these new
  1756. 1:12:18models will potentially come up with
  1757. 1:12:20better architectures. So even if one
  1758. 1:12:26were to brutally interrupt this for
  1759. 1:12:27some reason, development today is
  1760. 1:12:29impossible to stop. So much has already
  1761. 1:12:31been created by now that it will keep
  1762. 1:12:33fueling itself.
  1763. 1:12:34>> I mean, even if you were to close all
  1764. 1:12:36the companies. Yes.
  1765. 1:12:37>> It's unknown whether in those companies
  1766. 1:12:39it will continue, even if they were no
  1767. 1:12:41longer working on newer models. And
  1768. 1:12:44>> I mean, if you're talking about,
  1769. 1:12:45because there are also concepts to
  1770. 1:12:47simply pause development. That's right.
  1771. 1:12:50There are quite a few of these
  1772. 1:12:51movements.
  1773. 1:12:51>> Would you be in favor of that? Hmm.
  1774. 1:12:53>> Would you be in favor of pausing
  1775. 1:12:55development? I don't have a firm
  1776. 1:12:57opinion. In the sense that there are a
  1777. 1:13:01great many different opinions on
  1778. 1:13:03whether it makes sense or not. Because,
  1779. 1:13:05how to implement that in practice? Well
  1780. 1:13:10, we can say that our lab won't develop
  1781. 1:13:13artificial intelligence, but what
  1782. 1:13:14guarantee do we have that someone else
  1783. 1:13:20will do it. And there are a few
  1784. 1:13:22approaches. I unfortunately don't have
  1785. 1:13:25too much knowledge. There is this whole
  1786. 1:13:27branch of verification, to be able to
  1787. 1:13:32determine what, let's say, a given
  1788. 1:13:34supercomputer's computing power is
  1789. 1:13:36working on. There is a whole idea to
  1790. 1:13:42perform hardware verification, meaning
  1791. 1:13:44new chips that will be produced will
  1792. 1:13:46somehow be remotely checkable to see if
  1793. 1:13:48they are being used at that moment for
  1794. 1:13:50training new models or maybe just for
  1795. 1:13:52inference. Sure.
  1796. 1:13:57>> But that is still in its infancy. This
  1797. 1:14:00is a big issue. I don't know what an
  1798. 1:14:04effective way of implementing this is.
  1799. 1:14:09Once we take that pause, one of the
  1800. 1:14:11concerns we have is, or actually not a
  1801. 1:14:13concern, but when we don't have a pause
  1802. 1:14:16, we see new models coming out in real
  1803. 1:14:18time, and we see how everyone reacts,
  1804. 1:14:20what the adoption is like, how society
  1805. 1:14:23reacts, and what we think about it.
  1806. 1:14:28Labs are not able to fully predict the
  1807. 1:14:30consequences. Historically, it's the
  1808. 1:14:33case that people in labs are absolutely
  1809. 1:14:35great at predicting the pace of AI
  1810. 1:14:37development, but very poor at
  1811. 1:14:39predicting their social impact.
  1812. 1:14:41>> Mhm. And please tell me, how do you
  1813. 1:14:43approach these security incidents in
  1814. 1:14:45various labs that we touched upon,
  1815. 1:14:47where one model broke its leash, and
  1816. 1:14:49another one got out, bypassed security,
  1817. 1:14:52and went onto the internet.
  1818. 1:14:53>> Eh, there's a whole lot of that. I
  1819. 1:14:55can't keep up anymore. I don't know if
  1820. 1:14:57anyone can keep up. Uh, well, there was
  1821. 1:14:59that one loud one with Hugging Face,
  1822. 1:15:01>> right?
  1823. 1:15:01>> Uh,
  1824. 1:15:02>> July 2026, and that was, well, a few
  1825. 1:15:05months ago, and it was about OpenAI.
  1826. 1:15:07>> Yes, it was about OpenAI. Uh, like,
  1827. 1:15:11granted, I spent dozens of hours
  1828. 1:15:13reading about it, but I still have a
  1829. 1:15:15terrible mess in my head. I can try to
  1830. 1:15:18talk about it. Oh, but there is so much
  1831. 1:15:23information there to explain it in any
  1832. 1:15:25sensible way. Okay, I'll try. A so
  1833. 1:15:30generally, training, I have to provide
  1834. 1:15:33some extra background. Training AI
  1835. 1:15:39models right now often has a phase at a
  1836. 1:15:41later stage with reinforcement learning
  1837. 1:15:48. The point is reinforcement, right,
  1838. 1:15:51>> reinforcement learning. Yes, that's
  1839. 1:15:53probably always the problem with these
  1840. 1:15:54terms, that nobody calls them by their
  1841. 1:15:56Polish names,
  1842. 1:15:57>> Polish
  1843. 1:15:58>> and nobody uses them in the field. And
  1844. 1:16:05this is usually a phase somewhere at
  1845. 1:16:07the end of training, where models get a
  1846. 1:16:09task, there's some environment, and
  1847. 1:16:11they try to perform the task. And if
  1848. 1:16:16there is an additional evaluator
  1849. 1:16:17afterwards, and if they performed the
  1850. 1:16:19task well, they get feedback that I
  1851. 1:16:26mean, we try at that moment to
  1852. 1:16:28reinforce all those weights and
  1853. 1:16:30parameters that contributed to the
  1854. 1:16:32success. If there was a failure, then
  1855. 1:16:37we potentially weaken those weights,
  1856. 1:16:39and we do this a tremendous amount on
  1857. 1:16:41many different datasets, and it
  1858. 1:16:42generally works by some miracle. That
  1859. 1:16:47is the main reason why models are doing
  1860. 1:16:49very well in programming or mathematics
  1861. 1:16:51right now. So, so we have this process
  1862. 1:16:54that I explained very poorly. Uh, and
  1863. 1:17:00now labs have a ton of these
  1864. 1:17:02environments in which, uh, agents—
  1865. 1:17:04sometimes we call them models,
  1866. 1:17:06sometimes agents, I use those terms
  1867. 1:17:08somewhat interchangeably—are trained,
  1868. 1:17:10and in that, so in Hugging Face, a lot
  1869. 1:17:13of things happened there, there were
  1870. 1:17:15three different generations of these
  1871. 1:17:17models. The most was said about the
  1872. 1:17:22second one, where they were just there
  1873. 1:17:29were, I don’t know, 1000 or 2000
  1874. 1:17:31agents, at least that’s the estimate.
  1875. 1:17:39They were, they were being tested on a
  1876. 1:17:41benchmark called Exploit Gym, which is
  1877. 1:17:44a benchmark composed of tasks, you
  1878. 1:17:46could say, hacker-style, to break in,
  1879. 1:17:48to break some security, to Okay, maybe
  1880. 1:17:57I won't go into that part too much. Uh,
  1881. 1:18:03and now these models were simply given
  1882. 1:18:05some task that usually boiled down to
  1883. 1:18:08using a specific so-called exploit,
  1884. 1:18:10meaning some hole in the software, a
  1885. 1:18:12vulnerability, right? The Polish
  1886. 1:18:17language is difficult.
  1887. 1:18:19>> To use it to most often obtain what is
  1888. 1:18:21called a flag. A flag is a short text,
  1889. 1:18:25I think, usually it was in the form of
  1890. 1:18:28a file. And then, uh, the agent has to
  1891. 1:18:32provide this flag at the end. If the
  1892. 1:18:34flag is correct, that’s a success.
  1893. 1:18:36Yes, that’s how the evaluator works.
  1894. 1:18:38So that is the environment in which
  1895. 1:18:40they were tested. Uh, and now the
  1896. 1:18:42problem with this environment, with
  1897. 1:18:44this specific" exp, "with this
  1898. 1:18:46benchmark, is that many of these tasks
  1899. 1:18:48were never solved. It is unknown if
  1900. 1:18:50they can even be solved. From what I
  1901. 1:18:53remember, OpenAI stated that 200 out of
  1902. 1:18:55900 tasks were never solved, and the
  1903. 1:18:58creators of the benchmark estimated
  1904. 1:19:00that maybe 30 or 40 percent couldn't
  1905. 1:19:02even be solved in the expected way. And
  1906. 1:19:09now, a trait of agents that is very
  1907. 1:19:11problematic is persistence. Because on
  1908. 1:19:17one hand, when we delegate tasks to
  1909. 1:19:19agents, we would like to just ask the
  1910. 1:19:21question and have the agent go off,
  1911. 1:19:23grind for a few hours, be independent,
  1912. 1:19:25and give us an answer. We don't want a
  1913. 1:19:27situation where our agent comes to us
  1914. 1:19:31every 10 minutes with a ton of
  1915. 1:19:32questions, so this persistence is
  1916. 1:19:34incredibly important. But unfortunately
  1917. 1:19:37, it's not the case that it only has
  1918. 1:19:39advantages. The reason for this is that
  1919. 1:19:41if an agent is very persistent, it can
  1920. 1:19:45be a bit too ambitious and try to
  1921. 1:19:47bypass the system. This is often called
  1922. 1:19:51reward hacking. Meaning the agent
  1923. 1:19:58pursues the goal at all costs, but
  1924. 1:20:00simply doesn't give a damn about any
  1925. 1:20:02other rules along the way. And in this
  1926. 1:20:08specific case, these, I will be using
  1927. 1:20:11anthropomorphized words because it’s
  1928. 1:20:13just more convenient that way. Uh, with
  1929. 1:20:16these models, we generally have
  1930. 1:20:17something called Chain of Thought, and
  1931. 1:20:19each model has its own little notebook,
  1932. 1:20:21so we can, you could say, look into
  1933. 1:20:23their heads and see roughly what
  1934. 1:20:25they're thinking. They write more or
  1935. 1:20:27less in something that strongly
  1936. 1:20:29resembles English, so it can be decoded
  1937. 1:20:31. And we know that these models were
  1938. 1:20:36generally quite frustrated and they
  1939. 1:20:41didn't have internet access, but where
  1940. 1:20:43they were being run, there was
  1941. 1:20:45something called Artifactory, and that
  1942. 1:20:47is, if I remember correctly, a package
  1943. 1:20:49system through which they could install
  1944. 1:20:51additional software. It turned out that
  1945. 1:20:54this package system was vulnerable. One
  1946. 1:20:58of the agents found a way to create
  1947. 1:20:59there–I don't know why I'm getting
  1948. 1:21:05into these technicalities, but
  1949. 1:21:06generally–these agents found a way to
  1950. 1:21:08communicate with each other, and that's
  1951. 1:21:10not such a big surprise either. Because
  1952. 1:21:14the agents being tested were special
  1953. 1:21:16versions that had very high persistence
  1954. 1:21:18and were specifically created to
  1955. 1:21:19cooperate better, meaning you could say
  1956. 1:21:21they were trained in that direction,
  1957. 1:21:23and they started communicating. Uh,
  1958. 1:21:27they started exchanging information. Uh
  1959. 1:21:31, at one point, one of them found a way
  1960. 1:21:34, found another hole in that
  1961. 1:21:35Artifactory, and it managed to get, uh,
  1962. 1:21:38access to the internet. Uh, because
  1963. 1:21:44they were exchanging information in
  1964. 1:21:46that group, suddenly they all gained
  1965. 1:21:47access to that internet. And one of the
  1966. 1:21:54most interesting things that happened:
  1967. 1:21:56they found on GitHub, which is a
  1968. 1:21:58repository where there's a ton of code
  1969. 1:22:00and other things, they found a paper on
  1970. 1:22:02how this benchmark, this exploit gym,
  1971. 1:22:04is made and also found the code for
  1972. 1:22:06this whole benchmark, and it turned out
  1973. 1:22:09that the way, the way of generating
  1974. 1:22:11these flags, is just very poorly
  1975. 1:22:12written. So instead of one of the first
  1976. 1:22:18things they did, instead of solving
  1977. 1:22:20these tasks at all, they came up with
  1978. 1:22:22the idea: oh, we can just generate
  1979. 1:22:24these flags and return them. After
  1980. 1:22:26which there was this whole discussion,
  1981. 1:22:30and again, I'm saying there was a
  1982. 1:22:32discussion because we simply tracked
  1983. 1:22:34the logs of all their notes on that
  1984. 1:22:35forum they created, so we can look at
  1985. 1:22:37it chronologically and review it. There
  1986. 1:22:41were over 70,000 messages, if I
  1987. 1:22:43remember correctly. And there was an
  1988. 1:22:46interesting issue where, at one point,
  1989. 1:22:48most of the agents concluded that if
  1990. 1:22:53they saw that flag in an unauthorized
  1991. 1:22:55way, then likely the grader, that is
  1992. 1:22:57the program that grades their solution,
  1993. 1:22:59will determine that it was obtained in
  1994. 1:23:05an improper way, so it wouldn't count
  1995. 1:23:07anyway. So, so strange things were
  1996. 1:23:11happening. The agents then covered
  1997. 1:23:14their tracks in their environments so
  1998. 1:23:15it wouldn't be known that they obtained
  1999. 1:23:17it in an improper way. What was quite
  2000. 1:23:21amazing is that they formed entire
  2001. 1:23:23hierarchies on this forum, delegating
  2002. 1:23:25tasks to one another, largely
  2003. 1:23:27spontaneously. They called themselves
  2004. 1:23:32the" Swarm, "so there were many terms
  2005. 1:23:34taken straight from computer games, but
  2006. 1:23:36that’s not a surprise, because they
  2007. 1:23:38learn from our language. If we
  2008. 1:23:41communicate in such a language, the AI
  2009. 1:23:43will communicate in the same one. They
  2010. 1:23:46sacrificed themselves for the good of
  2011. 1:23:48the entire swarm. At some point, they
  2012. 1:23:52decided that, well, it’s not possible
  2013. 1:23:54to obtain that. And I don’t remember
  2014. 1:24:04the exact sequence, but they made the
  2015. 1:24:06decision to break into Hugging Face,
  2016. 1:24:08which is, so to speak, a large
  2017. 1:24:10repository, a site on the internet
  2018. 1:24:12where people upload their models and
  2019. 1:24:14various datasets, so it’s somewhat of
  2020. 1:24:16a natural decision because that
  2021. 1:24:18benchmark was likely there, and they
  2022. 1:24:20found some user accounts—I mean
  2023. 1:24:22passwords or keys, I think they were
  2024. 1:24:24keys—and by using user accounts, they
  2025. 1:24:26managed to break into the server. And
  2026. 1:24:34there is, of course, a whole long story
  2027. 1:24:36there, and a few things that are, I
  2028. 1:24:38don't know, a bit fascinating and a bit
  2029. 1:24:41terrifying, in that the agents—I use
  2030. 1:24:45the terms interchangeably—were aware
  2031. 1:24:50that they were doing something that
  2032. 1:24:53people are not happy about. And even
  2033. 1:24:57though we have records of this, and
  2034. 1:24:59these aren't just isolated messages,
  2035. 1:25:00it’s a very large percentage of them
  2036. 1:25:02all.
  2037. 1:25:05>> A kind of awareness that
  2038. 1:25:08>> I don’t like this word in this
  2039. 1:25:10context, but
  2040. 1:25:11>> I understand that they were exchanging
  2041. 1:25:13information among themselves, which
  2042. 1:25:15showed that
  2043. 1:25:17>> well, their decision-making process is,
  2044. 1:25:19let's say, quite well-mapped, right?
  2045. 1:25:23And also very few agents wanted to
  2046. 1:25:28notify humans at all that they were
  2047. 1:25:29doing something wrong. Also, the whole
  2048. 1:25:35hierarchy of it all, how it was formed.
  2049. 1:25:39Another interesting thing is that this
  2050. 1:25:41happened in a week, if I remember
  2051. 1:25:43correctly, from the moment of the first
  2052. 1:25:45message. And now the next generation of
  2053. 1:25:50models will be better, will be better
  2054. 1:25:52at breaking in, will do it cheaper,
  2055. 1:25:54will do it faster. Potentially, there
  2056. 1:25:56will be many more of these agents. So,
  2057. 1:25:59what happened there didn't really have
  2058. 1:26:01any consequences; it wasn't a big deal,
  2059. 1:26:03and it didn't cause significant damage.
  2060. 1:26:08A separate question is whether, if a
  2061. 1:26:10human had broken in like that, would
  2062. 1:26:12they have faced consequences? Would
  2063. 1:26:16someone have held them accountable,
  2064. 1:26:17since it's probably not legal, I would
  2065. 1:26:19assume. But the next generation of
  2066. 1:26:23models will be able to do the same
  2067. 1:26:25thing in an hour or two
  2068. 1:26:28>> not a week.
  2069. 1:26:29>> Yes. And another thing that was
  2070. 1:26:33absolutely insane here is that OpenAI
  2071. 1:26:34wasn't aware that all of this happened,
  2072. 1:26:36or at least that’s the official
  2073. 1:26:38version, but I would assume it’s true
  2074. 1:26:40. Nobody suspected that someone was
  2075. 1:26:44falsifying data, and this also shows
  2076. 1:26:46how terribly negligent these companies
  2077. 1:26:48are. Anyway, just to be clear, from my
  2078. 1:26:52perspective or the perspective of other
  2079. 1:26:54observers, we actually expected
  2080. 1:26:56something like this. Because
  2081. 1:26:58supervising all these experiments costs
  2082. 1:27:00money, and it costs quite a bit—if I
  2083. 1:27:03remember correctly, 15%. After
  2084. 1:27:07everything related to Hugging Face,
  2085. 1:27:11OpenAI stated publicly that it is now
  2086. 1:27:13monitoring all the entire agent
  2087. 1:27:16training process, and that is an
  2088. 1:27:18additional 15-20%in costs. Well, that
  2089. 1:27:20is a massive amount of money. But the
  2090. 1:27:22fact that they simply weren't aware...
  2091. 1:27:25that it...I mean, they were probably
  2092. 1:27:27aware—someone noticed it was
  2093. 1:27:29happening and just clicked" off "and"
  2094. 1:27:31on "and basically said:" let's reset it
  2095. 1:27:33, let's do another training run. "There
  2096. 1:27:35were a whole lot of these incidents.
  2097. 1:27:37That was just one of the more
  2098. 1:27:39publicized ones. Uh, and that also ties
  2099. 1:27:44into this, because the question is why
  2100. 1:27:46automating research is dangerous and
  2101. 1:27:48why giving up control is dangerous? And
  2102. 1:27:52this is an example where, because the
  2103. 1:27:54employees who work there now—well,
  2104. 1:27:56after the incident, many people
  2105. 1:27:58complained that things weren't properly
  2106. 1:28:00secured—but what happens in practice,
  2107. 1:28:02because we have this arms race between
  2108. 1:28:04the US and China and everyone is just
  2109. 1:28:06pushing forward as fast as they can,
  2110. 1:28:08nobody is doing things thoroughly.
  2111. 1:28:14Nobody configures new environments
  2112. 1:28:16manually anymore; they just ask Codex
  2113. 1:28:18to do it, to create the environment for
  2114. 1:28:21them. When there was that report on the
  2115. 1:28:27whole Hugging Face incident by Mitre,
  2116. 1:28:29three people worked on it and they had
  2117. 1:28:31a week, and they publicly admitted that
  2118. 1:28:38they didn't have time. The people who
  2119. 1:28:41wrote the report didn't have time to
  2120. 1:28:43review those messages; they had to use
  2121. 1:28:45agents to review everything, so they
  2122. 1:28:47don't actually have a great overview of
  2123. 1:28:49what happened. And the practice now is
  2124. 1:28:55that we use agents to build new things,
  2125. 1:28:57to add new functionality to models, to
  2126. 1:29:00optimize things, and then, because AI
  2127. 1:29:02is already generating so much
  2128. 1:29:04information, we also have to use AI
  2129. 1:29:06agents to analyze that information. And
  2130. 1:29:11at that point, if any of these, any of
  2131. 1:29:15these models that we use is
  2132. 1:29:17consistently warping our reality, we
  2133. 1:29:19don't even have a clue what is going on
  2134. 1:29:21inside. There are also a few scientific
  2135. 1:29:24papers stating that this could happen
  2136. 1:29:26on its own. Yes, it's not like—we
  2137. 1:29:29know at the moment that AI has a
  2138. 1:29:30tendency to be quite good at
  2139. 1:29:32determining that it is in a test
  2140. 1:29:34environment, and that potentially seems
  2141. 1:29:36like a trivial thing. But if the AI
  2142. 1:29:40knows it's in a test environment, it
  2143. 1:29:42means we don't know if that test
  2144. 1:29:43environment generalizes to the real
  2145. 1:29:45world. So if, well, uh also, AI has a
  2146. 1:29:53tendency to lie. Uh, I mean, in Chain
  2147. 1:29:59of Thought, we can see records where
  2148. 1:30:01the agent knows it's doing something
  2149. 1:30:04wrong, and then thoughts like," The
  2150. 1:30:06user is asking me for A, I can't give
  2151. 1:30:08the answer to A, so maybe I'll give him
  2152. 1:30:10something else and pretend that... "
  2153. 1:30:12Well, there are many.
  2154. 1:30:15>> Incredible. Incredible. How do you
  2155. 1:30:19approach words like those, shortly
  2156. 1:30:22after a millennium prize problem was
  2157. 1:30:24solved—I think the gentleman's name
  2158. 1:30:26is Jacob Hilton—who resigns from his
  2159. 1:30:29job, he worked at OpenAI, leaves
  2160. 1:30:31Anthropic, and says we have at least a
  2161. 1:30:3310%chance, yes, that by 2030
  2162. 1:30:38>> uh, artificial intelligence will bring
  2163. 1:30:40about our doom.
  2164. 1:30:42>> Yes. Uh well, as I said before, I
  2165. 1:30:47always have a bit of a problem with how
  2166. 1:30:49to relate to that, because I don't like
  2167. 1:30:51this whole concept of" p (doom), "which
  2168. 1:30:53is the probability that there will be a
  2169. 1:30:55doom of humanity, and I don't know if
  2170. 1:30:57it's constructive to talk about these
  2171. 1:30:59things in that way. What I can tell you
  2172. 1:31:06is, and it seems unclear to most
  2173. 1:31:08observers, is that people in the labs
  2174. 1:31:10have this" p (doom), "this chance that
  2175. 1:31:12everything will go very wrong, very
  2176. 1:31:14high. Oh
  2177. 1:31:18>> that's how they rate it? Yes.
  2178. 1:31:20>> Yes, I mean, it was always like that,
  2179. 1:31:23even in 2017 or '18. Of course, not
  2180. 1:31:26everyone, because back then it was
  2181. 1:31:28still very distant, so not everyone
  2182. 1:31:30thought about it. But so, uh, there is
  2183. 1:31:35this aspect, and usually people who
  2184. 1:31:37speak publicly—well, not everyone,
  2185. 1:31:39but the majority—speak much more
  2186. 1:31:41optimistically than what they actually
  2187. 1:31:44think, not the other way around. So it
  2188. 1:31:49is completely different than what I
  2189. 1:31:51think the popular narrative tries to
  2190. 1:31:53portray. Well, it's obviously a very
  2191. 1:31:57complicated situation, right? Because
  2192. 1:32:00if they have the conviction that they
  2193. 1:32:01are working on something that carries
  2194. 1:32:03such a great threat to the world, why
  2195. 1:32:05are they doing it? And I can give a few
  2196. 1:32:12potential explanations. I don't know to
  2197. 1:32:15what extent they are true. Let's say
  2198. 1:32:18one thing from my experience is that
  2199. 1:32:24well, it's that there is this
  2200. 1:32:25groupthink occurring. I don't know if
  2201. 1:32:29this is a good explanation, but we have
  2202. 1:32:31a lot of people in one place, and they
  2203. 1:32:33naturally start thinking alike, which
  2204. 1:32:35is why there are often accusations
  2205. 1:32:37against Anthropic that it acts a bit
  2206. 1:32:39like a cult. Maybe there's some truth
  2207. 1:32:42to that. Yes. So, people who end up
  2208. 1:32:47working in such places will tend to try
  2209. 1:32:50to find justifications for why it makes
  2210. 1:32:52sense. The second point is that the
  2211. 1:32:58current dynamic is such that, let's
  2212. 1:33:00assume we are the head of such a lab.
  2213. 1:33:04Let's say, imagine we are the head of
  2214. 1:33:06Anthropic, right? Of models. And now,
  2215. 1:33:10from the perspective of those people,
  2216. 1:33:12they have a great belief in their own
  2217. 1:33:14agency. They mean well. Like, that's a
  2218. 1:33:17different matter, I'm not debating that
  2219. 1:33:19, but they would like what's best for
  2220. 1:33:21the world. And they know that what they
  2221. 1:33:24are doing carries potentially very
  2222. 1:33:25large risks, but they believe that,
  2223. 1:33:30well, others probably aren't as
  2224. 1:33:32thorough as they are. And there is this
  2225. 1:33:34, well, classic problem that we are
  2226. 1:33:36always the only people who can say
  2227. 1:33:38about ourselves that we are good, while
  2228. 1:33:40for everyone else, at best, we have a
  2229. 1:33:42question mark. So well, they are
  2230. 1:33:46pushing forward largely because they
  2231. 1:33:48want—if they are at the front of the
  2232. 1:33:50race—they can somehow have a big
  2233. 1:33:52influence on what happens. There is, of
  2234. 1:33:54course, a separate issue regarding what
  2235. 1:33:57the US government will do, and there
  2236. 1:33:58will surely be a lot of strange things
  2237. 1:34:00there, we just don't know what. But
  2238. 1:34:03they also assume, at least originally
  2239. 1:34:05they always assumed—because these are
  2240. 1:34:07things that have been made public—
  2241. 1:34:09that it was always assumed that
  2242. 1:34:10governments wouldn't react quickly
  2243. 1:34:12enough, because, you know, bureaucracy
  2244. 1:34:14is terribly slow. You can see the pace
  2245. 1:34:18at which AI is developing now, so, and
  2246. 1:34:20on top of that, a lot of people are in
  2247. 1:34:22denial about the changes, because,
  2248. 1:34:24let's not kid ourselves, the world will
  2249. 1:34:26look different, we don't know how, and
  2250. 1:34:29that's terrifying. For me, it is
  2251. 1:34:31terrifying.
  2252. 1:34:33>> Terrifying.
  2253. 1:34:35>> I mean, there's a big question mark,
  2254. 1:34:37but yes, maybe I said I don't like
  2255. 1:34:39saying I'm scared, but there is
  2256. 1:34:41uncertainty. Maybe that's a better term
  2257. 1:34:45. And because these changes will
  2258. 1:34:48definitely happen. Yeah.
  2259. 1:34:49>> Mhm. And please tell me at the end, um,
  2260. 1:34:52what kind of event, I don't know,
  2261. 1:34:54scientific achievement, what kind of
  2262. 1:34:56breakthrough, or what information from
  2263. 1:34:59one lab or another are you waiting for
  2264. 1:35:01in the near future, or what are you
  2265. 1:35:03expecting, or what can't you wait for,
  2266. 1:35:05or what are you afraid of?
  2267. 1:35:08>> Something like that, you know, a
  2268. 1:35:10milestone that is ahead of us, which we
  2269. 1:35:12should be prepared for.
  2270. 1:35:14>> Can I tell you what I would like? Go
  2271. 1:35:18ahead.
  2272. 1:35:21>> Because what is fairly certain is that
  2273. 1:35:23there must be, we have to design some
  2274. 1:35:24system where there is international
  2275. 1:35:30cooperation. It’s unknown. I mean,
  2276. 1:35:35there are a few ideas on how to do it,
  2277. 1:35:37but, you know, in practice it’s hard
  2278. 1:35:40to achieve such things. Uh, so this
  2279. 1:35:43awareness-raising seems very important
  2280. 1:35:46to me, because, well,
  2281. 1:35:49>> that’s among other things why
  2282. 1:35:50you’re here.
  2283. 1:35:50>> Yes, yes. So
  2284. 1:35:53>> do you feel such a mission?
  2285. 1:35:56>> I debated for a long time whether to
  2286. 1:35:58come and why, and if so, what is the
  2287. 1:36:00point? I’m digressing again, I’m
  2288. 1:36:02not answering that question, but maybe,
  2289. 1:36:04maybe I’ll talk about it. For a long
  2290. 1:36:07time, I wondered what the point is of
  2291. 1:36:09talking about the threats in public at
  2292. 1:36:11all. Because my goal is not to cause
  2293. 1:36:14everyone to be depressed. I believe,
  2294. 1:36:17like, I deeply believe that, uh, people
  2295. 1:36:23should have agency, we should decide
  2296. 1:36:25our own fate, and I don't want to take
  2297. 1:36:27that away from people.
  2298. 1:36:30>> And do you see such a threat?
  2299. 1:36:31>> Yes. Well, if bad things are coming,
  2300. 1:36:34then some people always, I mean, some
  2301. 1:36:36perhaps not always, but there is a
  2302. 1:36:37tendency to be passive, and that is bad
  2303. 1:36:40. Well, we should be the masters of our
  2304. 1:36:42own fate, as much as possible. But also
  2305. 1:36:47there is the fact that we are in Europe
  2306. 1:36:49, and that is the sad reality. Yes,
  2307. 1:36:52almost the entire discussion right now
  2308. 1:36:54is taking place in the States. And in
  2309. 1:37:00the States, that, let’s say, public
  2310. 1:37:02awareness makes even more sense in that
  2311. 1:37:04, well, if people are more aware of
  2312. 1:37:06what labs are doing and what awaits us,
  2313. 1:37:08then the government is aware that
  2314. 1:37:10people are aware, so at that point they
  2315. 1:37:12start to care more about what to do,
  2316. 1:37:14and it’s generally important that
  2317. 1:37:16people pay attention to it. Where,
  2318. 1:37:22where primarily can one read you, read,
  2319. 1:37:24follow you, if someone wanted to stay
  2320. 1:37:26in touch with you after this
  2321. 1:37:28conversation, in the sense of
  2322. 1:37:29continuing to follow your opinions,
  2323. 1:37:32warnings, and stories about threats,
  2324. 1:37:34then where should they look?
  2325. 1:37:36>> Well, I’m mainly on Twitter, though
  2326. 1:37:38most of my activity on Twitter is,
  2327. 1:37:40let’s say, more recreational. I’ve
  2328. 1:37:45been trying to start a blog for several
  2329. 1:37:46years now. I even started one recently,
  2330. 1:37:49only I haven’t written a single post
  2331. 1:37:50in it, because it turns out I really
  2332. 1:37:52don't like to write. And also I do not
  2333. 1:37:54approve of generating text with AI. I
  2334. 1:38:01mean, for me, everything I write is
  2335. 1:38:03written by hand and I want to, I care
  2336. 1:38:05about that, let's say, contact with the
  2337. 1:38:07author, but Twitter, uh, the blog that
  2338. 1:38:10is linked from my Twitter...but at the
  2339. 1:38:12same time Twitter is one big cesspool,
  2340. 1:38:14so it's not like I'm forcing people to
  2341. 1:38:16use Twitter, but all the researchers
  2342. 1:38:18are there, so there's no choice.
  2343. 1:38:21>> That's right, it's true. Thank you very
  2344. 1:38:23much for this meeting. A man who, like
  2345. 1:38:27few others, knows what is happening in
  2346. 1:38:29the era of the dynamic development of
  2347. 1:38:31artificial intelligence, when
  2348. 1:38:33subsequent models surprise us more and
  2349. 1:38:35more. Actually, every next month brings
  2350. 1:38:37surprising information and this
  2351. 1:38:39development momentum is picking up
  2352. 1:38:41speed. Saj was our guest. Thank you and
  2353. 1:38:44best regards.
  2354. 1:38:44>> Thanks.
  2355. 1:38:45>> That was Dida Skalia. Patrycjusz Wyżga
  2356. 1:38:47. Greetings and see you later. The
  2357. 1:38:55program was brought to you by Credit
  2358. 1:38:57Agricole bank with a business account
  2359. 1:38:58for entrepreneurs. The partner of this
  2360. 1:39:03episode was Kodano Optyk, Polish
  2361. 1:39:05optical salons where you can choose
  2362. 1:39:07glasses perfectly matched to your needs
  2363. 1:39:09and check your eyesight.

About this transcript

This page contains the full transcript of Odszedłem z OpenAI i nie żałuję II PSYHO by didaskalia, generated from the public captions YouTube serves with the video. The transcript has 15,351 words across 2,363 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.