YouTube2Text

Is this the future of homelabbing? — Transcript

by Jim's Garage · 7,068 words · 1,012 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hey everybody and welcome back to Jim's
  2. 0:02Quen's garage. Today I'm going to be
  3. 0:04showing you about a little project I've
  4. 0:06had up and running for a couple of weeks
  5. 0:08and been following my socials, you'll
  6. 0:10know what I'm talking about. I'm slowly
  7. 0:12building up a new home lab, but there's
  8. 0:14one catch. I can only build it with AI.
  9. 0:17And to be honest, I suspect this is
  10. 0:19going to form a larger part of my home
  11. 0:21lab in the future than I ever suspected.
  12. 0:24That's because I've been able to build
  13. 0:25all of this just using Quen 3.8 Flash
  14. 0:28Next on a Strix Halo. And this will kick
  15. 0:31start a new series where I'll be looking
  16. 0:33at how I can build out my entire home
  17. 0:34lab and have it all managed and
  18. 0:36monitored by AI. And I'll also be
  19. 0:39exploring different levels of hardware
  20. 0:41we can use because I appreciate that
  21. 0:43this stuff isn't affordable for
  22. 0:44everyone. It wouldn't be affordable for
  23. 0:46me if I didn't receive some of the
  24. 0:47sponsored review units. So first, let's
  25. 0:50have a look at the setup that I've got
  26. 0:51and then we'll dive into what it's
  27. 0:53actually managed to configure, which has
  28. 0:54totally blown my mind. I'm literally
  29. 0:57saving hours per week on managing my
  30. 0:59home lab and I want to share that with
  31. 1:01you. So just to hook you in before we
  32. 1:03dive into that, this is an example of
  33. 1:06what I've built here. This is a full
  34. 1:08Docker virtual machine I've got running
  35. 1:10in Proxmox. Everything you see here, all
  36. 1:12of these containers and there are more,
  37. 1:14just some of them don't have a homepage
  38. 1:16integration was all set up using a LLM.
  39. 1:20Now you'll even notice that things like
  40. 1:22the certificates, all of those are
  41. 1:24working. This is a valid cert that's
  42. 1:25been pulled down from Let's Encrypt.
  43. 1:28I've got things like Home Assistant set
  44. 1:29up. I've got Paperless in there. I've
  45. 1:32got Audiobook Shelf, Netbird. I can even
  46. 1:34monitor the LLM itself as well. I 've
  47. 1:37got Uptime Kuma to report everything.
  48. 1:40All of that is then configured to run to
  49. 1:42Gotify so I get real-time messaging
  50. 1:44alerting of what's going on. I have
  51. 1:46Vaultwarden where it automatically every
  52. 1:48time I spin up a container, it
  53. 1:50automatically stores all of the
  54. 1:51credentials in Vaultwarden and only
  55. 1:53accesses all of the APIs and command
  56. 1:55lines via credentials stored in
  57. 1:57Vaultwarden. I 've also got this handy
  58. 1:59Grafana which it created itself to
  59. 2:01monitor itself. And I've even got NA10
  60. 2:05workflows, which I'll come on to later,
  61. 2:07which it created because it used Hermes
  62. 2:09agent skills and then transferred them
  63. 2:11over into automations. All of that's
  64. 2:13great because it gives me consistent
  65. 2:15workflows and it also saves on token
  66. 2:17generation.
  67. 2:19Now don't worry, we're going to come
  68. 2:20back to all of this if that all sounds a
  69. 2:22bit foreign at the moment. So how have I
  70. 2:24got this set up? Well, here you can see
  71. 2:25my Minisforum MSS1 Max and this is a
  72. 2:28Tricks Halo device that was sent to me
  73. 2:30for review about a year ago. And this
  74. 2:32thing has been absolutely awesome and
  75. 2:34I've only just recently dusted it off
  76. 2:36because I'm curious to get more involved
  77. 2:38with AI. And to be honest, the local
  78. 2:40models that you can now run are really
  79. 2:43catching up. This thing is about Opus
  80. 2:454.6 level, so it's totally usable within
  81. 2:48a home lab. And I've generated now about
  82. 2:5120 million odd tokens a week. So the
  83. 2:53return on investment if you bought this,
  84. 2:55you're probably looking at low order
  85. 2:57number of weeks compared to something
  86. 2:59like say Claude. Sure, don't get me
  87. 3:01wrong, something like Opus 5.5 or any of
  88. 3:04the Fable Mythos models, they are
  89. 3:06definitely better, more capable than
  90. 3:07this. But as I'm going to show you in
  91. 3:09this video, the local models are
  92. 3:12surprisingly advanced and what they can
  93. 3:14do is still just blowing my mind. Oh,
  94. 3:16and did I mention that all of this I can
  95. 3:18actually talk to the LLM through
  96. 3:20something like Matrix? There's a ton of
  97. 3:22other gateways you can use, but I can
  98. 3:24basically talk to this thing on my
  99. 3:26phone. I can ask it say, what's going on
  100. 3:28in my home lab? Is everything running?
  101. 3:30And it will come back and say yes or no.
  102. 3:32And if it does find a problem, it will
  103. 3:34present a fix for it. So here you can
  104. 3:36see I ask it, you see the eyeballs come
  105. 3:38up. It will go away now, it will query
  106. 3:40that Docker virtual machine and it will
  107. 3:42come back with an answer of what's
  108. 3:43happening on there. One thing I found
  109. 3:45even more amazing is the fact that I
  110. 3:47actually found a root cause of why this
  111. 3:49machine was crashing. I was pulling my
  112. 3:51hair out thinking what on earth has gone
  113. 3:53wrong here. And it turns out that the
  114. 3:55version of Ubuntu I'm using 2604 runs
  115. 3:58container D 2.2.2 which has a shim
  116. 4:01deadlock issue which is a known bug. And
  117. 4:04the only way to fix that is to either
  118. 4:05upgrade manually or wait for it to be
  119. 4:08available via APT.
  120. 4:10So that is awesome that it can debug and
  121. 4:12save me hours that I would have to spend
  122. 4:14searching forums, etc. Now there are
  123. 4:16many inference engines that you can run,
  124. 4:18but the one I'm running is called
  125. 4:20Halogen Server. Now I've been following
  126. 4:22Halogen Server now for quite a few
  127. 4:24weeks. Um there's always a stream of
  128. 4:26updates. Unfortunately, not all of it is
  129. 4:29open source. Still some of the weights
  130. 4:31for the models are locked. I hope that
  131. 4:34that does change in the near future, but
  132. 4:36for now I'm pretty happy with using it.
  133. 4:39To get this up and running was pretty
  134. 4:41straightforward. I simply spun up a
  135. 4:43Fedora 44 server and then run the
  136. 4:46command down here, which is pretty
  137. 4:48straightforward. Just a quick stop.
  138. 4:51There are a ton of options for this
  139. 4:53Halogen Server, so do go and check those
  140. 4:56out. It even includes being able to run
  141. 4:58smaller models on the NPU, which is
  142. 5:00pretty epic and takes this to a whole
  143. 5:02new level. I'll do a deep dive on this
  144. 5:05in the future, but for now the key stats
  145. 5:07that you're looking at are really up to
  146. 5:09about 94 tokens per second. On average I
  147. 5:12see more around sort of the low 70s and
  148. 5:15around 1800/1740-ish
  149. 5:19for prefill. For anyone who doesn't know
  150. 5:21what that means, the decode is kind of
  151. 5:22the output and the words you see. The
  152. 5:24prefill is basically working out what it
  153. 5:26is that you want it to do, i.e. the
  154. 5:27prompt you put it. Now all of this runs
  155. 5:30through a virtual machine that I
  156. 5:31created. And as I mentioned at the
  157. 5:34beginning, this is a mockup of a home
  158. 5:36lab because I really want to test this
  159. 5:37thing out, make sure I iron out all the
  160. 5:39wrinkles before I set it free on my
  161. 5:41actual home lab. And even then I'm
  162. 5:43probably going to take a staged approach
  163. 5:45where I roll containers over slowly just
  164. 5:47to make sure that it doesn't go crazy.
  165. 5:50Now the way I get this working is I run
  166. 5:52Hermes agent. So Hermes agent is
  167. 5:56basically a
  168. 5:58harness that you can use, and the key
  169. 6:00advantage of something like this is the
  170. 6:03fact that it comes with what called
  171. 6:05skills. So here, for example, in Hermes,
  172. 6:07you can actually see skills. Now, what
  173. 6:10skills are are basically a little bit
  174. 6:12like automations, but they're basically
  175. 6:15repositories of knowledge that the agent
  176. 6:18that the harness can use. So, for
  177. 6:20example, whenever I connect to this home
  178. 6:22lab Docker stack here, you'll see it's
  179. 6:24created a skill where it goes through
  180. 6:26and it basically says
  181. 6:28all the things that I've been asking it
  182. 6:29to do. It creates a memory of that. So,
  183. 6:32obviously with an LLM, it tries to work
  184. 6:34things out every time because that's
  185. 6:37just the nature of how they work. But
  186. 6:39here is basically a
  187. 6:41standard operating procedure for many of
  188. 6:43the things I commonly ask. So, when I
  189. 6:45tell it to create a container,
  190. 6:47it knows that firstly it needs to go and
  191. 6:49find the SSH key to connect. It keeps a
  192. 6:51record of where that is within these
  193. 6:53skills. And it does that for pretty much
  194. 6:55everything. When I ask it to perform
  195. 6:57updates of containers, it knows how to
  196. 6:59do that. We'll come onto that more
  197. 7:00later. And I've also, for value, created
  198. 7:03some NA10 workflows because they give me
  199. 7:06fixed automations where I have much more
  200. 7:08sort of tight granular control over the
  201. 7:10output. But basically, how I started all
  202. 7:12of this off was once Hermes was up and
  203. 7:14running, I created this first
  204. 7:17instructions.md.
  205. 7:18And I'll put all of these up on my
  206. 7:20GitHub in case you're interested. You
  207. 7:22will have to go through and modify them.
  208. 7:24But let's have a quick look at that and
  209. 7:25then we'll see what actually happened.
  210. 7:27Now, just so you know, I am by no means
  211. 7:29an AI expert and this is me learning as
  212. 7:31well. So, any other hints, tips you've
  213. 7:34got, please let me know.
  214. 7:37So, I started out by saying you're an
  215. 7:38autonomous systems agent. Your objective
  216. 7:40is to deploy a complete home lab stack
  217. 7:42on an existing Ubuntu machine running in
  218. 7:44Proxmox. So, I gave it the IP address,
  219. 7:47gave it the user, told it where the SSH
  220. 7:49key was, gave it some variables which is
  221. 7:51in a separate file, told it that my base
  222. 7:54domain will be Jim's Garage. And I also
  223. 7:56said that every single project I want
  224. 7:58you to spin up, you need to put it in
  225. 8:00sort of this home repository layout. And
  226. 8:03also I want to use bind mounts as well
  227. 8:06in this kind of format. I like that just
  228. 8:08because it keeps my Docker Compose and
  229. 8:10this script will actually go and build
  230. 8:12all of those Docker Compose files. And
  231. 8:14then it will also create a repository
  232. 8:16where all of the files for those
  233. 8:18services are stored. Not using volumes
  234. 8:20because they're a bit harder to
  235. 8:22manipulate, just with typical bind
  236. 8:23mounts so you can actually go in as a
  237. 8:25human and look at the configuration
  238. 8:27files and all the files that are needed
  239. 8:28for the container. I also told it that I
  240. 8:30want an internal Docker network of
  241. 8:32proxy, so you guessed it, yeah, I want a
  242. 8:34reverse proxy setting up here.
  243. 8:36So what it does is basically run all
  244. 8:38commands remotely over the SSH, so it
  245. 8:40connects that machine. It's going to
  246. 8:42read the variables file, which is down
  247. 8:44here. I won't show that because it's cut
  248. 8:46some secrets for things like pulling my
  249. 8:49valid SSL cert for Let's Encrypt.
  250. 8:52It takes the Vaultwarden user and the
  251. 8:53master key as well which I've specified
  252. 8:56and then generates all passwords for
  253. 8:58everything. So anytime it pulls down a
  254. 9:00container or spins up something like the
  255. 9:02traffic dashboard or it creates the web
  256. 9:05interface and oh, by the way, anything
  257. 9:07that supports OAuth, I've automatically
  258. 9:09told it that
  259. 9:10go and put that and integrate it into
  260. 9:12Authentic.
  261. 9:13It then stages all the credentials into
  262. 9:15a temporary file. It will then stick
  263. 9:17them all into Vaultwarden, stores a
  264. 9:20record of all of the Compose files
  265. 9:22and then basically stores that in
  266. 9:24Authentic. It will always use the latest
  267. 9:27versions of containers, always use
  268. 9:29official documentation and you can see
  269. 9:31further down I've created links where it
  270. 9:34shows you where the official
  271. 9:35documentation is. I do find that that
  272. 9:37helps it out a little bit. I want I then
  273. 9:40want it to add all the containers to
  274. 9:41homepage and then configure any widgets
  275. 9:43that are available. After that it
  276. 9:45basically goes through and does the host
  277. 9:46preparation. It will connect via SSH, it
  278. 9:49will disable anything. So this bit here
  279. 9:51I think was for configuring
  280. 9:55technician, so it turns off the resolve
  281. 9:57config.
  282. 9:58It installed the Docker engine if it was
  283. 9:59needed, created some shared credentials
  284. 10:02and then got onto installing
  285. 10:03Vaultwarden. So needed to install
  286. 10:05Vaultwarden first just because that's
  287. 10:07kind of where I was going to store all
  288. 10:08the credentials. We then set up traffic
  289. 10:11so that we can serve up every
  290. 10:12container's dashboard with valid SSLs.
  291. 10:15And I also want a crowd sec, which can
  292. 10:16often be a bit of fun to install, but
  293. 10:19actually using something like this, it
  294. 10:20does takes the strain out of it. And
  295. 10:22this is where I have that sort of
  296. 10:24dissonance with it because on the one
  297. 10:26hand, dead easy to go and set up a home
  298. 10:28lab,
  299. 10:29but at the same time I've got the
  300. 10:30benefit of I've done this a hundred
  301. 10:32times and know how it works. And I do
  302. 10:34worry that people will just use the AI
  303. 10:36and not actually understand and be able
  304. 10:37to debug things when things do go wrong.
  305. 10:40And for me personally, I think a lot of
  306. 10:41the value and the knowledge is from
  307. 10:42actually understanding what's going on
  308. 10:44the hood and using this more of a tool
  309. 10:46just to expedite things. That's my take
  310. 10:49on it anyway.
  311. 10:50It then set up a technician cluster and
  312. 10:52as I mentioned before, um that was two
  313. 10:54machines and all the records for every
  314. 10:56single dashboard and service I have will
  315. 10:58automatically be added via the CLI into
  316. 11:02technician itself.
  317. 11:03Then set up notifications and monitoring
  318. 11:05want go to five. I've got crowd sec and
  319. 11:07uptime Kuma, so any of those plugged in
  320. 11:09automatically will generate the
  321. 11:11notifications and it will automatically
  322. 11:13send it to my element server here
  323. 11:16whenever I get some initial issues.
  324. 11:19Whenever I get any issues. So here for
  325. 11:21example, something's gone down and it'll
  326. 11:22even say what do you want me to do on
  327. 11:24those when you open up these threads. So
  328. 11:26here you can see it's down. It's going
  329. 11:28to bring it back up and then it went and
  330. 11:29brought it back up, which was pretty
  331. 11:31awesome. I I spun that one down. We'll
  332. 11:33do a demo of that later. After that, I
  333. 11:34got into sort of the application suite.
  334. 11:36So you could basically modify this
  335. 11:38instructions.md file and just add
  336. 11:40whatever containers you want. So for
  337. 11:42each one, I've basically done it in this
  338. 11:44format here. So you got the service,
  339. 11:45you've got the documentation where it
  340. 11:47sits, the subdomain you want it on and
  341. 11:49any special requirements. So for
  342. 11:51example, for container, here's the docs.
  343. 11:53I want it sitting at portainer.
  344. 11:55You can mount the sock for it and also
  345. 11:58create an admin user and generate a
  346. 12:00password and that will then get stored
  347. 12:01into Vault Warden.
  348. 12:03I then spun up some additional servers
  349. 12:04down here and then said I want to
  350. 12:06basically talk to you, this Hermes agent
  351. 12:09through Matrix synapse. So it went away.
  352. 12:11It spun up Matrix itself. It has a API
  353. 12:14gateway or web gateway for Hermes
  354. 12:16naturally, so it can integrate into
  355. 12:18Matrix. That was really nice to see.
  356. 12:21And then lastly, basically all the
  357. 12:23secrets that it created during all of
  358. 12:26these applications, it then went and
  359. 12:28stored them from that temporary file
  360. 12:30using the Bitwarden CLI, logged into
  361. 12:32Vault Warden and stored them all, then
  362. 12:35verified that they all worked, which is
  363. 12:36what's awesome. I think one of the key
  364. 12:38values of this is every single time it
  365. 12:41does something, I always ask it to go
  366. 12:42and verify that it works. So, this is
  367. 12:45probably a bad example, but let's say
  368. 12:47for example at the moment I've actually
  369. 12:48forgotten my key for Vault Warden. I can
  370. 12:51ask it what's
  371. 12:54my
  372. 12:55login for
  373. 12:57Vault Warden?
  374. 12:59It should even be able to understand my
  375. 13:01terrible typing.
  376. 13:05So, here you see it read the Vault
  377. 13:06Warden skill for Vault Warden access
  378. 13:08because I've asked this a few times and
  379. 13:10it's recorded it as a skill. So, it
  380. 13:12should be able to come back now, check
  381. 13:14that out using the skill, and here you
  382. 13:16go, it's admin at Jim's Garage, and then
  383. 13:18I've got the password beneath it. So, I
  384. 13:21can copy this now. We'll log into Vault
  385. 13:23Warden up here.
  386. 13:33And now that's unlocked, you can see all
  387. 13:35of the services that I've got running in
  388. 13:36here, and also all of the things like
  389. 13:38API keys, etc., which aren't stored as
  390. 13:41logins.
  391. 13:42So, now for example, what can we do?
  392. 13:44Well, we can do things like Portainer.
  393. 13:46And as I mentioned before, because I've
  394. 13:48got authentic set up, I can actually log
  395. 13:50into Portainer with authentic. And all
  396. 13:52of this again was done just through the
  397. 13:55LLM. So, if I log in with OAuth, it will
  398. 13:57forward me to authentic. We now need to
  399. 14:00go in with authentic. We'll log in.
  400. 14:03And then hopefully it should return me
  401. 14:05back to Portainer and log me in as that
  402. 14:07user. And then yeah,
  403. 14:09here's the 46 containers I've actually
  404. 14:11got running on this machine.
  405. 14:14So, that was pretty awesome. All these
  406. 14:17are up and running. I've even got things
  407. 14:18like posters for social media and stuff.
  408. 14:21I've got Restic. Restic is automatically
  409. 14:23configured as well. So, when I was
  410. 14:24deploying this container, it asks, do
  411. 14:27you want me to just set this up for
  412. 14:28regular backups? So, I said yes. And
  413. 14:30then when I deployed our clone, it said,
  414. 14:33do you want to automatically map the
  415. 14:35Restic volume to backup? I said yes. I
  416. 14:38haven't actually gone through and linked
  417. 14:39it to anything yet, but I've got the
  418. 14:41capability in the configuration there.
  419. 14:43Now, hopping onto the Docker VM as well
  420. 14:45here. So, this is the one it's actually
  421. 14:47deploying all of those containers too.
  422. 14:49You can see here I've got the Docker
  423. 14:50folder, which stores all of the files
  424. 14:52for the containers. And then we've got
  425. 14:54the Docker compose here as well. So, it
  426. 14:56actually creates all the compose files
  427. 14:58as well. Goes and pulls them down,
  428. 14:59modifies them, uses best practices like
  429. 15:02using variables here for all of the
  430. 15:05values, which are then mapped to a .env
  431. 15:07file for security, which is great. And
  432. 15:09it even managed things like Jellyfin.
  433. 15:11So, I told it for Jellyfin,
  434. 15:14go and load um the GPU into this so that
  435. 15:18you can do GPU transcoding. And it did
  436. 15:20that, and it works, and it was dead
  437. 15:22straightforward. It's even put sensible
  438. 15:24things like putting all the memory
  439. 15:25limits onto the containers, which again
  440. 15:27is another good practice. And it also
  441. 15:29put everything behind that reverse proxy
  442. 15:31that I set up within Traffic.
  443. 15:33It was just amazing to see that. Now,
  444. 15:35going into something like NA10, here are
  445. 15:37three of the workflows I've asked it to
  446. 15:39create.
  447. 15:40So, instead of it having to now rely on
  448. 15:42a skill, I've got it to deploy
  449. 15:44containers using these workflows, which
  450. 15:47is great because as you can see, it will
  451. 15:48deploy webhook, it will have an import
  452. 15:51deploy request, it will validate it, it
  453. 15:53will then write the compose file, create
  454. 15:55the records here for the DNS, it will
  455. 15:58then have a failure condition, it will
  456. 15:59store the vault, and then it will return
  457. 16:01a success of true.
  458. 16:03Verifying the deployment starting the
  459. 16:04stack. You can see now the value of
  460. 16:08having this level of automation baked
  461. 16:10into an LLM.
  462. 16:11So, now when I send it a prompt to spin
  463. 16:12up a new container, instead of having to
  464. 16:14work out each time, how do I spin up a
  465. 16:17Docker container? Because the first few
  466. 16:19cranks of the handle, it would actually
  467. 16:20have to figure out how do I deploy a
  468. 16:22container, say write a compose file,
  469. 16:24what's a compose file?
  470. 16:26Now it knows exactly what those things
  471. 16:27are. So instead of working it out, it
  472. 16:29just points straight to this automation,
  473. 16:31which it built itself in NA 10. And what
  474. 16:34that means is much more consistent
  475. 16:36output because it's all written here in
  476. 16:38code. You can see what it's actually
  477. 16:39doing here.
  478. 16:41And it also means that there's far fewer
  479. 16:42tokens being consumed because once it
  480. 16:44comes into NA 10, it's not actually
  481. 16:46having to do any work. It's just saying
  482. 16:48kick this process off and wait for the
  483. 16:50output. It either is a success or it's a
  484. 16:52failure, and then it will come and ask
  485. 16:54me what to do. We'll show an example of
  486. 16:56that in a minute.
  487. 16:57I've also got some other automations
  488. 16:59which were Vault Warden getting items.
  489. 17:01Again, what does it need to do? So it
  490. 17:03needs to go here and it's got the query
  491. 17:05it needs to run to actually get that. So
  492. 17:07again, before I had this in here, and
  493. 17:10you could do this in a skill as well of
  494. 17:11course,
  495. 17:13each time it was having to manually
  496. 17:14figure out well, how do I get a Vault
  497. 17:16Warden item? And it was having to read
  498. 17:18through the documentation, burn a ton of
  499. 17:20tokens before it got to the answer.
  500. 17:22And then finally I had this run an SSH
  501. 17:25command on here. So it basically says
  502. 17:27check the secrets, the secret okay, run
  503. 17:29it as either denied or it worked, and
  504. 17:31then you could run on the Docker VM. So
  505. 17:33again, a simple way for it to connect to
  506. 17:36a host, use a certificate, and then get
  507. 17:38access to it. Again, the more things I
  508. 17:41can get as skills through NA 10, the
  509. 17:43better because it gives me a more
  510. 17:45consistent output. So before I actually
  511. 17:47go and start letting this thing loose,
  512. 17:50the beautiful thing is now I've actually
  513. 17:52got things like these NA 10 automations.
  514. 17:55I've also got the skills that are within
  515. 17:57my Hermes agent that I don't need to
  516. 17:59start fresh. I've got the initial
  517. 18:01prompt. I've got all of the skills it
  518. 18:02can use. I'll just need to update things
  519. 18:04like the IP addresses etc., which I can
  520. 18:07probably change the hostnames as well,
  521. 18:08which would make a lot more sense. I can
  522. 18:10also bring over all these NA 10
  523. 18:12automations and just import those so
  524. 18:15that again, the first few cranks of the
  525. 18:17handle of spinning this up, it should be
  526. 18:18able to one shot it, which it pretty
  527. 18:20much the first time. Here you can see
  528. 18:22with Technetium as well, it was able to
  529. 18:24basically create a two-node cluster
  530. 18:26straight away. And if you go to the
  531. 18:28administration and the cluster, you'll
  532. 18:30see these are both set up, and they're
  533. 18:31also set up with my cluster domain, so
  534. 18:34my actual domain name. It even created
  535. 18:36these on different IP addresses. Sure,
  536. 18:38at the moment they're on the same host,
  537. 18:40which doesn't work obviously for
  538. 18:41redundancy purposes, but it's only a
  539. 18:44testing bed. But it sorted all of that
  540. 18:46out automatically and automatically
  541. 18:48clustered everything together. So I've
  542. 18:50got all of the zones and then all of the
  543. 18:54uh DNS records all in here, and they're
  544. 18:56all replicated across both of these
  545. 18:58servers, which is awesome. So even
  546. 19:01things like Myriota Talk, which can be a
  547. 19:03bit of fun to set up, those are all
  548. 19:05working as well. I have a video on
  549. 19:07Myriota Talk, and if you've never used
  550. 19:08it before, definitely go and recommend
  551. 19:10it. It's literally this simple to just
  552. 19:12go and spin up a room where you can do
  553. 19:14VoIP chats, where you can do voice
  554. 19:15chats, you can share files, you can do
  555. 19:18all sorts. Obviously not going to work
  556. 19:19very easily in a VM. So yeah, that's
  557. 19:22been pretty awesome to spin up. Here you
  558. 19:24can see some of the token usage. This
  559. 19:26isn't quite right because this is
  560. 19:27running on the uh virtual machine, so
  561. 19:29it's actually about more like 20 million
  562. 19:31tokens I've been using this week. Um the
  563. 19:34other server's running on my machine I'm
  564. 19:36recording this on, and that's been up
  565. 19:38far longer doing that recording.
  566. 19:40But this is a cool dashboard as well.
  567. 19:42Definitely go check this out if you've
  568. 19:43got a Strix Halo device.
  569. 19:45Um but let's have a bit of fun now, and
  570. 19:47I'll show you some real-world examples
  571. 19:49of using Hermes to basically manage this
  572. 19:52thing. So as I mentioned at the start of
  573. 19:53this video, I've been having um a bit of
  574. 19:56an issue with that container D
  575. 19:58because the version that comes on this
  576. 20:00machine is causing it to crash with so
  577. 20:02many containers on here. So if I ask it
  578. 20:05what's causing
  579. 20:07the
  580. 20:11Docker VM to crash,
  581. 20:14the beauty is now it will go away and
  582. 20:16check that machine. It will check all of
  583. 20:18the logs for me, all of the logs for all
  584. 20:20of the containers, etc. And hopefully,
  585. 20:22this will be real world, Don't let me
  586. 20:24down. It will come back with that issue
  587. 20:26around container D.
  588. 20:28Now, it will offer as well to go and fix
  589. 20:30that for me.
  590. 20:31I'm choosing not to do that just because
  591. 20:33I don't want to break the APT repository
  592. 20:35at the moment. But here, let's have a
  593. 20:37look. So crash one, it found out on
  594. 20:39November 6th, crash two on the 7th.
  595. 20:42And it's saying again it's because of
  596. 20:44container D.
  597. 20:45Because of container D 2.2 package.
  598. 20:48Um
  599. 20:50it has an out of memory killed app
  600. 20:52process and that again is that
  601. 20:54deadlocked in the run C shim command,
  602. 20:57which is a well-known bug.
  603. 21:00Quite surprised that it hasn't actually
  604. 21:01been patched yet and one of the reasons
  605. 21:03if I'm going to go down this route, I'm
  606. 21:05probably going to go with something like
  607. 21:06Fedora. Sure, it's not an LTS build, but
  608. 21:09it'll hopefully be able to diagnose and
  609. 21:11fix some of these problems and I don't
  610. 21:12mind having a bit more of a risky home
  611. 21:14lab if I can go and fix things quickly.
  612. 21:16So now then, let's have a little bit of
  613. 21:18fun and let's set something up like I
  614. 21:20don't know, let's set up Keycloak. I
  615. 21:22don't want it to be automatically
  616. 21:23configured. I want it to automatically
  617. 21:25basically follow all of the things I've
  618. 21:28got here. So I want it behind a proxy. I
  619. 21:30want all of the credentials being saved
  620. 21:32into Vault Warden.
  621. 21:34Let's see how it gets on. I'm going to
  622. 21:35write the following prompt. So I'm going
  623. 21:37to keep this one pretty simple because
  624. 21:39I've got to the position now where most
  625. 21:42of the things I want to use are either
  626. 21:43skills or automations. So I want you to
  627. 21:46deploy Keycloak using the official
  628. 21:48documentation and then I've linked it.
  629. 21:50It's obviously got Docker in it, so it
  630. 21:52should know it's a Docker uh deployment
  631. 21:54I want. I want it to follow the
  632. 21:56deployment instructions in the
  633. 21:57instructions.md.
  634. 21:59So that's one I mentioned earlier.
  635. 22:01So it should go away now and start to
  636. 22:04work on doing that. First thing it will
  637. 22:06check for that instructions md, which it
  638. 22:08found. It's now going to the website for
  639. 22:10Keycloak that I linked it and it's
  640. 22:12reading through those instructions.
  641. 22:14Here you can see everything that's in
  642. 22:15the instructions md, so all of those IP
  643. 22:18addresses, all of the things that we
  644. 22:19saw. It'll understand exactly how it's
  645. 22:22going to deploy this, all of the things
  646. 22:24it needs to have in so DNS records, it
  647. 22:27needs to tie in with authentic, if it
  648. 22:28can use it, those sorts of things. It's
  649. 22:31got the skill for the Docker VM, and
  650. 22:33it's also got the skill for the Docker
  651. 22:34stack. So it basically knows how to
  652. 22:36create compose files, it knows where to
  653. 22:38go. And hopefully, you can see here it's
  654. 22:40generating some passwords. It should
  655. 22:42also then follow the NA 10 deployment,
  656. 22:45and we'll be able to check NA 10 if it
  657. 22:47actually used it because it keeps a
  658. 22:48record of all the times it's been
  659. 22:49executed.
  660. 22:51Here you can see the plan it's going to
  661. 22:52follow, so check the VM state, create
  662. 22:54the password, technician record, Docker
  663. 22:56compose up, create the realm, then
  664. 22:58create widgets for homepage, etc., and
  665. 23:01then validate that everything is up and
  666. 23:03running. Here you can see the traffic
  667. 23:04labels it's creating. You can see the
  668. 23:06secrets it's creating. Obviously,
  669. 23:09security is a big concern
  670. 23:11when you're running things like a large
  671. 23:12language model locally, especially when
  672. 23:14you're giving it things like internet
  673. 23:16access. And this is kind of where I want
  674. 23:18to get more control around it and why I
  675. 23:21haven't just let this thing loose in my
  676. 23:23current home lab. My current home lab is
  677. 23:25where I keep all of my stuff, right? So
  678. 23:28I need to make sure that it's secure.
  679. 23:30And all of this is going to be that test
  680. 23:31bed. And I'm going to do some subsequent
  681. 23:33videos where we talk about things like
  682. 23:34the guard rails, some absolute
  683. 23:37must-haves, and some absolute no, you're
  684. 23:39not touching that.
  685. 23:40And in future, what I'll probably do is
  686. 23:43restrict access to Vault Warden, but
  687. 23:45whilst it's setting up and I've not got
  688. 23:47actually any useful data in there, it
  689. 23:49can have access and set the home lab up.
  690. 23:51In future, I'll have it ask me for the
  691. 23:53password so it can't just go and access
  692. 23:55it anytime it wants. Here you can see
  693. 23:56it's generating the compose file, which
  694. 23:58it's going to go and deploy.
  695. 24:00And you can see on the bottom some
  696. 24:01statistics. So currently we've used
  697. 24:03about 40,000
  698. 24:05tokens out of a context window of 262 K.
  699. 24:09It's generating at around 56 tokens per
  700. 24:12second at the moment, and the whole
  701. 24:13request has taken about two and a half
  702. 24:15minutes.
  703. 24:16So if you think how long it takes for
  704. 24:17you to probably follow one of my videos,
  705. 24:20and certainly when I deployed this for
  706. 24:21the first few times,
  707. 24:23it was probably around half an hour to
  708. 24:25an hour to get every single thing
  709. 24:27configured. Now we're in a position
  710. 24:29where we can just point it to some
  711. 24:30instructions, say how we want it
  712. 24:32configured and then let it work it all
  713. 24:34out. Now the first time I ran uh this
  714. 24:37prompt and set up some of these uh
  715. 24:39containers, it was actually asking me
  716. 24:41quite a few questions. But because I've
  717. 24:43got this thing dialed in and the more
  718. 24:45you use it, the more it learns your
  719. 24:46behavior,
  720. 24:48it basically knows what I'm going to say
  721. 24:49already and it saves those as skills. So
  722. 24:52when it says things like, oh can I
  723. 24:53connect to that machine? It already
  724. 24:55knows. Can I generate a password? Sure
  725. 24:56you can. Can I store that? Yes, you can
  726. 24:59because I've explicitly told you to go
  727. 25:00and do that.
  728. 25:02Now you can see it's generated the
  729. 25:04config file with all of the added
  730. 25:05variables put in. And it is quite
  731. 25:07fascinating seeing how it talks to
  732. 25:09itself and works things out.
  733. 25:11I haven't actually had an issue where
  734. 25:13it's got into a loop. I have had that
  735. 25:15before when I've used um older versions
  736. 25:17of Kwen, things like 3.5, 3.6, whereby
  737. 25:22it would start to hallucinate and get
  738. 25:23stuck in a loop. Actually, 3.8 Flash
  739. 25:25next. I don't think I've had it loop
  740. 25:27once. It has kind of burned through
  741. 25:29tokens unnecessarily at times,
  742. 25:32but that's usually down to my fault and
  743. 25:34me not being explicit or asking it to
  744. 25:36ask me questions when it's unsure, which
  745. 25:38is absolutely something you can do. And
  746. 25:40we'll touch on some better prompting in
  747. 25:41another video. Here you can see it's now
  748. 25:43created that DNS record and it will be
  749. 25:46logging in and doing that. It's now
  750. 25:48checking whether it can make a widget
  751. 25:50within homepage as well for Keycloak.
  752. 25:53I'll edit this little bit out because it
  753. 25:54is sleeping for 75 seconds. It's just
  754. 25:57making sure that it's got time to spin
  755. 25:59up the container and then make sure it's
  756. 26:01before it does that readiness check. So
  757. 26:03now that time's passed and hopefully
  758. 26:05everything was up and running. The good
  759. 26:07news is if it isn't up and running,
  760. 26:09it'll often find out why and it will
  761. 26:11check all of the containers and make
  762. 26:12sure everything's good.
  763. 26:14Says the container started. Keycloak
  764. 26:16still booting. Says the first boot takes
  765. 26:18a minute, so it's going to wait again
  766. 26:19and recheck. So again, I'll edit this
  767. 26:21forward. This is not the LLM being slow.
  768. 26:24This is just it giving it time to boot
  769. 26:26up as it recommends by the container.
  770. 26:28Now we go. It has found it, it has
  771. 26:30waited and it's now checked that it's
  772. 26:31listening on port 8080. Interestingly as
  773. 26:34well, if the things like a port clash,
  774. 26:37it'll often figure that out and manually
  775. 26:39change and keep a record of what ports
  776. 26:41it's put on. Obviously that's less of an
  777. 26:43issue if you're running something like a
  778. 26:45proxy because everything will just run
  779. 26:47through the proxy anyway. Seems to have
  780. 26:49found a problem with the healthy state
  781. 26:51returning a 404. So it will probably go
  782. 26:53away now and check the script and find
  783. 26:55out what happens. Well, that was because
  784. 26:57the health point is disabled by default,
  785. 26:59so the script died early. Here you can
  786. 27:01see also it's having a bit of a conflict
  787. 27:04with some of the versions. So I think it
  788. 27:05was originally quoting version 24, but
  789. 27:08now it's saying 26.8. So sometimes it
  790. 27:11can get fused based upon what the LLM
  791. 27:13was trained around, but that's why I
  792. 27:15feed it the latest documents, so it
  793. 27:17should go away and do those.
  794. 27:19It did the Keycloak admin login and that
  795. 27:21succeeded. And now it's found an issue
  796. 27:24with the client's
  797. 27:26account that's being created. Now I
  798. 27:28can't actually remember off the top of
  799. 27:30my head how you actually do that, but it
  800. 27:32might be quite complicated for it to go
  801. 27:34and work that out. You might be better
  802. 27:36off sometimes just going in through the
  803. 27:38traditional way through the web GUI and
  804. 27:41doing it that way.
  805. 27:44But that's the beauty of this. Usually
  806. 27:45when I've just left this to work, it
  807. 27:47will basically go and figure it all out
  808. 27:50and it will also give you a dump at the
  809. 27:51end as we'll see of exactly the stages
  810. 27:53that it followed. So you can go through
  811. 27:55and check and make sure it didn't do
  812. 27:56anything stupid. Now it's doing a
  813. 27:58headless check using a browser to
  814. 28:00actually go to the website and log in
  815. 28:01with the new credentials.
  816. 28:03Again, that's one of the awesome
  817. 28:04features, all of the tools that the
  818. 28:06Hermes agents come with. It's able to
  819. 28:08take screenshots of things as well that
  820. 28:10it's changed and I've been building some
  821. 28:12five coded apps with this and it will
  822. 28:14even take pictures of it and just make
  823. 28:16sure that what you're saying in words is
  824. 28:18actually then replicated on screen. So
  825. 28:20it's now working out the flow using the
  826. 28:23web interface and then it will be
  827. 28:24changing the script so that it's got a
  828. 28:26script which basically validates it via
  829. 28:29code instead of having to go through the
  830. 28:31web user interface. So that's great for
  831. 28:33when you wanted do debugging checking
  832. 28:35things. It can do that
  833. 28:37it can do that via a script as opposed
  834. 28:38to going through the web front end. So
  835. 28:40as all of this is going on, we can be a
  836. 28:42nerd as well and we can check the
  837. 28:44dashboard for the machine itself.
  838. 28:46You can see here all the tokens that are
  839. 28:48being generated, which is pretty
  840. 28:50awesome. And if we want to go to the
  841. 28:52Grafana dashboard, we can see sort of
  842. 28:54the CPU and RAM usage, disk usage and
  843. 28:56what's going on there. Now it's included
  844. 28:58the home page card and the admin flow
  845. 29:01has failed, so it's going to double
  846. 29:02check what happened there. Home page and
  847. 29:04uptime Kuma are now done as well. So it
  848. 29:07looks like the issue was with traffic
  849. 29:08and not using the X Forwarded protocol.
  850. 29:11So hopefully it's going to go away now
  851. 29:12and reconfigure that, redeploy the
  852. 29:14container so that it passes on the HTTPS
  853. 29:17traffic and fingers crossed that should
  854. 29:18now work. So apparently after 70 minutes
  855. 29:21of which a few of those minutes were
  856. 29:23actually it's just waiting for checks.
  857. 29:26It's stating that Keycloak is deployed
  858. 29:28and it's been verified. So what did it
  859. 29:30do? It deployed 26.8.2 which I think is
  860. 29:33the latest version.
  861. 29:35It created the compose files as I said.
  862. 29:37It set the data directory where I wanted
  863. 29:39it. Put it on the traffic network, which
  864. 29:41is all good. It also integrated it with
  865. 29:43CrowdSec, so anything going through
  866. 29:44there is going to be scanned by CrowdSec
  867. 29:46for any security vulnerabilities.
  868. 29:49Followed the instructions, the document
  869. 29:52flow.
  870. 29:53It also then
  871. 29:55generated the passwords. It then created
  872. 29:57all of the accounts and verified that
  873. 29:59they worked and fingers crossed added it
  874. 30:02all to the necessary containers, things
  875. 30:04like the home page, uptime Kuma, etc. So
  876. 30:07let's now have a little look and see if
  877. 30:09that all went to plan. So back on this
  878. 30:11machine, let's click on home page and
  879. 30:13what do we get? So we've got Keycloak
  880. 30:16down here. So we should be able to click
  881. 30:17on that.
  882. 30:19Um this looks promising as it
  883. 30:21automatically added it here so that I
  884. 30:22can click it. Yep, there's something
  885. 30:24here. Let's sign in.
  886. 30:28Temporary admin user. I'll probably need
  887. 30:30to go and fix that.
  888. 30:32I don't know what a temporary admin is.
  889. 30:34Forgotten how to use Keycloak.
  890. 30:36Um but sure we can just basically save
  891. 30:38it. But
  892. 30:39the long and short of it is this is all
  893. 30:41up and running, and all that was was a
  894. 30:45single prompt with a few skills that
  895. 30:46I've tweaked over a few iterations.
  896. 30:49And also things like Ollama. Where is it
  897. 30:53in here?
  898. 30:57Here you can see Keycloak up and
  899. 30:58running, and it's automatically if I go
  900. 31:01to the edit it should have the
  901. 31:02notifications. Yeah, it's got the
  902. 31:03notifications to go to fire a matrix
  903. 31:05already set up, which is awesome. And I
  904. 31:08guess things like Grafana should
  905. 31:11automatically pick it up as well. Yep,
  906. 31:13here you can see that Keycloak has been
  907. 31:15picked up. So it's automatically looking
  908. 31:17at that as well. Probably doing that
  909. 31:19more just through the Docker sock than
  910. 31:21any formal sort of tweaks to any config
  911. 31:23files.
  912. 31:24It's obviously got the records added
  913. 31:27into technician, which is awesome. And
  914. 31:29we can also log into Portainer if we
  915. 31:31wanted to see it on here already spun
  916. 31:33up. So here it is Keycloak as well. And
  917. 31:35then this will behave just like any old
  918. 31:37sort of Portainer instance, which was
  919. 31:39also deployed by this script as well.
  920. 31:43Which again just really shows the power
  921. 31:45of this.
  922. 31:46So where do we end? Let's wrap up.
  923. 31:49There's lots of things we need to talk
  924. 31:50about here. Now as I said at the
  925. 31:51beginning, this is going to form the
  926. 31:53start of kind of this AI Home Lab
  927. 31:56journey. And that was just to sort of
  928. 31:58wet your whistle.
  929. 31:59Now don't get me wrong. I absolutely
  930. 32:02love doing all of the tutorials and
  931. 32:03setting things up by hand, reviewing all
  932. 32:06of the code and setting all the
  933. 32:07configurations, and that's not going to
  934. 32:08change.
  935. 32:10What might change is the way in which
  936. 32:12you deploy things with those
  937. 32:13configurations. So you might want to
  938. 32:15watch the video and understand what's
  939. 32:17going on in the hood, but you don't
  940. 32:18actually want to spend all of that time
  941. 32:20configuring it yourself, pressing
  942. 32:22keystrokes, when all you need to do is
  943. 32:23send it to an LLM with a config file and
  944. 32:25give it the instructions that you want.
  945. 32:28Now from an efficiency standpoint,
  946. 32:30replicability, all of that sort of
  947. 32:32stuff, security, things like having it
  948. 32:34in code via things like Ansible and
  949. 32:36Terraform are definitely going to be
  950. 32:39better in terms of guaranteed outputs,
  951. 32:41and you have full control over it.
  952. 32:43There's basically no AI. There's no sort
  953. 32:45of unknown variable in the middle.
  954. 32:48But I think we're soon going to get to a
  955. 32:49place where most of this stuff is going
  956. 32:51to be unavoidable. And especially for
  957. 32:54diagnosing and fault tolerances and
  958. 32:56monitoring and all of that, I must say
  959. 32:58having this integrated into Element,
  960. 33:00having that on my mobile phone and being
  961. 33:02able to remotely manage my home lab on
  962. 33:05the fly or at the moment this mock home
  963. 33:07lab
  964. 33:09has been amazing. It's honestly when
  965. 33:11things used to break and I'd be away at
  966. 33:12work or away from home, it would often
  967. 33:15be hours of trying to get into a virtual
  968. 33:17machine or over the VPN and all that
  969. 33:19stuff and try and diagnose it on small
  970. 33:21laptop screens or phones and it was just
  971. 33:22an absolute nightmare.
  972. 33:24I must say when I've tried to break
  973. 33:25containers and shut them down, it's
  974. 33:28automatically worked, notified me and
  975. 33:29come up with a recommendation. So simply
  976. 33:32spin it back up. Or if there is a more
  977. 33:35sort of difficult fault, it will check
  978. 33:36all of the container logs, it will check
  979. 33:38the host logs and do what it recommends,
  980. 33:40maybe upgrade the container, change the
  981. 33:42RAM, up the CPU core counts, all of
  982. 33:44those sorts of things.
  983. 33:47And that capability is only going to get
  984. 33:48better when I give it more control.
  985. 33:51Potentially, we'll see how that goes.
  986. 33:53Things like integrating it with Proxmox
  987. 33:55will allow it to control the Proxmox
  988. 33:57host. Things like allowing it to have
  989. 33:59access to my NAS will be enable it to
  990. 34:01set up shares and all of those sorts of
  991. 34:03things and potentially even one day
  992. 34:05maybe set up things like the firewall so
  993. 34:07I can set up some of that routing.
  994. 34:09Now all of that is going to come in this
  995. 34:11series. I don't know exactly what the
  996. 34:13flavor of that will be, but there's also
  997. 34:14going to be some obvious things like
  998. 34:16spinning up a Kubernetes cluster, maybe
  999. 34:18things like a Docker Swarm. And I'll be
  1000. 34:20documenting all of that because I'm
  1001. 34:21really keen to see how it goes.
  1002. 34:24Anyway, let me know if this is something
  1003. 34:26that you're interested in. Let me know
  1004. 34:27if you've already integrated LLMs into
  1005. 34:29your home lab and how that's going. What
  1006. 34:31are the pitfalls? What can you share?
  1007. 34:33What are some of the things that I need
  1008. 34:34to avoid?
  1009. 34:36But anyway, if you've liked this video,
  1010. 34:37please give it a thumbs up, hit that
  1011. 34:39subscribe. I'll see you on the next one.
  1012. 34:41Take care, everybody.

About this transcript

This page contains the full transcript of Is this the future of homelabbing? by Jim's Garage, generated from the public captions YouTube serves with the video. The transcript has 7,068 words across 1,012 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.