Is this the future of homelabbing? — Transcript
Full transcript
- 0:00Hey everybody and welcome back to Jim's
- 0:02Quen's garage. Today I'm going to be
- 0:04showing you about a little project I've
- 0:06had up and running for a couple of weeks
- 0:08and been following my socials, you'll
- 0:10know what I'm talking about. I'm slowly
- 0:12building up a new home lab, but there's
- 0:14one catch. I can only build it with AI.
- 0:17And to be honest, I suspect this is
- 0:19going to form a larger part of my home
- 0:21lab in the future than I ever suspected.
- 0:24That's because I've been able to build
- 0:25all of this just using Quen 3.8 Flash
- 0:28Next on a Strix Halo. And this will kick
- 0:31start a new series where I'll be looking
- 0:33at how I can build out my entire home
- 0:34lab and have it all managed and
- 0:36monitored by AI. And I'll also be
- 0:39exploring different levels of hardware
- 0:41we can use because I appreciate that
- 0:43this stuff isn't affordable for
- 0:44everyone. It wouldn't be affordable for
- 0:46me if I didn't receive some of the
- 0:47sponsored review units. So first, let's
- 0:50have a look at the setup that I've got
- 0:51and then we'll dive into what it's
- 0:53actually managed to configure, which has
- 0:54totally blown my mind. I'm literally
- 0:57saving hours per week on managing my
- 0:59home lab and I want to share that with
- 1:01you. So just to hook you in before we
- 1:03dive into that, this is an example of
- 1:06what I've built here. This is a full
- 1:08Docker virtual machine I've got running
- 1:10in Proxmox. Everything you see here, all
- 1:12of these containers and there are more,
- 1:14just some of them don't have a homepage
- 1:16integration was all set up using a LLM.
- 1:20Now you'll even notice that things like
- 1:22the certificates, all of those are
- 1:24working. This is a valid cert that's
- 1:25been pulled down from Let's Encrypt.
- 1:28I've got things like Home Assistant set
- 1:29up. I've got Paperless in there. I've
- 1:32got Audiobook Shelf, Netbird. I can even
- 1:34monitor the LLM itself as well. I 've
- 1:37got Uptime Kuma to report everything.
- 1:40All of that is then configured to run to
- 1:42Gotify so I get real-time messaging
- 1:44alerting of what's going on. I have
- 1:46Vaultwarden where it automatically every
- 1:48time I spin up a container, it
- 1:50automatically stores all of the
- 1:51credentials in Vaultwarden and only
- 1:53accesses all of the APIs and command
- 1:55lines via credentials stored in
- 1:57Vaultwarden. I 've also got this handy
- 1:59Grafana which it created itself to
- 2:01monitor itself. And I've even got NA10
- 2:05workflows, which I'll come on to later,
- 2:07which it created because it used Hermes
- 2:09agent skills and then transferred them
- 2:11over into automations. All of that's
- 2:13great because it gives me consistent
- 2:15workflows and it also saves on token
- 2:17generation.
- 2:19Now don't worry, we're going to come
- 2:20back to all of this if that all sounds a
- 2:22bit foreign at the moment. So how have I
- 2:24got this set up? Well, here you can see
- 2:25my Minisforum MSS1 Max and this is a
- 2:28Tricks Halo device that was sent to me
- 2:30for review about a year ago. And this
- 2:32thing has been absolutely awesome and
- 2:34I've only just recently dusted it off
- 2:36because I'm curious to get more involved
- 2:38with AI. And to be honest, the local
- 2:40models that you can now run are really
- 2:43catching up. This thing is about Opus
- 2:454.6 level, so it's totally usable within
- 2:48a home lab. And I've generated now about
- 2:5120 million odd tokens a week. So the
- 2:53return on investment if you bought this,
- 2:55you're probably looking at low order
- 2:57number of weeks compared to something
- 2:59like say Claude. Sure, don't get me
- 3:01wrong, something like Opus 5.5 or any of
- 3:04the Fable Mythos models, they are
- 3:06definitely better, more capable than
- 3:07this. But as I'm going to show you in
- 3:09this video, the local models are
- 3:12surprisingly advanced and what they can
- 3:14do is still just blowing my mind. Oh,
- 3:16and did I mention that all of this I can
- 3:18actually talk to the LLM through
- 3:20something like Matrix? There's a ton of
- 3:22other gateways you can use, but I can
- 3:24basically talk to this thing on my
- 3:26phone. I can ask it say, what's going on
- 3:28in my home lab? Is everything running?
- 3:30And it will come back and say yes or no.
- 3:32And if it does find a problem, it will
- 3:34present a fix for it. So here you can
- 3:36see I ask it, you see the eyeballs come
- 3:38up. It will go away now, it will query
- 3:40that Docker virtual machine and it will
- 3:42come back with an answer of what's
- 3:43happening on there. One thing I found
- 3:45even more amazing is the fact that I
- 3:47actually found a root cause of why this
- 3:49machine was crashing. I was pulling my
- 3:51hair out thinking what on earth has gone
- 3:53wrong here. And it turns out that the
- 3:55version of Ubuntu I'm using 2604 runs
- 3:58container D 2.2.2 which has a shim
- 4:01deadlock issue which is a known bug. And
- 4:04the only way to fix that is to either
- 4:05upgrade manually or wait for it to be
- 4:08available via APT.
- 4:10So that is awesome that it can debug and
- 4:12save me hours that I would have to spend
- 4:14searching forums, etc. Now there are
- 4:16many inference engines that you can run,
- 4:18but the one I'm running is called
- 4:20Halogen Server. Now I've been following
- 4:22Halogen Server now for quite a few
- 4:24weeks. Um there's always a stream of
- 4:26updates. Unfortunately, not all of it is
- 4:29open source. Still some of the weights
- 4:31for the models are locked. I hope that
- 4:34that does change in the near future, but
- 4:36for now I'm pretty happy with using it.
- 4:39To get this up and running was pretty
- 4:41straightforward. I simply spun up a
- 4:43Fedora 44 server and then run the
- 4:46command down here, which is pretty
- 4:48straightforward. Just a quick stop.
- 4:51There are a ton of options for this
- 4:53Halogen Server, so do go and check those
- 4:56out. It even includes being able to run
- 4:58smaller models on the NPU, which is
- 5:00pretty epic and takes this to a whole
- 5:02new level. I'll do a deep dive on this
- 5:05in the future, but for now the key stats
- 5:07that you're looking at are really up to
- 5:09about 94 tokens per second. On average I
- 5:12see more around sort of the low 70s and
- 5:15around 1800/1740-ish
- 5:19for prefill. For anyone who doesn't know
- 5:21what that means, the decode is kind of
- 5:22the output and the words you see. The
- 5:24prefill is basically working out what it
- 5:26is that you want it to do, i.e. the
- 5:27prompt you put it. Now all of this runs
- 5:30through a virtual machine that I
- 5:31created. And as I mentioned at the
- 5:34beginning, this is a mockup of a home
- 5:36lab because I really want to test this
- 5:37thing out, make sure I iron out all the
- 5:39wrinkles before I set it free on my
- 5:41actual home lab. And even then I'm
- 5:43probably going to take a staged approach
- 5:45where I roll containers over slowly just
- 5:47to make sure that it doesn't go crazy.
- 5:50Now the way I get this working is I run
- 5:52Hermes agent. So Hermes agent is
- 5:56basically a
- 5:58harness that you can use, and the key
- 6:00advantage of something like this is the
- 6:03fact that it comes with what called
- 6:05skills. So here, for example, in Hermes,
- 6:07you can actually see skills. Now, what
- 6:10skills are are basically a little bit
- 6:12like automations, but they're basically
- 6:15repositories of knowledge that the agent
- 6:18that the harness can use. So, for
- 6:20example, whenever I connect to this home
- 6:22lab Docker stack here, you'll see it's
- 6:24created a skill where it goes through
- 6:26and it basically says
- 6:28all the things that I've been asking it
- 6:29to do. It creates a memory of that. So,
- 6:32obviously with an LLM, it tries to work
- 6:34things out every time because that's
- 6:37just the nature of how they work. But
- 6:39here is basically a
- 6:41standard operating procedure for many of
- 6:43the things I commonly ask. So, when I
- 6:45tell it to create a container,
- 6:47it knows that firstly it needs to go and
- 6:49find the SSH key to connect. It keeps a
- 6:51record of where that is within these
- 6:53skills. And it does that for pretty much
- 6:55everything. When I ask it to perform
- 6:57updates of containers, it knows how to
- 6:59do that. We'll come onto that more
- 7:00later. And I've also, for value, created
- 7:03some NA10 workflows because they give me
- 7:06fixed automations where I have much more
- 7:08sort of tight granular control over the
- 7:10output. But basically, how I started all
- 7:12of this off was once Hermes was up and
- 7:14running, I created this first
- 7:17instructions.md.
- 7:18And I'll put all of these up on my
- 7:20GitHub in case you're interested. You
- 7:22will have to go through and modify them.
- 7:24But let's have a quick look at that and
- 7:25then we'll see what actually happened.
- 7:27Now, just so you know, I am by no means
- 7:29an AI expert and this is me learning as
- 7:31well. So, any other hints, tips you've
- 7:34got, please let me know.
- 7:37So, I started out by saying you're an
- 7:38autonomous systems agent. Your objective
- 7:40is to deploy a complete home lab stack
- 7:42on an existing Ubuntu machine running in
- 7:44Proxmox. So, I gave it the IP address,
- 7:47gave it the user, told it where the SSH
- 7:49key was, gave it some variables which is
- 7:51in a separate file, told it that my base
- 7:54domain will be Jim's Garage. And I also
- 7:56said that every single project I want
- 7:58you to spin up, you need to put it in
- 8:00sort of this home repository layout. And
- 8:03also I want to use bind mounts as well
- 8:06in this kind of format. I like that just
- 8:08because it keeps my Docker Compose and
- 8:10this script will actually go and build
- 8:12all of those Docker Compose files. And
- 8:14then it will also create a repository
- 8:16where all of the files for those
- 8:18services are stored. Not using volumes
- 8:20because they're a bit harder to
- 8:22manipulate, just with typical bind
- 8:23mounts so you can actually go in as a
- 8:25human and look at the configuration
- 8:27files and all the files that are needed
- 8:28for the container. I also told it that I
- 8:30want an internal Docker network of
- 8:32proxy, so you guessed it, yeah, I want a
- 8:34reverse proxy setting up here.
- 8:36So what it does is basically run all
- 8:38commands remotely over the SSH, so it
- 8:40connects that machine. It's going to
- 8:42read the variables file, which is down
- 8:44here. I won't show that because it's cut
- 8:46some secrets for things like pulling my
- 8:49valid SSL cert for Let's Encrypt.
- 8:52It takes the Vaultwarden user and the
- 8:53master key as well which I've specified
- 8:56and then generates all passwords for
- 8:58everything. So anytime it pulls down a
- 9:00container or spins up something like the
- 9:02traffic dashboard or it creates the web
- 9:05interface and oh, by the way, anything
- 9:07that supports OAuth, I've automatically
- 9:09told it that
- 9:10go and put that and integrate it into
- 9:12Authentic.
- 9:13It then stages all the credentials into
- 9:15a temporary file. It will then stick
- 9:17them all into Vaultwarden, stores a
- 9:20record of all of the Compose files
- 9:22and then basically stores that in
- 9:24Authentic. It will always use the latest
- 9:27versions of containers, always use
- 9:29official documentation and you can see
- 9:31further down I've created links where it
- 9:34shows you where the official
- 9:35documentation is. I do find that that
- 9:37helps it out a little bit. I want I then
- 9:40want it to add all the containers to
- 9:41homepage and then configure any widgets
- 9:43that are available. After that it
- 9:45basically goes through and does the host
- 9:46preparation. It will connect via SSH, it
- 9:49will disable anything. So this bit here
- 9:51I think was for configuring
- 9:55technician, so it turns off the resolve
- 9:57config.
- 9:58It installed the Docker engine if it was
- 9:59needed, created some shared credentials
- 10:02and then got onto installing
- 10:03Vaultwarden. So needed to install
- 10:05Vaultwarden first just because that's
- 10:07kind of where I was going to store all
- 10:08the credentials. We then set up traffic
- 10:11so that we can serve up every
- 10:12container's dashboard with valid SSLs.
- 10:15And I also want a crowd sec, which can
- 10:16often be a bit of fun to install, but
- 10:19actually using something like this, it
- 10:20does takes the strain out of it. And
- 10:22this is where I have that sort of
- 10:24dissonance with it because on the one
- 10:26hand, dead easy to go and set up a home
- 10:28lab,
- 10:29but at the same time I've got the
- 10:30benefit of I've done this a hundred
- 10:32times and know how it works. And I do
- 10:34worry that people will just use the AI
- 10:36and not actually understand and be able
- 10:37to debug things when things do go wrong.
- 10:40And for me personally, I think a lot of
- 10:41the value and the knowledge is from
- 10:42actually understanding what's going on
- 10:44the hood and using this more of a tool
- 10:46just to expedite things. That's my take
- 10:49on it anyway.
- 10:50It then set up a technician cluster and
- 10:52as I mentioned before, um that was two
- 10:54machines and all the records for every
- 10:56single dashboard and service I have will
- 10:58automatically be added via the CLI into
- 11:02technician itself.
- 11:03Then set up notifications and monitoring
- 11:05want go to five. I've got crowd sec and
- 11:07uptime Kuma, so any of those plugged in
- 11:09automatically will generate the
- 11:11notifications and it will automatically
- 11:13send it to my element server here
- 11:16whenever I get some initial issues.
- 11:19Whenever I get any issues. So here for
- 11:21example, something's gone down and it'll
- 11:22even say what do you want me to do on
- 11:24those when you open up these threads. So
- 11:26here you can see it's down. It's going
- 11:28to bring it back up and then it went and
- 11:29brought it back up, which was pretty
- 11:31awesome. I I spun that one down. We'll
- 11:33do a demo of that later. After that, I
- 11:34got into sort of the application suite.
- 11:36So you could basically modify this
- 11:38instructions.md file and just add
- 11:40whatever containers you want. So for
- 11:42each one, I've basically done it in this
- 11:44format here. So you got the service,
- 11:45you've got the documentation where it
- 11:47sits, the subdomain you want it on and
- 11:49any special requirements. So for
- 11:51example, for container, here's the docs.
- 11:53I want it sitting at portainer.
- 11:55You can mount the sock for it and also
- 11:58create an admin user and generate a
- 12:00password and that will then get stored
- 12:01into Vault Warden.
- 12:03I then spun up some additional servers
- 12:04down here and then said I want to
- 12:06basically talk to you, this Hermes agent
- 12:09through Matrix synapse. So it went away.
- 12:11It spun up Matrix itself. It has a API
- 12:14gateway or web gateway for Hermes
- 12:16naturally, so it can integrate into
- 12:18Matrix. That was really nice to see.
- 12:21And then lastly, basically all the
- 12:23secrets that it created during all of
- 12:26these applications, it then went and
- 12:28stored them from that temporary file
- 12:30using the Bitwarden CLI, logged into
- 12:32Vault Warden and stored them all, then
- 12:35verified that they all worked, which is
- 12:36what's awesome. I think one of the key
- 12:38values of this is every single time it
- 12:41does something, I always ask it to go
- 12:42and verify that it works. So, this is
- 12:45probably a bad example, but let's say
- 12:47for example at the moment I've actually
- 12:48forgotten my key for Vault Warden. I can
- 12:51ask it what's
- 12:54my
- 12:55login for
- 12:57Vault Warden?
- 12:59It should even be able to understand my
- 13:01terrible typing.
- 13:05So, here you see it read the Vault
- 13:06Warden skill for Vault Warden access
- 13:08because I've asked this a few times and
- 13:10it's recorded it as a skill. So, it
- 13:12should be able to come back now, check
- 13:14that out using the skill, and here you
- 13:16go, it's admin at Jim's Garage, and then
- 13:18I've got the password beneath it. So, I
- 13:21can copy this now. We'll log into Vault
- 13:23Warden up here.
- 13:33And now that's unlocked, you can see all
- 13:35of the services that I've got running in
- 13:36here, and also all of the things like
- 13:38API keys, etc., which aren't stored as
- 13:41logins.
- 13:42So, now for example, what can we do?
- 13:44Well, we can do things like Portainer.
- 13:46And as I mentioned before, because I've
- 13:48got authentic set up, I can actually log
- 13:50into Portainer with authentic. And all
- 13:52of this again was done just through the
- 13:55LLM. So, if I log in with OAuth, it will
- 13:57forward me to authentic. We now need to
- 14:00go in with authentic. We'll log in.
- 14:03And then hopefully it should return me
- 14:05back to Portainer and log me in as that
- 14:07user. And then yeah,
- 14:09here's the 46 containers I've actually
- 14:11got running on this machine.
- 14:14So, that was pretty awesome. All these
- 14:17are up and running. I've even got things
- 14:18like posters for social media and stuff.
- 14:21I've got Restic. Restic is automatically
- 14:23configured as well. So, when I was
- 14:24deploying this container, it asks, do
- 14:27you want me to just set this up for
- 14:28regular backups? So, I said yes. And
- 14:30then when I deployed our clone, it said,
- 14:33do you want to automatically map the
- 14:35Restic volume to backup? I said yes. I
- 14:38haven't actually gone through and linked
- 14:39it to anything yet, but I've got the
- 14:41capability in the configuration there.
- 14:43Now, hopping onto the Docker VM as well
- 14:45here. So, this is the one it's actually
- 14:47deploying all of those containers too.
- 14:49You can see here I've got the Docker
- 14:50folder, which stores all of the files
- 14:52for the containers. And then we've got
- 14:54the Docker compose here as well. So, it
- 14:56actually creates all the compose files
- 14:58as well. Goes and pulls them down,
- 14:59modifies them, uses best practices like
- 15:02using variables here for all of the
- 15:05values, which are then mapped to a .env
- 15:07file for security, which is great. And
- 15:09it even managed things like Jellyfin.
- 15:11So, I told it for Jellyfin,
- 15:14go and load um the GPU into this so that
- 15:18you can do GPU transcoding. And it did
- 15:20that, and it works, and it was dead
- 15:22straightforward. It's even put sensible
- 15:24things like putting all the memory
- 15:25limits onto the containers, which again
- 15:27is another good practice. And it also
- 15:29put everything behind that reverse proxy
- 15:31that I set up within Traffic.
- 15:33It was just amazing to see that. Now,
- 15:35going into something like NA10, here are
- 15:37three of the workflows I've asked it to
- 15:39create.
- 15:40So, instead of it having to now rely on
- 15:42a skill, I've got it to deploy
- 15:44containers using these workflows, which
- 15:47is great because as you can see, it will
- 15:48deploy webhook, it will have an import
- 15:51deploy request, it will validate it, it
- 15:53will then write the compose file, create
- 15:55the records here for the DNS, it will
- 15:58then have a failure condition, it will
- 15:59store the vault, and then it will return
- 16:01a success of true.
- 16:03Verifying the deployment starting the
- 16:04stack. You can see now the value of
- 16:08having this level of automation baked
- 16:10into an LLM.
- 16:11So, now when I send it a prompt to spin
- 16:12up a new container, instead of having to
- 16:14work out each time, how do I spin up a
- 16:17Docker container? Because the first few
- 16:19cranks of the handle, it would actually
- 16:20have to figure out how do I deploy a
- 16:22container, say write a compose file,
- 16:24what's a compose file?
- 16:26Now it knows exactly what those things
- 16:27are. So instead of working it out, it
- 16:29just points straight to this automation,
- 16:31which it built itself in NA 10. And what
- 16:34that means is much more consistent
- 16:36output because it's all written here in
- 16:38code. You can see what it's actually
- 16:39doing here.
- 16:41And it also means that there's far fewer
- 16:42tokens being consumed because once it
- 16:44comes into NA 10, it's not actually
- 16:46having to do any work. It's just saying
- 16:48kick this process off and wait for the
- 16:50output. It either is a success or it's a
- 16:52failure, and then it will come and ask
- 16:54me what to do. We'll show an example of
- 16:56that in a minute.
- 16:57I've also got some other automations
- 16:59which were Vault Warden getting items.
- 17:01Again, what does it need to do? So it
- 17:03needs to go here and it's got the query
- 17:05it needs to run to actually get that. So
- 17:07again, before I had this in here, and
- 17:10you could do this in a skill as well of
- 17:11course,
- 17:13each time it was having to manually
- 17:14figure out well, how do I get a Vault
- 17:16Warden item? And it was having to read
- 17:18through the documentation, burn a ton of
- 17:20tokens before it got to the answer.
- 17:22And then finally I had this run an SSH
- 17:25command on here. So it basically says
- 17:27check the secrets, the secret okay, run
- 17:29it as either denied or it worked, and
- 17:31then you could run on the Docker VM. So
- 17:33again, a simple way for it to connect to
- 17:36a host, use a certificate, and then get
- 17:38access to it. Again, the more things I
- 17:41can get as skills through NA 10, the
- 17:43better because it gives me a more
- 17:45consistent output. So before I actually
- 17:47go and start letting this thing loose,
- 17:50the beautiful thing is now I've actually
- 17:52got things like these NA 10 automations.
- 17:55I've also got the skills that are within
- 17:57my Hermes agent that I don't need to
- 17:59start fresh. I've got the initial
- 18:01prompt. I've got all of the skills it
- 18:02can use. I'll just need to update things
- 18:04like the IP addresses etc., which I can
- 18:07probably change the hostnames as well,
- 18:08which would make a lot more sense. I can
- 18:10also bring over all these NA 10
- 18:12automations and just import those so
- 18:15that again, the first few cranks of the
- 18:17handle of spinning this up, it should be
- 18:18able to one shot it, which it pretty
- 18:20much the first time. Here you can see
- 18:22with Technetium as well, it was able to
- 18:24basically create a two-node cluster
- 18:26straight away. And if you go to the
- 18:28administration and the cluster, you'll
- 18:30see these are both set up, and they're
- 18:31also set up with my cluster domain, so
- 18:34my actual domain name. It even created
- 18:36these on different IP addresses. Sure,
- 18:38at the moment they're on the same host,
- 18:40which doesn't work obviously for
- 18:41redundancy purposes, but it's only a
- 18:44testing bed. But it sorted all of that
- 18:46out automatically and automatically
- 18:48clustered everything together. So I've
- 18:50got all of the zones and then all of the
- 18:54uh DNS records all in here, and they're
- 18:56all replicated across both of these
- 18:58servers, which is awesome. So even
- 19:01things like Myriota Talk, which can be a
- 19:03bit of fun to set up, those are all
- 19:05working as well. I have a video on
- 19:07Myriota Talk, and if you've never used
- 19:08it before, definitely go and recommend
- 19:10it. It's literally this simple to just
- 19:12go and spin up a room where you can do
- 19:14VoIP chats, where you can do voice
- 19:15chats, you can share files, you can do
- 19:18all sorts. Obviously not going to work
- 19:19very easily in a VM. So yeah, that's
- 19:22been pretty awesome to spin up. Here you
- 19:24can see some of the token usage. This
- 19:26isn't quite right because this is
- 19:27running on the uh virtual machine, so
- 19:29it's actually about more like 20 million
- 19:31tokens I've been using this week. Um the
- 19:34other server's running on my machine I'm
- 19:36recording this on, and that's been up
- 19:38far longer doing that recording.
- 19:40But this is a cool dashboard as well.
- 19:42Definitely go check this out if you've
- 19:43got a Strix Halo device.
- 19:45Um but let's have a bit of fun now, and
- 19:47I'll show you some real-world examples
- 19:49of using Hermes to basically manage this
- 19:52thing. So as I mentioned at the start of
- 19:53this video, I've been having um a bit of
- 19:56an issue with that container D
- 19:58because the version that comes on this
- 20:00machine is causing it to crash with so
- 20:02many containers on here. So if I ask it
- 20:05what's causing
- 20:07the
- 20:11Docker VM to crash,
- 20:14the beauty is now it will go away and
- 20:16check that machine. It will check all of
- 20:18the logs for me, all of the logs for all
- 20:20of the containers, etc. And hopefully,
- 20:22this will be real world, Don't let me
- 20:24down. It will come back with that issue
- 20:26around container D.
- 20:28Now, it will offer as well to go and fix
- 20:30that for me.
- 20:31I'm choosing not to do that just because
- 20:33I don't want to break the APT repository
- 20:35at the moment. But here, let's have a
- 20:37look. So crash one, it found out on
- 20:39November 6th, crash two on the 7th.
- 20:42And it's saying again it's because of
- 20:44container D.
- 20:45Because of container D 2.2 package.
- 20:48Um
- 20:50it has an out of memory killed app
- 20:52process and that again is that
- 20:54deadlocked in the run C shim command,
- 20:57which is a well-known bug.
- 21:00Quite surprised that it hasn't actually
- 21:01been patched yet and one of the reasons
- 21:03if I'm going to go down this route, I'm
- 21:05probably going to go with something like
- 21:06Fedora. Sure, it's not an LTS build, but
- 21:09it'll hopefully be able to diagnose and
- 21:11fix some of these problems and I don't
- 21:12mind having a bit more of a risky home
- 21:14lab if I can go and fix things quickly.
- 21:16So now then, let's have a little bit of
- 21:18fun and let's set something up like I
- 21:20don't know, let's set up Keycloak. I
- 21:22don't want it to be automatically
- 21:23configured. I want it to automatically
- 21:25basically follow all of the things I've
- 21:28got here. So I want it behind a proxy. I
- 21:30want all of the credentials being saved
- 21:32into Vault Warden.
- 21:34Let's see how it gets on. I'm going to
- 21:35write the following prompt. So I'm going
- 21:37to keep this one pretty simple because
- 21:39I've got to the position now where most
- 21:42of the things I want to use are either
- 21:43skills or automations. So I want you to
- 21:46deploy Keycloak using the official
- 21:48documentation and then I've linked it.
- 21:50It's obviously got Docker in it, so it
- 21:52should know it's a Docker uh deployment
- 21:54I want. I want it to follow the
- 21:56deployment instructions in the
- 21:57instructions.md.
- 21:59So that's one I mentioned earlier.
- 22:01So it should go away now and start to
- 22:04work on doing that. First thing it will
- 22:06check for that instructions md, which it
- 22:08found. It's now going to the website for
- 22:10Keycloak that I linked it and it's
- 22:12reading through those instructions.
- 22:14Here you can see everything that's in
- 22:15the instructions md, so all of those IP
- 22:18addresses, all of the things that we
- 22:19saw. It'll understand exactly how it's
- 22:22going to deploy this, all of the things
- 22:24it needs to have in so DNS records, it
- 22:27needs to tie in with authentic, if it
- 22:28can use it, those sorts of things. It's
- 22:31got the skill for the Docker VM, and
- 22:33it's also got the skill for the Docker
- 22:34stack. So it basically knows how to
- 22:36create compose files, it knows where to
- 22:38go. And hopefully, you can see here it's
- 22:40generating some passwords. It should
- 22:42also then follow the NA 10 deployment,
- 22:45and we'll be able to check NA 10 if it
- 22:47actually used it because it keeps a
- 22:48record of all the times it's been
- 22:49executed.
- 22:51Here you can see the plan it's going to
- 22:52follow, so check the VM state, create
- 22:54the password, technician record, Docker
- 22:56compose up, create the realm, then
- 22:58create widgets for homepage, etc., and
- 23:01then validate that everything is up and
- 23:03running. Here you can see the traffic
- 23:04labels it's creating. You can see the
- 23:06secrets it's creating. Obviously,
- 23:09security is a big concern
- 23:11when you're running things like a large
- 23:12language model locally, especially when
- 23:14you're giving it things like internet
- 23:16access. And this is kind of where I want
- 23:18to get more control around it and why I
- 23:21haven't just let this thing loose in my
- 23:23current home lab. My current home lab is
- 23:25where I keep all of my stuff, right? So
- 23:28I need to make sure that it's secure.
- 23:30And all of this is going to be that test
- 23:31bed. And I'm going to do some subsequent
- 23:33videos where we talk about things like
- 23:34the guard rails, some absolute
- 23:37must-haves, and some absolute no, you're
- 23:39not touching that.
- 23:40And in future, what I'll probably do is
- 23:43restrict access to Vault Warden, but
- 23:45whilst it's setting up and I've not got
- 23:47actually any useful data in there, it
- 23:49can have access and set the home lab up.
- 23:51In future, I'll have it ask me for the
- 23:53password so it can't just go and access
- 23:55it anytime it wants. Here you can see
- 23:56it's generating the compose file, which
- 23:58it's going to go and deploy.
- 24:00And you can see on the bottom some
- 24:01statistics. So currently we've used
- 24:03about 40,000
- 24:05tokens out of a context window of 262 K.
- 24:09It's generating at around 56 tokens per
- 24:12second at the moment, and the whole
- 24:13request has taken about two and a half
- 24:15minutes.
- 24:16So if you think how long it takes for
- 24:17you to probably follow one of my videos,
- 24:20and certainly when I deployed this for
- 24:21the first few times,
- 24:23it was probably around half an hour to
- 24:25an hour to get every single thing
- 24:27configured. Now we're in a position
- 24:29where we can just point it to some
- 24:30instructions, say how we want it
- 24:32configured and then let it work it all
- 24:34out. Now the first time I ran uh this
- 24:37prompt and set up some of these uh
- 24:39containers, it was actually asking me
- 24:41quite a few questions. But because I've
- 24:43got this thing dialed in and the more
- 24:45you use it, the more it learns your
- 24:46behavior,
- 24:48it basically knows what I'm going to say
- 24:49already and it saves those as skills. So
- 24:52when it says things like, oh can I
- 24:53connect to that machine? It already
- 24:55knows. Can I generate a password? Sure
- 24:56you can. Can I store that? Yes, you can
- 24:59because I've explicitly told you to go
- 25:00and do that.
- 25:02Now you can see it's generated the
- 25:04config file with all of the added
- 25:05variables put in. And it is quite
- 25:07fascinating seeing how it talks to
- 25:09itself and works things out.
- 25:11I haven't actually had an issue where
- 25:13it's got into a loop. I have had that
- 25:15before when I've used um older versions
- 25:17of Kwen, things like 3.5, 3.6, whereby
- 25:22it would start to hallucinate and get
- 25:23stuck in a loop. Actually, 3.8 Flash
- 25:25next. I don't think I've had it loop
- 25:27once. It has kind of burned through
- 25:29tokens unnecessarily at times,
- 25:32but that's usually down to my fault and
- 25:34me not being explicit or asking it to
- 25:36ask me questions when it's unsure, which
- 25:38is absolutely something you can do. And
- 25:40we'll touch on some better prompting in
- 25:41another video. Here you can see it's now
- 25:43created that DNS record and it will be
- 25:46logging in and doing that. It's now
- 25:48checking whether it can make a widget
- 25:50within homepage as well for Keycloak.
- 25:53I'll edit this little bit out because it
- 25:54is sleeping for 75 seconds. It's just
- 25:57making sure that it's got time to spin
- 25:59up the container and then make sure it's
- 26:01before it does that readiness check. So
- 26:03now that time's passed and hopefully
- 26:05everything was up and running. The good
- 26:07news is if it isn't up and running,
- 26:09it'll often find out why and it will
- 26:11check all of the containers and make
- 26:12sure everything's good.
- 26:14Says the container started. Keycloak
- 26:16still booting. Says the first boot takes
- 26:18a minute, so it's going to wait again
- 26:19and recheck. So again, I'll edit this
- 26:21forward. This is not the LLM being slow.
- 26:24This is just it giving it time to boot
- 26:26up as it recommends by the container.
- 26:28Now we go. It has found it, it has
- 26:30waited and it's now checked that it's
- 26:31listening on port 8080. Interestingly as
- 26:34well, if the things like a port clash,
- 26:37it'll often figure that out and manually
- 26:39change and keep a record of what ports
- 26:41it's put on. Obviously that's less of an
- 26:43issue if you're running something like a
- 26:45proxy because everything will just run
- 26:47through the proxy anyway. Seems to have
- 26:49found a problem with the healthy state
- 26:51returning a 404. So it will probably go
- 26:53away now and check the script and find
- 26:55out what happens. Well, that was because
- 26:57the health point is disabled by default,
- 26:59so the script died early. Here you can
- 27:01see also it's having a bit of a conflict
- 27:04with some of the versions. So I think it
- 27:05was originally quoting version 24, but
- 27:08now it's saying 26.8. So sometimes it
- 27:11can get fused based upon what the LLM
- 27:13was trained around, but that's why I
- 27:15feed it the latest documents, so it
- 27:17should go away and do those.
- 27:19It did the Keycloak admin login and that
- 27:21succeeded. And now it's found an issue
- 27:24with the client's
- 27:26account that's being created. Now I
- 27:28can't actually remember off the top of
- 27:30my head how you actually do that, but it
- 27:32might be quite complicated for it to go
- 27:34and work that out. You might be better
- 27:36off sometimes just going in through the
- 27:38traditional way through the web GUI and
- 27:41doing it that way.
- 27:44But that's the beauty of this. Usually
- 27:45when I've just left this to work, it
- 27:47will basically go and figure it all out
- 27:50and it will also give you a dump at the
- 27:51end as we'll see of exactly the stages
- 27:53that it followed. So you can go through
- 27:55and check and make sure it didn't do
- 27:56anything stupid. Now it's doing a
- 27:58headless check using a browser to
- 28:00actually go to the website and log in
- 28:01with the new credentials.
- 28:03Again, that's one of the awesome
- 28:04features, all of the tools that the
- 28:06Hermes agents come with. It's able to
- 28:08take screenshots of things as well that
- 28:10it's changed and I've been building some
- 28:12five coded apps with this and it will
- 28:14even take pictures of it and just make
- 28:16sure that what you're saying in words is
- 28:18actually then replicated on screen. So
- 28:20it's now working out the flow using the
- 28:23web interface and then it will be
- 28:24changing the script so that it's got a
- 28:26script which basically validates it via
- 28:29code instead of having to go through the
- 28:31web user interface. So that's great for
- 28:33when you wanted do debugging checking
- 28:35things. It can do that
- 28:37it can do that via a script as opposed
- 28:38to going through the web front end. So
- 28:40as all of this is going on, we can be a
- 28:42nerd as well and we can check the
- 28:44dashboard for the machine itself.
- 28:46You can see here all the tokens that are
- 28:48being generated, which is pretty
- 28:50awesome. And if we want to go to the
- 28:52Grafana dashboard, we can see sort of
- 28:54the CPU and RAM usage, disk usage and
- 28:56what's going on there. Now it's included
- 28:58the home page card and the admin flow
- 29:01has failed, so it's going to double
- 29:02check what happened there. Home page and
- 29:04uptime Kuma are now done as well. So it
- 29:07looks like the issue was with traffic
- 29:08and not using the X Forwarded protocol.
- 29:11So hopefully it's going to go away now
- 29:12and reconfigure that, redeploy the
- 29:14container so that it passes on the HTTPS
- 29:17traffic and fingers crossed that should
- 29:18now work. So apparently after 70 minutes
- 29:21of which a few of those minutes were
- 29:23actually it's just waiting for checks.
- 29:26It's stating that Keycloak is deployed
- 29:28and it's been verified. So what did it
- 29:30do? It deployed 26.8.2 which I think is
- 29:33the latest version.
- 29:35It created the compose files as I said.
- 29:37It set the data directory where I wanted
- 29:39it. Put it on the traffic network, which
- 29:41is all good. It also integrated it with
- 29:43CrowdSec, so anything going through
- 29:44there is going to be scanned by CrowdSec
- 29:46for any security vulnerabilities.
- 29:49Followed the instructions, the document
- 29:52flow.
- 29:53It also then
- 29:55generated the passwords. It then created
- 29:57all of the accounts and verified that
- 29:59they worked and fingers crossed added it
- 30:02all to the necessary containers, things
- 30:04like the home page, uptime Kuma, etc. So
- 30:07let's now have a little look and see if
- 30:09that all went to plan. So back on this
- 30:11machine, let's click on home page and
- 30:13what do we get? So we've got Keycloak
- 30:16down here. So we should be able to click
- 30:17on that.
- 30:19Um this looks promising as it
- 30:21automatically added it here so that I
- 30:22can click it. Yep, there's something
- 30:24here. Let's sign in.
- 30:28Temporary admin user. I'll probably need
- 30:30to go and fix that.
- 30:32I don't know what a temporary admin is.
- 30:34Forgotten how to use Keycloak.
- 30:36Um but sure we can just basically save
- 30:38it. But
- 30:39the long and short of it is this is all
- 30:41up and running, and all that was was a
- 30:45single prompt with a few skills that
- 30:46I've tweaked over a few iterations.
- 30:49And also things like Ollama. Where is it
- 30:53in here?
- 30:57Here you can see Keycloak up and
- 30:58running, and it's automatically if I go
- 31:01to the edit it should have the
- 31:02notifications. Yeah, it's got the
- 31:03notifications to go to fire a matrix
- 31:05already set up, which is awesome. And I
- 31:08guess things like Grafana should
- 31:11automatically pick it up as well. Yep,
- 31:13here you can see that Keycloak has been
- 31:15picked up. So it's automatically looking
- 31:17at that as well. Probably doing that
- 31:19more just through the Docker sock than
- 31:21any formal sort of tweaks to any config
- 31:23files.
- 31:24It's obviously got the records added
- 31:27into technician, which is awesome. And
- 31:29we can also log into Portainer if we
- 31:31wanted to see it on here already spun
- 31:33up. So here it is Keycloak as well. And
- 31:35then this will behave just like any old
- 31:37sort of Portainer instance, which was
- 31:39also deployed by this script as well.
- 31:43Which again just really shows the power
- 31:45of this.
- 31:46So where do we end? Let's wrap up.
- 31:49There's lots of things we need to talk
- 31:50about here. Now as I said at the
- 31:51beginning, this is going to form the
- 31:53start of kind of this AI Home Lab
- 31:56journey. And that was just to sort of
- 31:58wet your whistle.
- 31:59Now don't get me wrong. I absolutely
- 32:02love doing all of the tutorials and
- 32:03setting things up by hand, reviewing all
- 32:06of the code and setting all the
- 32:07configurations, and that's not going to
- 32:08change.
- 32:10What might change is the way in which
- 32:12you deploy things with those
- 32:13configurations. So you might want to
- 32:15watch the video and understand what's
- 32:17going on in the hood, but you don't
- 32:18actually want to spend all of that time
- 32:20configuring it yourself, pressing
- 32:22keystrokes, when all you need to do is
- 32:23send it to an LLM with a config file and
- 32:25give it the instructions that you want.
- 32:28Now from an efficiency standpoint,
- 32:30replicability, all of that sort of
- 32:32stuff, security, things like having it
- 32:34in code via things like Ansible and
- 32:36Terraform are definitely going to be
- 32:39better in terms of guaranteed outputs,
- 32:41and you have full control over it.
- 32:43There's basically no AI. There's no sort
- 32:45of unknown variable in the middle.
- 32:48But I think we're soon going to get to a
- 32:49place where most of this stuff is going
- 32:51to be unavoidable. And especially for
- 32:54diagnosing and fault tolerances and
- 32:56monitoring and all of that, I must say
- 32:58having this integrated into Element,
- 33:00having that on my mobile phone and being
- 33:02able to remotely manage my home lab on
- 33:05the fly or at the moment this mock home
- 33:07lab
- 33:09has been amazing. It's honestly when
- 33:11things used to break and I'd be away at
- 33:12work or away from home, it would often
- 33:15be hours of trying to get into a virtual
- 33:17machine or over the VPN and all that
- 33:19stuff and try and diagnose it on small
- 33:21laptop screens or phones and it was just
- 33:22an absolute nightmare.
- 33:24I must say when I've tried to break
- 33:25containers and shut them down, it's
- 33:28automatically worked, notified me and
- 33:29come up with a recommendation. So simply
- 33:32spin it back up. Or if there is a more
- 33:35sort of difficult fault, it will check
- 33:36all of the container logs, it will check
- 33:38the host logs and do what it recommends,
- 33:40maybe upgrade the container, change the
- 33:42RAM, up the CPU core counts, all of
- 33:44those sorts of things.
- 33:47And that capability is only going to get
- 33:48better when I give it more control.
- 33:51Potentially, we'll see how that goes.
- 33:53Things like integrating it with Proxmox
- 33:55will allow it to control the Proxmox
- 33:57host. Things like allowing it to have
- 33:59access to my NAS will be enable it to
- 34:01set up shares and all of those sorts of
- 34:03things and potentially even one day
- 34:05maybe set up things like the firewall so
- 34:07I can set up some of that routing.
- 34:09Now all of that is going to come in this
- 34:11series. I don't know exactly what the
- 34:13flavor of that will be, but there's also
- 34:14going to be some obvious things like
- 34:16spinning up a Kubernetes cluster, maybe
- 34:18things like a Docker Swarm. And I'll be
- 34:20documenting all of that because I'm
- 34:21really keen to see how it goes.
- 34:24Anyway, let me know if this is something
- 34:26that you're interested in. Let me know
- 34:27if you've already integrated LLMs into
- 34:29your home lab and how that's going. What
- 34:31are the pitfalls? What can you share?
- 34:33What are some of the things that I need
- 34:34to avoid?
- 34:36But anyway, if you've liked this video,
- 34:37please give it a thumbs up, hit that
- 34:39subscribe. I'll see you on the next one.
- 34:41Take care, everybody.
About this transcript
This page contains the full transcript of Is this the future of homelabbing? by Jim's Garage, generated from the public captions YouTube serves with the video. The transcript has 7,068 words across 1,012 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.