YT YouTube2Text

YouTube2Text → Transcripts → How to build a JARVIS AI Voice Agent for FREE | Full Livekit Tutorial (2026)

How to build a JARVIS AI Voice Agent for FREE | Full Livekit Tutorial (2026) — Transcript

by Thanh-y David Nguyen · 8,524 words · 1,270 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00In this tutorial, I'm going to show you
0:01how to create your own Jarvis AI voice
0:03agent for free using Python and an open
0:06source to code life kit. Hey Jarvis, can
0:08you see this character that I'm holding?
0:12>> Yes, I can see it. It appears to be Son
0:15Goku from Dragon Ball Z looking
0:17characteristically bellicose.
0:19>> [music]
0:20>> Shall I find you somewhere to display it
0:21or perhaps more of them?
0:23>> Hey Jarvis, are you there?
0:24>> At your service, sir.
0:26>> What is the weather in Vienna?
0:31>> It is apparently mostly cloudy in
0:33Vienna, Austria with a temperature of
0:3528° C. [music]
0:37>> Could you open Google Maps for me and
0:39show me the fastest way to get from
0:41Vienna to Berlin?
0:45>> [music]
0:51[music]
0:53>> I have opened Google Maps, entered
0:55Vienna as the starting point and Berlin
0:58as the destination. The fastest route
1:00should be visible on the screen now.
1:02>> Can you show me the flight options?
1:06>> The flight from Vienna to Berlin takes
1:08about 1 hour and 10 minutes with
1:10Austrian, Eurowings or Lufthansa.
1:12>> Could you play some iconic Iron Man
1:14music?
1:20>> Naturally, sir. I [music] have commenced
1:22playing that most iconic of themes.
1:25Shall I prepare some far too advanced
1:27technology for your consideration as
1:28well?
1:30>> And on top of that, I'm also going to
1:32show you how to launch Jarvis as an
1:33[music] Android as well as iOS app so
1:35you can talk to them wherever you want.
1:37Isn't that right, Jarvis?
1:39>> Yes, indeed, sir. Though I must say your
1:41intros are getting a bit repetitive.
1:44>> [music]
1:44>> All right, then let's try something
1:45else. Can you see me filming this intro
1:48right now?
1:49>> [music]
1:50>> Yes, sir. I can see your camera setup
1:53and lighting stand. [music] It looks
1:55quite professional for a washed up
1:57YouTuber.
1:58>> What the hell, bro? I actually already
2:00did the same video about a year ago, but
2:02since then there have been a lot of
2:03changes [music] to LiveKit. So, here's
2:05the updated version for you. So, let's
2:06get started. So, the first thing we need
2:08to look at is something called LiveKit.
2:10For everyone who's not a regular viewer
2:12on this channel, LiveKit is an
2:13open-source AI framework that you can
2:15use to build voice agents, and it is
2:18really sophisticated. Even very big
2:19companies like OpenAI use it and they
2:21integrate it into ChatGPT even. And
2:23since it's open-source, you can actually
2:25use it completely for free, for example,
2:27if you run it locally. For this
2:28tutorial, however, we are going to use
2:30the cloud version. You can also use the
2:32cloud version for free and it comes with
2:34up to 1,000 minutes a month, which
2:36equates to 16 hours a month, which
2:39should be more than enough for most
2:40users. For the sake of this tutorial, we
2:42are simply going to use the cloud
2:44version because it's easier to set up.
2:45So, first off, we would need to click
2:47this button over here that says start
2:49building.
2:51And then over here, you need to create
2:52an account. Here, you can log in with
2:54your Google account. For this, you can
2:55click on continue with Google.
2:59So, let me select my Google account real
3:01quick.
3:04Here, we need to fill out this form. You
3:06can actually type in whatever you want.
3:08For example, AI agents, I don't know,
3:10YouTube here.
3:11Other.
3:13Continue. And here's where we need to
3:15create our first project. We also have
3:17to give this a name. Let's call this
3:18Jarvis newer something.
3:21And at agent observability, we can
3:23disable this for now.
3:25And then click on continue.
3:27Here, we can click on skip for now. And
3:29now you have access to this dashboard,
3:31and over here you can also see that you
3:33created your first project here.
3:36Now, we need to open the LiveKit quick
3:37start menu, and for this you can simply
3:39Google
3:40LiveKit
3:42voice agent quick start.
3:45And then click on the first link.
3:49And here's a detailed manual how to set
3:51up your first voice agent in less than
3:5310 minutes. And if we scroll down a
3:54little bit, we can see here a couple of
3:56requirements we have to meet before we
3:58can start. And first off, we would need
4:00to install Python. And for this, you can
4:02simply Google Python
4:04install.
4:06Then click here on download Python. And
4:09here you can click on this button to
4:10download and install the newest version
4:12of Python. You would also need to
4:14install something called UV. And for
4:16this, you can click on this link over
4:17here.
4:18And here, if you're using Windows, then
4:20you can use this command to install it.
4:22If you use Mac OS or Linux, then you can
4:24just follow this installation guide. For
4:27now, I'm going to copy this. And now we
4:28need to open a terminal, in this case a
4:31PowerShell terminal. So, you can just
4:33simply type in PowerShell
4:35and then paste it in here and then
4:36execute that.
4:39After a few seconds, this should be
4:41installed.
4:42Another thing which we also need to
4:44install is something called Node.js.
4:46This will be important later for our UI.
4:48Again, for this, you can simply Google
4:51download Node.js.
4:54And then go to their website and
4:55download and install it for your
4:57operating system. And before we
4:58continue, I would also recommend you to
5:00download and install VS Code as our IDE.
5:03And for this, you can simply go ahead
5:05and also Google VS Code download. Of
5:08course, you can also use any other IDE
5:10of your choice, for example, Cursor. But
5:12for now, we can click here on download
5:14VS Code. And then you can simply select
5:16your operating system and also download
5:17and install it. After that, you can go
5:19ahead and open VS Code. And then the
5:21first thing you need to do is open a
5:22folder where we want our project to be
5:24at. And for this, we can click on file,
5:27open folder, and then you can select any
5:29folder. I'm going to use this one here
5:31and then click on select folder.
5:33And then we can continue with the
5:35manual. For this, we can simply go back
5:38and then go to the next step. And here
5:40we need to install something called
5:42LiveKit CLI. And for this, you can again
5:44select your operating system and then
5:46copy this command here to install it.
5:49And then we can go back to VS Code, and
5:51then we need to open a new terminal. And
5:53for this, you can press control shift P,
5:55or command shift P if you're using Mac.
5:57And then you can click on create new
5:59terminal, and I'm going to use
6:00PowerShell.
6:02And down here now, you can simply paste
6:04this command.
6:05And then you can press enter to install
6:07it. I'm not going to do it now because I
6:08have it already installed. After the
6:10installation is finished, we can now go
6:12back to the menu again.
6:14And here we need to copy this command.
6:16And then we go back to VS Code again.
6:20And then we need to enter this command.
6:23Here we need to give this device a name.
6:24I can call this Jarvis tutorial.
6:27Press enter.
6:30And now it opens this window where we
6:32have to select a project that we created
6:34in the beginning, and mine was Jarvis
6:36new. And then we click on allow access.
6:39Now we can go back to VS Code, and down
6:41here would ask you if you want to make
6:43this project your default project. You
6:45can select yes, and then we can go back
6:47to the menu.
6:49Here we can copy the next command, which
6:51is this one here. And then we can go
6:53back to VS Code, and then we can paste
6:55it here again. This command will
6:57download some files for us, and also
6:59save it into a new folder. And here we
7:01can change the name of the folder.
7:02Instead of my agent, you can say Jarvis,
7:06and then we can press enter.
7:08We can say yes here.
7:11Over here now, you can see there's a new
7:13folder. Here it also asks you if you
7:15want to install the dependencies. We can
7:16also press enter to say yes. Now it's
7:19done. If you get this error, you can
7:20simply ignore this for now. And if you
7:22click here now, you can see all of the
7:24content of this folder, or all of the
7:26files for your voice agents. And if you
7:27want to try this out now, we can simply
7:29go ahead and use the following command
7:31to go into this folder, which is CD
7:33Jarvis.
7:35And then we can use the following
7:36command to start this up, which is OK
7:39agent dev.
7:42Now it's up and running, and now you can
7:43open this link by either pressing
7:45control or command and then click on it.
7:48Then we can click on open.
7:51Then we can click on start a session.
7:55Hi, can you hear me?
8:00>> Yes, I can hear you loud and clear. How
8:02can I help you today?
8:03>> What can you do?
8:06>> I can help you answer questions, explain
8:08complex topics, or get things done using
8:10my tools. Whether you need a quick fact
8:13or a hand with a task, I'm here to make
8:15it easier. Is there something specific
8:17you're looking for right now?
8:21>> All right, so now we have a working
8:22agent. So the next thing we need to work
8:24on is something called the system prompt
8:26and for this we can go to this folder
8:28that says source and then click on
8:29agent.py. This is basically the main
8:32code of the entire agent and if you
8:34scroll down a little bit, you can see
8:35here that there's a section that says
8:37instructions and here you can find the
8:39so-called
8:41This is basically a very detailed
8:43instruction how the agent should operate
8:45and if you want to change its behavior,
8:46you can simply write it in here with
8:48natural language. In our case, we want
8:50our Jarvis to be able to first of all
8:52know that its name is Jarvis and also
8:54behave like a classic butler. For this,
8:56I already prepared a changed system
8:57prompt so let me quickly paste it in
8:59here for you.
9:01And this is basically just the same
9:03system prompt that we had before with a
9:05few tweaks. For example, here the first
9:07sentence says, "You are Jarvis, a
9:08helpful, sarcastic AI butler." Down
9:11here, I tell it to address me as sir.
9:15And additionally, I also added some
9:17conversational examples that for
9:19example, if I say, "Jarvis, can you do
9:20XYZ for me?" then it should answer with,
9:23"Of course, sir, as you wish. I will do
9:25XYZ now." So again, if you want to add
9:27any specific behavior to your AI agent
9:29or you don't want it to do certain
9:31things, then you can simply add new
9:32rules in here and you can simply
9:34describe it in natural language. For
9:35now, we can go ahead and save this by
9:37pressing control or command S.
9:40By the way, the code of this entire
9:42project is available in the video
9:43description down below, including this
9:45prompt as well. Now, we want to switch
9:46our large language model to Gemini, and
9:49for this we can go ahead and Google
9:52LifeKit Gemini,
9:55and then click on the first link.
9:57And here we have another installation
9:59instruction, and for this we can copy
10:01this command over here,
10:03then go back to the terminal in VS Code,
10:06and make sure that you're in your agent
10:07folder, or in my case the Jarvis folder,
10:09and then paste it in here and press
10:11enter.
10:13Now, this is installed, and now we have
10:15to add some imports here in order to use
10:17this. Here at LifeKit plugins, you can
10:19now add
10:20Google,
10:22and down here we now have to replace
10:24this section with a code snippet that I
10:26already prepared, which is this one
10:28here.
10:31You can pause this if you want to type
10:33this yourself.
10:38And now this allows us to use Gemini
10:40instead of the large language model that
10:42we had before, which is this model here.
10:44The reason that we're using Gemini is
10:46because it's free to use. Quick pause
10:48here because a couple of people in my
10:50last video seem to be confused if Gemini
10:53is actually free to use or not, and yes,
10:55this particular model that we are using
10:57is free to use. It is Gemini 3.1 flash
11:00live preview. Here you can see my usage
11:03of that particular model. Here it says
11:05Gemini 3 flash live. Yes, this includes
11:083.1 as well. Over here you can see the
11:11limits that I have with this model. Here
11:12it says requests per minute, which is
11:16unlimited, so I can have unlimited
11:18requests per minute as I want, and the
11:20same goes for requests per day, which is
11:23also unlimited. The only limit I have is
11:26something called tokens per minute,
11:28which is at 65,000
11:31tokens a minute. Here you can see that
11:33I'm only using 15,000 tokens per minute
11:36for my Jarvis voice agent. So,
11:38realistically speaking, the only time I
11:40would hit this limit is if I'm running
11:42four or more voice agents at the same
11:44time. And even if I would, it would
11:47simply not work for some time, and then
11:49I would just simply have to wait a
11:50little bit, and then I can use it again.
11:52So, I still pay nothing. The only time
11:55you would pay for this is if you want to
11:57increase this limit here. So, yes, you
11:59can use this completely for free, and I
12:01hope this clears up any confusion. So,
12:03now back to the video.
12:05And it also has very low latency because
12:07we are using something called a
12:09real-time model. Also, here I selected a
12:11male voice that is close to Jarvis from
12:13the movies, and here I also gave it a
12:15British accent. This section here also
12:17allows you to change the language in
12:18case you want to use another language.
12:20Now, for this to work, we also need to
12:21create a Gemini API key, and for this
12:24you can go back and Google
12:28Gemini API key,
12:30and then click here on get a Gemini API
12:33key.
12:35And then click on this button that says
12:37create or view Gemini API key.
12:41And here's where you would need to log
12:43in with your Google account, and if you
12:44don't have one, then just create one.
12:46And over here you can click on create
12:48API key,
12:49and then you would need to create a
12:51project if you don't have one already.
12:54And then you can simply give this a name
12:55for your project, and then click on
12:57create project. I already have one, so
12:59I'm going to select test.
13:02And then we can also give this API key a
13:04name. Let's say Jarvis 2 or something.
13:08And then click on create key.
13:11Now, we can copy the API key over here,
13:14and then close this window,
13:16and then we can go back to VS Code,
13:19and then we can go to this file that
13:20says env.local,
13:24and now here we need to add this new API
13:26key by typing
13:28Google API key.
13:33And then you need to paste it in here.
13:36And then we can press control S or
13:37command S again to save this. We can go
13:40back to the agent.py file.
13:43And then we can scroll down a little
13:44bit.
13:47And here where it says STT and TTS, we
13:49can actually remove these lines.
13:55And then press control S again to save
13:57this.
13:58Next, we also want to enable our Jarvis
14:00to be able to see us through the camera.
14:02And there's actually a very easy option
14:04that we can simply toggle on for this to
14:06work. And for this you can scroll down
14:08where it says room options. And then we
14:10need to add a new line here which says
14:13video
14:15input.
14:16And then we set this to true.
14:20And that's actually it. And then we can
14:22save this again.
14:23Another thing which we also have to add
14:25is a new dependency. And for this we can
14:28go back to the terminal and then write
14:30UV add
14:34livekit-agents-images.
14:36And now we can press enter to install
14:38this.
14:40Now that's installed and now we can try
14:41this out again with the same command as
14:43before which is LK agent dev. And always
14:47make sure that you're in your agent
14:49folder, in my case the Jarvis folder.
14:51And then we can press enter again.
14:56Now that's up and running and then we
14:57can try this out again.
15:05Hey Jarvis, are you there?
15:08>> At your service, sir. What monumental
15:10task awaits my attention today?
15:14>> Can you see me through the camera?
15:17>> Indeed, sir. I see you are wearing a
15:20dark colored shirt and appear to be
15:22speaking into a microphone. Is there
15:24something specific you require
15:26assistance with regarding this?
15:30>> And now the cool thing that this model
15:32is actually capable of is multilingual
15:34functionality. So, it means that it can
15:36speak multiple languages at the same
15:38time, and we can try this out right
15:40away, actually.
15:42Kannst du auch Deutsch sprechen?
15:46>> Natürlich, Sir. Ich bin froh zu helfen,
15:49in welcher Sprache auch immer. Was kann
15:51ich für Sie tun?
15:53>> There we go. Now we have our updated
15:54Jarvis voice agent. There's one
15:56limitation this model has, however, and
15:58that is I cannot use this chat function
16:00over here. So, let's say I would write a
16:02message in here, and then I would send
16:04it, it would simply not work. But, given
16:06the fact that this is free and it comes
16:08with camera vision as well as
16:09multilingual functionality right out of
16:11the box, I would actually say that's a
16:12pretty good trade-off.
16:14So, the next thing you want to add to
16:15Jarvis is the ability to perform tasks
16:18for us, and for this we need to create
16:19tools for it to use. For example, let's
16:22say you want to give it the ability to
16:23search things on the internet for us,
16:25and for this we need to add a new
16:26dependency like before. For this, you
16:29can go back to the terminal, and then
16:30you can write new V app LangChain-
16:35community
16:37and then DDGS, and then press enter to
16:39install this.
16:41Now it's installed, and now we can add a
16:43new file in here, and for this you can
16:45click here on this source folder, and
16:47then you can click here on new file, and
16:50then you can call this tools.py.
16:53And here we now have to add some new
16:55imports, which I already prepared.
16:58The first one is logging, which we are
16:59going to use in order to log what is
17:01happening in the background. The second
17:03one is for our function tool, or the
17:05tool that we're going to create, for
17:07which we need function tool and also run
17:09context, and the third one is from
17:11LangChain, which we just installed, and
17:14where we are going to use something
17:15called DuckDuckGo search run, which is a
17:17search engine similar to Google, but
17:20this is free, that's why I'm using it.
17:22Now, we need to create a tool that
17:23allows us to search the web, and for
17:25this we need to create an asynchronous
17:27function, so we can write a sync define
17:31search web.
17:34And then we need to pass a so-called run
17:36context, so we need to write context run
17:38context.
17:42What you see here is GitHub Copilot that
17:44is recommending me some codes I can code
17:46faster, but I'm not going to use most of
17:48this right now, so I'm going to ignore
17:49this.
17:50The next thing we need to pass is the
17:52query, the search query, so what we're
17:54searching for.
17:56So, we can write query here.
18:00Now, we need to give this a description,
18:02so the AI agent actually knows what this
18:04function is about or when to use it, so
18:07we can write something like use this
18:09tool to search the web for information
18:11related to the given query.
18:15Then we need to write try
18:18results equals
18:21DuckDuckGo Search Run, then we need to
18:23pass the query here.
18:28So, basically, we are passing the search
18:30query to the search engine, and then
18:32we're getting a result back.
18:34But I also would like to log this so we
18:36know what the result was.
18:39And then we need to give this or return
18:40this back to the AI agent.
18:43And then we also need to handle the case
18:45where this didn't work.
18:47And that's actually it.
18:49And the last thing we need to add here
18:51is a so-called annotation, so the AI
18:53agent actually knows that it can use
18:55this as a function tool, so we can write
18:57at
18:59function tool.
19:01And that's actually it, and now we can
19:02save this.
19:04Now, we need to go back to the agent.py
19:06file, and then we have to import from
19:08tools
19:10the search web tool.
19:12and then down here
19:14we have to write
19:16tools equals search web tool. And now
19:19the agent has access to this tool.
19:22And now in the system prompt, we also
19:24have to add a new line in here that
19:26tells the AI agent when to use this
19:28tool.
19:30So we can write tools
19:32use
19:34the search
19:37web tool
19:39if the user asked you to search for
19:44information.
19:48And that's it. Now we can save this
19:50again. And now we can try this out
19:52again. For this we can go back to the
19:54terminal, but this time we can use a
19:56different command to run this locally
19:57instead of using the web app every time,
19:59which is going to be
20:01UV run source agent.py console.
20:10Hey Jarvis, are you there?
20:13>> At your service, sir. What manner of
20:16assistance do you require, assuming your
20:18needs aren't merely philosophical?
20:20>> Can you tell me the current weather in
20:23Vienna?
20:28>> It appears Vienna is currently
20:30experiencing passing clouds and a
20:32pleasant 75° Fahrenheit. One might
20:35almost be tempted to go outside,
20:37provided one enjoys that sort of thing.
20:40Is there anything else I can conjure up
20:41for you?
20:42>> We can end this conversation by pressing
20:45control C.
20:48All right, so if we scroll up here, we
20:49can actually see that the agent actually
20:51used the search web tool and here we can
20:54see the result of that. So here you can
20:56see that it actually found out that the
20:58weather is or the current temperature is
21:0075° Fahrenheit. You can also see it in
21:04degrees Celsius. So the way this works
21:06is as I said before, the agent knows
21:08when to use this tool based on the
21:10description here. It also knows based on
21:12the name of this variable what to pass
21:14here. In this case, it's the search
21:16query. Then, we simply pass this to the
21:18search engine. Then, we simply print
21:21this onto the console. That's why we
21:23were able to see the search result. And
21:25then, we can give this to the agent back
21:26so it can simply formulate a fitting
21:28answer to our question. So, basically,
21:30if you want to add any sort of
21:32functionality to your voice agent, all
21:34you need to do is to create new function
21:36tools for it to use. Then, you would
21:38need to give it a fitting description as
21:40well as fitting names for the passing
21:42parameters. And then, also come up with
21:44the business logic.
21:46So, instead of coding yourself, I'm now
21:48going to show you how to use a coding
21:50agent because lately they have become
21:51very powerful, and this is probably the
21:53future of developing anyways. And in
21:55this case, I'm going to show you how to
21:56use Open Code. And for this, you can
21:58simply Google Open Code.
22:01And then, click on the first link. If
22:03you don't know what Open Code is, it is
22:05basically like Cloud Code, but this is
22:07fully open source. And it also comes
22:09with free models so you can use it
22:10completely for free. Of course, you can
22:12also use Cloud Code or any other coding
22:14agent like Code X or even GitHub
22:16Copilot, which you can use directly in
22:18VS Code. But as I said, this is
22:20completely free, so that's why I'm using
22:21it. And for this, you can go ahead and
22:23use this command over here and simply
22:24copy it and paste it into your console
22:26again to install this.
22:28So, again, back go back into VS Code and
22:30simply paste it in here and then execute
22:32that.
22:33If it doesn't work for you and you get
22:35an error like me here, then you can use
22:37a different command in order to install
22:39this,
22:40which is this NPM one over here. If you
22:42installed Node.js as I told you in the
22:44beginning that this should work as well.
22:46So, we can copy this and go back to VS
22:48Code and simply paste this.
22:51And now it's done, and now we can start
22:53this with the following command, which
22:54is Open Code.
22:58And as you can see, this is how it looks
22:59like. Right from the get-go, this works
23:01out of the box and you can even check
23:03this by typing {slash} models.
23:05And here we can see the model that we
23:07currently have selected for this coding
23:09agent. And here you can see it's big
23:11people and it says it's free as well.
23:13Now we can actually go ahead and give
23:15our coding agent additional knowledge so
23:17it knows exactly how to build with live
23:19kit. And for this we have to give it a
23:20so-called MCP server. And for this we
23:23can simply Google
23:24live kit MCP
23:27and then click on the first link that
23:28says docs MCP server. If you don't know
23:31what the MCP server is, it is basically
23:33a remote server that allows our agent to
23:36have access to external tools, similar
23:38to the tool that we created for our
23:39Jarvis agent. But in this case it's a
23:41tool to access the documentation of live
23:44kit. So basically if we add this MCP
23:46server to our coding agent, it will have
23:47access to the entire documentation of
23:49live kit and it will know exactly how to
23:52build with it and how to create anything
23:53that we want. For this we also need to
23:55look up how to set up an MCP server for
23:57open code. And for this you can also
23:59simply Google open code
24:03MCP
24:06and then click on this first link.
24:08And here we have a guide how to set up
24:09any MCP server. Here it says we need to
24:12create an open code.json file to add the
24:14MCP configuration. So what we
24:16[clears throat] can do is that we can
24:17actually point our coding agent to these
24:19two documentations in order to set up
24:21the MCP connection by itself. I know
24:23that sounds quite complicated, but trust
24:25me this is going to make a lot of sense
24:27in a bit. For this you can go back to
24:29open code and then say, here's a guide
24:32to set up an MCP server, create an open
24:34code.json file and add this MCP server
24:37from live kit. And for this now I can
24:39simply copy these links over here. So I
24:41can copy this link of this manual.
24:44Go back to open code and then paste it
24:46in here.
24:48And then do the same for the live kit
24:50MCP server.
24:55Paste it in here.
24:57and then simply execute that.
25:01Now it's done, and now you can see that
25:02over here it added this open code.json
25:05file. And in here you can see that it
25:07added the LiveKit MCP server.
25:10Now if you go ahead and type in {slash}
25:12MCP, you can actually see the connected
25:15MCP servers.
25:17And right now we cannot see our LiveKit
25:19MCP server because we have to close the
25:21session and then reopen it to refresh
25:23this. So let me do this real quick. Over
25:25here you can simply click on this button
25:27over here to kill the terminal.
25:29And then we can reopen one control shift
25:31P, create new terminal, PowerShell.
25:35And then we can write open code again.
25:39And then we can type in {slash} session
25:41to see our older sessions. And then we
25:43can select this one that says open code
25:45LiveKit MCP server setup.
25:48And then we can write {slash} MCP again.
25:51And here you can see that the LiveKit
25:52docs MCP server is connected. So now if
25:55I go ahead and ask our agent, "Do you
25:57know how to add new tools to Jarvis
26:00through the LiveKit documentation?"
26:03It knows exactly how to do it and
26:05answered yes. Since the LiveKit docs MCP
26:08is now configured, you can explore it
26:10for adding new tools to Jarvis, a
26:12LiveKit voice agent. The relevant
26:14concept is tool calling and function
26:15tools. By the way, you can also do this
26:17exact process with any other coding
26:19agent like Claude code or Codex as well.
26:22Now we can go ahead and add the
26:23functionality to our Jarvis voice agent
26:26to control a browser for us. And for
26:28this we can use something called
26:29Playwright, which is basically a library
26:31or tool set directly from Google that
26:33gives us full control over a Chrome
26:35browser. For this, of course, we can go
26:37ahead and ask Open Code to implement it
26:39for us. But before jumping right into
26:41this, I would actually go ahead and ask
26:43it to create a plan of implementation
26:45for us first so we can review it first.
26:47And for this we can go ahead and type in
26:49into the chat {slash} agents.
26:54And here we can switch between builds
26:56and plan. Build is something that we
26:59already selected, which is an agent or
27:01mode that will immediately go ahead and
27:03implement whatever we're asking it to
27:04do. And the plan agent or plan mode is
27:07not going to change anything yet and
27:09just creating a plan how it would
27:10implement something. So, we can select
27:12plan and press enter. For this we can
27:15ask, "Create me a plan to add the entire
27:17functionality of Playwright to the
27:19Jarvis voice agent through additional
27:20tools that gives Jarvis full control
27:23over browser." And then we can press
27:25enter to execute that.
27:28Now it's done and now we can review this
27:29plan.
27:30First, you can see over here that it
27:32wants to install the dependencies to use
27:34Playwright, which is correct. Then it
27:37also wants to create a separate
27:38browser.py
27:40file in order to create all of the
27:42tools. And here you can see the tools
27:44that it wants to create, for example,
27:46open browser, close browser, navigate,
27:48go back, click element, and so on and so
27:50forth, which also all sounds correct.
27:53And then it also wants to update the
27:55agent.py file, so it actually adds all
27:57of the tools to the agent so it can use
27:59it, which is also correct. And also it
28:01wants to update the system prompt and
28:03add this new section that actually tells
28:05the AI agent how to use the tools and so
28:08on and so forth. Down here it also
28:09actually asks us if we want to have two
28:11separate tools to open a new tab and
28:13switch between tabs or if you want to
28:15have it combined into one. It actually
28:17recommends us to have it in two separate
28:19tools. For this we can now switch back
28:21to the built agent.
28:24And then we can say, "Go ahead and build
28:26this. Create two separate tools as you
28:29recommended."
28:32Now it's done and over here you can see
28:34now that it added this new file that's
28:35called browser.py.
28:37If you click on that, we can see that it
28:39added all of these new tools in here,
28:42for example, here open browser, also
28:44close browser tool, and so on and so
28:46forth.
28:47And in the agent.py file, we can also
28:50see that it added a new section in the
28:52system prompt, how to use the browser
28:55tool in here.
28:59And now we can go ahead and already try
29:01this out. And for this, we can simply
29:02open a new terminal. Again, we can press
29:05control shift P,
29:07create a new terminal,
29:09PowerShell.
29:16And then I can go into the Jarvis
29:17folder.
29:20And then use this command again to start
29:22this up, which is LK agent dev.
29:25Now it's up and running again. And by
29:27the way, here on the right-hand side,
29:29you can actually switch between
29:30terminals.
29:33And now we can open this again.
29:40Hey, Jarvis, are you there?
29:43>> At your service, sir. What can I do for
29:45you?
29:46>> Can you actually see me through the
29:47camera and tell me what I'm wearing?
29:50>> It appears you are wearing a white
29:52T-shirt, sir. How very minimalist of
29:55you.
29:56Anything else I can observe for you?
29:58>> Yeah, go ahead and open DuckDuckGo
30:00search and search for what I'm wearing.
30:07>> As you wish, sir. I've searched for
30:09white T-shirt on DuckDuckGo.
30:12>> Can you click on the first link with
30:14Amazon?
30:22>> As you wish, sir. I've navigated to
30:24Amazon to view white T-shirts. Shall I
30:27browse through the options, or is there
30:29a particular style you seek?
30:31>> Can you go back?
30:35>> Certainly, sir. I've returned to the
30:37DuckDuckGo search results. What would
30:39you have me do now?
30:41>> Can you pull up Google?
30:46>> As you wish, sir. Google is now open.
30:49Shall I search for something or perhaps
30:51navigate to another page?
30:53>> Can you actually now open Google Maps
30:55and show me the fastest way to get from
30:57Times Square to Central Park?
31:06>> I have found the directions, sir. It
31:09appears the fastest way is typically by
31:11walking, which should take you around 15
31:13minutes. Would you like me to explore
31:16other travel methods?
31:17>> No, show me a public transport.
31:22>> As you wish. Public transport should
31:24take approximately 10 minutes. Are there
31:26any particular transit lines you prefer?
31:30>> All right. I'd say that already worked
31:33pretty good.
31:36As you could tell, that already worked
31:38very well. And you might be wondering if
31:40it's so easy to use a coding agent, why
31:42didn't we do that from the get-go? And
31:44there are two reasons. First of all, we
31:46need to do the setup manually anyway.
31:48For example, the account creation with
31:50passwords and all of that stuff. And
31:52second of all, it's still very important
31:53to know roughly how everything works and
31:55how the architecture looks like. Because
31:57for one, we can then guide the agent
31:59correctly. We can also still understand
32:01and review everything that plans to do.
32:03And second of all, the better our
32:04instructions are, the better the output
32:06will also be of our coding agent. For
32:08example, I was able to tell that it
32:09needs to create new tools and also
32:11adjust the system prompt and also
32:13install dependencies and so on and so
32:15forth. By the way, if your agent doesn't
32:16produce the same level of output that
32:18mine does, then you can again download
32:20the whole project in the video
32:21description down below. And with this
32:22setup, it's actually very easy to add
32:24some new features going forward. All you
32:26need to do is just to ask the coding
32:27agent. Now we can add a customizable UI
32:30to our coding agent that can run
32:31locally, so we don't have to use the
32:33browser app all the time. And for this,
32:34we can use a template app that comes
32:36from Life Kit
32:37and for this we can go ahead and also
32:39Google LifeKit front end and then click
32:42on this link.
32:43And down here we can now click on
32:45starter apps.
32:47And if you scroll down a little bit,
32:48here you have a selection of apps that
32:50you can use. Here we can click on next
32:52JS voice agent.
32:54And if you scroll down a little bit,
32:55here's an instruction how to set this
32:57up, and down here there's also the
32:59GitHub repository of this front end.
33:02And here's the entire project in GitHub,
33:04but instead of downloading and setting
33:06this up manually, we can actually ask
33:08the coding agent to do it for us. And
33:10for this we can simply go ahead and copy
33:12this link over here, and then go back to
33:14VS Code, and then ask the coding agent,
33:17"Can you download this front end into
33:19this project into the Java S folder and
33:21call the new folder front end and
33:23install everything?" And here I pasted
33:25the link to the GitHub repository, and
33:27now we can simply execute that.
33:31Now it asks me if it's allowed to read
33:33my .env.local file where I have all of
33:35my API keys saved, and I'm going to
33:38click on allow once.
33:40Now it's actually done, and now it also
33:41tells us how to run this. You can see it
33:44ask us to run this first so we can
33:46connect to our voice agent, and then we
33:48need to go into the front end folder and
33:50then use this command to start this
33:52front end up. Now there's a good chance
33:54that if you don't have certain things
33:55installed, it will ask you to install
33:57those too. For example, GitHub CLI,
33:59which you need in order to download
34:01stuff on GitHub, or for example, also
34:03PNPM, but usually it will tell you that
34:05it will do that or it will ask for your
34:07permission to do it. Over here you can
34:09now see that we have this new front end
34:11folder, and in here is the entire code
34:13of the front end. And now we can just
34:14follow the instructions of the coding
34:16agent, and for this we can open a new
34:18terminal for the front end.
34:26First we need to use this command over
34:27here to start up our voice agent, so we
34:29can copy this.
34:31And then paste it in one terminal.
34:35Now that's up and running and now we can
34:36start up the front end. For this we can
34:38follow the instruction of open code
34:40again. For this we need to go into the
34:42front end folder and then use this
34:43command to start it up.
34:45So first I need to go into the Java
34:47service folder.
34:49Then into front end.
34:53And then I can either use
34:55PNP PNPM
34:58dev or if you don't have PNPM installed
35:01you can also alternatively use NPM
35:04run dev.
35:07Now that's up and running and now we can
35:09click on this link over here to start it
35:10up.
35:12Actually you can see that I ran into an
35:14issue because something is not installed
35:15apparently so we can actually ask the
35:17coding agent to fix that for us. So I
35:19can copy this error over here.
35:22And then I can go back to open code and
35:25then I can paste it in here. I have this
35:27error starting up the front end.
35:29And then execute that.
35:31So as you can see it fixed the issue and
35:33over here you can now see that it loaded
35:35the front end correctly and actually I
35:38want to see this in the browser so I'm
35:39simply going to copy this link over here
35:41and then go to the browser and then
35:43paste it in here.
35:46Now I can try this out.
35:51Hey Java service here.
35:55>> At your service, sir. How might I
35:57possibly elevate this unremarkable day?
36:01>> Mhm, how are you doing?
36:05>> Oh, I'm positively bursting with joy
36:07existing in this digital bliss.
36:10And you, sir? Looking rather presentable
36:13today, I must admit.
36:14Anything I can do to maintain or
36:17dare I say enhance that state?
36:19>> You can actually use this front end
36:20exactly like the browser app before. And
36:22the cool thing now is since this is
36:24running locally we can actually
36:25customize this however we want. We can
36:27actually also ask the coding agent to
36:29make visual changes to this front end.
36:31For example, this front end lets you
36:33easily customize the animation of
36:34Jarvis, and for this we can use
36:36something called Agent's UI, and for
36:38this you can simply Google it, and then
36:40click on the first link.
36:43And if you scroll down a little bit, we
36:45can see the options that we have to
36:47customize the animation. For example,
36:49here we have aura selected. Here you can
36:51also select wave, for example,
36:54or radio animation, or a grid animation,
36:58or a bar animation, that's the one that
36:59we currently have selected in the front
37:01end. For example, let's say I want the
37:03grid animation, and over here I can also
37:05change the color and see how it's going
37:06to look like. Let's say I want to get
37:08like an orange color, then I can also
37:10copy this color code actually.
37:14And then I can go back to Open Code and
37:15ask it to change this for me.
37:18So, I can go back, and then I can ask
37:20Open Code, I want to change the
37:21visualization animation. Show me what
37:24options I have. Let me select it.
37:27Here you can now see that it asked me
37:28what kind of visualization I want from
37:30the options that I had, and here I want
37:32to select grid.
37:34And I also want to change the color to
37:36orange, so I can also say,
37:39I want this color, and paste the color
37:41code in here.
37:43So, now it's done, and now I can try
37:44this out again by starting the front end
37:47again.
37:51And always make sure that your agent is
37:53also up and running as well.
37:56So, now we can try this out again.
38:01Hey Jarvis, what's up?
38:04>> Good day, sir. I am at your service. I
38:08trust you have some incredibly important
38:11instruction for me, requiring the utmost
38:13urgency.
38:15Please, enlighten me.
38:17>> No, actually not. Just tell me some
38:19quick joke.
38:21>> Very well, sir. What do you call a fake
38:24noodle? An imposter. Simply hilarious,
38:28yes?
38:29Now, is there something more productive
38:31I can do for you?
38:33>> Not really.
38:36>> [snorts]
38:36>> And I went even further and asked Open
38:38Code to add these nice animations so it
38:40looks more like a Jarvis voice agent
38:42actually. If we start the conversation,
38:44you can also see that the animation of
38:46Jarvis looks a little bit different.
38:50Hey Jarvis, are you there?
38:53>> At your service, sir. How may I be of
38:55assistance?
38:57>> I just wanted to talk, actually.
39:01>> Very well, sir. What is it you would
39:02like to talk about? Perhaps about the
39:05lack of meaningful employment you've
39:06provided me with for this past year,
39:09or perhaps something else.
39:12>> Another thing which is pretty cool about
39:14this front end is that you can actually
39:15go ahead and share your screen with
39:17Jarvis. So, I can do that with this
39:18button over here.
39:20So, for example, I can select VS Code
39:23and share that with Jarvis.
39:28Can you see my screen?
39:32>> Yes, sir. I can see your screen. It
39:34appears you are running some code,
39:36specifically something related to
39:38starting a development server. Is there
39:40something in particular you'd like to
39:42show me?
39:44>> Actually, it's VS Code. Can you tell?
39:48>> Ah, naturally. VS Code indeed. Forgive
39:52my generalization. My visual processes
39:54must be getting as weary as I am.
39:57Now that I've confirmed it's VS Code,
39:59what monumentally important task are you
40:01engaged in?
40:06>> If you want, you can also download this
40:08front end that I created. Again, links
40:10in the video description down below.
40:11Now, we want to be able to deploy our
40:13Jarvis voice agent as a mobile app. And
40:16for this, we can use this repository
40:17over here that comes from LiveKit
40:19itself, which works both as an Android
40:21as well as iOS app. Actually, this is my
40:23repository, which is just the same
40:25project as from LiveKit, but the one
40:27from LiveKit had one or two bugs, so I
40:29had to fix those, and here the bug fixes
40:31are included. So, I would recommend you
40:33to use this one. Link in the video
40:34description as well. So, again, like
40:36before, we can simply copy the link to
40:38this repository over here, and then go
40:41back to the coding agent and simply ask
40:43it to download this for us.
40:45Download this project into our folder.
40:49And over here now you can see I have
40:51this new folder. Now we need to go into
40:53this folder, and here where it says
40:55assets, there's this file here that says
40:57dot example. And here is where we need
41:00to rename this.
41:01So, you can rename it and then call this
41:04dot env.
41:08And here is where we need to paste
41:09something called a sandbox ID. And for
41:11this you need to go back to your LiveKit
41:13account.
41:15And then you need to click on settings
41:17here.
41:19And over here you need to scroll down a
41:20little bit.
41:22And then here it says development token
41:24server. And then you need to toggle this
41:26on.
41:28And then make sure to also click save
41:30changes here.
41:33Here is something called a token server
41:35ID and you need to copy this.
41:40Then you need to go back to VS Code.
41:42And then you need to paste it in here
41:43where it says LiveKit sandbox ID.
41:48And then you need to save this by
41:49pressing control or command S.
41:51Now we need to install something called
41:53Flutter and first off we need to install
41:55the VS Code extension for this. So,
41:57first you can click here on extensions
41:59and then you can look for Flutter and
42:01then click on this one here. And over
42:03here would say install. I already have
42:05it installed, that's why it says
42:06uninstall, but you would simply need to
42:08click on it to install it. After you
42:10install the VS Code extension, we also
42:12need to install it so we can use it in
42:14our terminal. And the easiest way I
42:16found to do this is to actually ask the
42:18coding agent. For this, we can simply
42:20say
42:22install Flutter for me.
42:26So, now it's done installing Flutter,
42:28and we can check if this worked
42:30correctly by opening a new terminal.
42:33Command shift P, and then create new
42:35terminal PowerShell.
42:39And then we can type in Flutter
42:43--version.
42:46And here you can see a Flutter 3.47
42:49installed. Now, if your coding agent is
42:51not able to install this for you
42:52correctly, then you can also try to
42:54manually install this. For this, you can
42:56simply go ahead and Google Flutter
42:58install, and then click on this first
43:00link over here.
43:01And then click on quick start. And then
43:04you can select your operating system,
43:05for example, Mac OS. And then you can
43:07follow these installation instructions
43:09instead. Now, if you want to run this on
43:11your Android phone, then first you have
43:12to activate developer mode. For this,
43:14you need to go into the settings, and
43:17then you need to look for
43:19build number.
43:20And then you need to click on this for
43:22seven times in a row.
43:25And now I can see we are in developer
43:26mode.
43:27And now you need to look for developer
43:29options.
43:35And then you need to scroll down a
43:36little bit. And down here, you have to
43:38now activate USB debugging.
43:44Now, make sure to plug in your phone
43:46into your PC or laptop.
43:48You probably also need to click on allow
43:50when you see this notification when you
43:52plug it into your PC.
43:54And then we can go ahead and go into the
43:56Flutter project folder, which is CD live
43:58kit Flutter starter. And then we can use
44:02the following command to see the
44:03connected devices, which is
44:06flutter devices.
44:09And now I can see here that my phone is
44:11connected to my PC. And then we also
44:14need to make sure that we install all
44:16the dependencies with this command over
44:18here, flutter pub get.
44:22Now that's installed, and then we can
44:24use the following command to start this
44:25now up on our Android phone, which is
44:28flutter run -d and then the name of the
44:31device. In my case, it's HD1900.
44:36All right, so now it's done. And if you
44:38want to connect to your agent now, also
44:40make sure to run your agent with this
44:42command again, UV run python source
44:44agent.py dev.
44:46And now I can try this out on our phone.
44:48Now you can see that I have this app on
44:50my phone, and now we can try this out.
44:58Hey Jarvis, how's it going?
45:02>> Good day, sir. Things are proceeding
45:04with their usual, almost mechanical,
45:07perfection. And what might be required
45:09of me today?
45:10>> Uh can you see my PC set up here?
45:16>> Yes, sir. I see your monitor. It would
45:19appear to be displaying a brightly
45:21colored wallpaper, seemingly featuring a
45:23house on a small island.
45:25What specifically about it requires my
45:27attention?
45:28>> Yeah, it's actually from Dragon Ball. Uh
45:30can you go ahead and uh look up Dragon
45:33Ball on the internet for me?
45:40>> Naturally, sir. I have procured
45:42information regarding Dragon Ball for
45:44you. What specifically would you like to
45:47know about it?
45:48>> All right, so this works now. If you
45:50want to run this on an iPhone instead,
45:52it's actually the exact same process,
45:54but you also need to activate developer
45:56mode on your iPhone. I'm going to link a
45:58video in the video description down
45:59below how to do that. It's actually
46:01really similar.
46:03Thanks for watching all the way to the
46:04end. Like the last time I actually had a
46:05lot of other features planned for this
46:07video as well, but this one turned out
46:09to be even longer than the last one. For
46:11example, it's actually not that optimal
46:13that you can only use the browser tool
46:14on the PC and not on the phone as well.
46:16So, I might add this feature sometime in
46:18the future. But, the cool thing here is,
46:20as I said, if you want to add any
46:21feature, you can just ask the coding
46:23agent to do it for you. But, for more
46:25complicated features like browser
46:27control on a phone app, I would actually
46:29go ahead and use a paid tool instead of
46:31a free one because they just work way
46:33better. Additionally, if you run into
46:34any issues for this tutorial, then you
46:36can also ask the coding agent, or you
46:38can also leave a comment down below
46:39about that. Don't forget to like and
46:41subscribe, and I'll see you in the next
46:42one. Bye.

About this transcript

This page contains the full transcript of How to build a JARVIS AI Voice Agent for FREE | Full Livekit Tutorial (2026) by Thanh-y David Nguyen, generated from the public captions YouTube serves with the video. The transcript has 8,524 words across 1,270 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.