YT YouTube2Text

YouTube2Text → Transcripts → Learn AI Engineering in 2026 | Build a Full Stack AI Websites with Agents

Learn AI Engineering in 2026 | Build a Full Stack AI Websites with Agents — Transcript

by JavaScript Mastery · 20,575 words · 2,963 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00Watch this. I ask for an entire scraping
0:03pipeline in one sentence. The AI reads
0:06my rules, writes its own implementation
0:08prompt, and asks me, "Is this good to
0:10execute?" I say, "Yes, and it builds the
0:14whole thing." That's the video. So, let
0:17me show you how to set it up. What if
0:18your agent didn't just build your
0:20feature, but wrote the prompt for it,
0:23too? Be honest. How many times have you
0:26gone back and forth with AI on the same
0:28feature and still not gotten what you
0:30wanted? You know what you needed, you
0:33just didn't know how to say it in a way
0:35that AI actually understood. That one
0:37thing kills more AI projects than
0:39anything else. So, hi, I'm Adrian. And
0:43in this video, you'll flip that problem
0:45around. You'll make the AI write its own
0:48implementation prompts. You'll then just
0:50review and approve them, and it builds
0:52the entire app from there. And to prove
0:55this works on something genuinely hard,
0:57you'll build skew, an AI powered news
1:00platform that pulls in real news and
1:03show you just how biased every article
1:05is before you even open it. So here's
1:08what you'll build. A home feed of real
1:11scraped articles with a bias metric
1:14right on every card. a news details page
1:17with the full article AI summary
1:19sentiment framing percentages related
1:22articles matched by meaning instead of
1:24keywords using PG vector semantic search
1:28authentication with middleware protected
1:30routes and redirects and a fully
1:33automated pipeline that when it scrapes
1:35the data using oxyabs it then filters
1:38saves and analyzes fresh news every hour
1:42you'll build it with NexJS for the full
1:44stack app Superbase the built-in a
1:47weekend back end that is also super
1:49powerful to run production apps using
1:52Postgress PG vector and the entire data
1:54layer. Clerk for complete authentication
1:57and user management so you're not
1:59rebuilding what you've built 10 times.
2:02Oxyabs an awardwinning web data platform
2:05for scraping real news sites that
2:08ensures uninterrupted access to public
2:10web data. Posthog for analytics and
2:13session replays. So you ship products
2:15and not just code. And you can get
2:17access to it directly through their
2:19Slack integration. I'll show you how
2:20that works. Versel AI SDK for structured
2:24AI analysis for every article. And of
2:26course, Tailwind for styles and
2:28TypeScript for safe code. All the in-
2:30demand industry tech and every feature,
2:33every route and every component of it
2:35will be built by the agent. But the app
2:38is not the real takeaway. Skew is just
2:41how you learn it. The real takeaway is
2:44the workflow. An approach I call vibe
2:47engineering where you define your
2:48product, write your agents and then let
2:51AI draft its own prompts. Approve them
2:54and build in small focused steps. Master
2:58that once and you can build any app
2:59after this. Same system, different idea,
3:03same results. And to make it stick, I
3:06prepared a free Vibe engineering guide
3:08that lays out the entire workflow step
3:11by step so you can apply to your own
3:13projects, get that job, get that gig, or
3:16just become better developer working
3:17with agents. Link is in the description
3:20so you can go grab it. And if you want
3:22the version where the AI does even more
3:25of this for you, planning and reviewing
3:27its own work and handling the enterprise
3:29stuff, which doesn't fit in this video,
3:32that's what I'm building into the
3:33Agentic Engineering course. It's not out
3:36yet, but the weight list link is in the
3:38description, and the founding price goes
3:41to that list first. Check it out. And
3:44without any further ado, let's dive
3:46right in.
3:55Okay, so you know how to code with AI.
3:58Small prompts, one task at a time,
4:01agents MD holding some project rules.
4:05That works really well for most apps. A
4:08marketing site, uh, a dashboard or a SAS
4:11tool with a clear scope. You keep the
4:13momentum and you can always see what
4:15just happened before moving forward. But
4:18at some point you hit a project that
4:21breaks that flow. Not because the
4:23approach was wrong, but because the
4:26project is complex enough that a short
4:29prompt leaves out too much. A scraper
4:32that feeds a processor that feeds an AI
4:35step that feeds a UI. If one step is
4:39wrong, everything after it is wrong,
4:41too. So, you naturally write a longer
4:44prompt to compensate. And now you're
4:47spending 20 minutes writing a prompt
4:49before you build anything. But the whole
4:51point of short prompts was to move fast.
4:54You've probably been on the other side
4:55of this, too. Frontloading everything.
4:58Nine markdown files, a whole planning
5:01system to fill out before the project
5:03could even start. Lots of context for
5:05AI, but you're doing most of the
5:08thinking the AI could have done for you.
5:11So, the question worth asking is this.
5:15If the AI is smart enough to write code
5:18from your instructions, is it not also
5:21smart enough to write the prompt from
5:23your instructions? Because these models
5:26have gotten genuinely capable, point one
5:29at a skill file or a piece of
5:31documentation and tell it what has to
5:34happen. And it doesn't need you to
5:36pre-true every technical decision. It
5:39can figure out the steps, identify the
5:41files that need to change, and write the
5:43implementation prompt. itself. What it
5:46needs from you is direction, not a
5:49script. Direction. So what are we
5:53building? What must they separate? What
5:56tools do we use? When to ask before
5:58assuming and how to structure its
6:01outputs so that you can verify it.
6:03That's the whole model. You stay in
6:05control of the decisions and the AI does
6:08the planning work and then it builds
6:10what it planned. That's what I like to
6:13call vibe engineering. We always start
6:16in that same file that AI works with.
6:19But for a complex project, it holds one
6:22thing more than just your stack and
6:24folder rules. It describes how you want
6:28features to be built. Not what features
6:30to build. The thinking process. When you
6:34sit down and build something yourself,
6:36you don't start typing. You ask, "What
6:39does this feature do? What data does it
6:41need? Where does that data come from?
6:44And what could go wrong? That process is
6:47what goes within the agents MD file, not
6:50the answers, but the process. So the AI
6:53runs it on every task before touching a
6:56file. Concretely, the file covers the
6:59workflow steps, what the product is and
7:02is not, how the layers relate, which
7:05tool owns the job, the data model, the
7:07API method contracts, security rule,
7:10code quality preferences, and a fallback
7:12rule for anything the file doesn't
7:14cover. And since this is a lot to
7:16remember, in the description down below,
7:18I'll leave a free guide that you can
7:20always refer to and just get it set up
7:22for your own projects. But only three of
7:25these things do the heaviest lifting. So
7:28here's the idea behind them. The out of
7:31scope list. See, AI agents love adding
7:34features that just sound useful.
7:38Comments, bookmarks, notifications,
7:40dashboard. They're not bad ideas, but
7:43they're just not for your plan. So write
7:46down what's out and they will stop
7:48appearing. The data model with hard
7:51rules. define once what the article
7:54needs such as an image URL or a
7:57published date before it can be saved
7:59and then the AI will enforce it in the
8:01scraper in the validation and in every
8:04feature that touches that article
8:07written once and enforced everywhere.
8:10And finally, the fallback rule. It's
8:13just one numbered list at the bottom for
8:15situations that the file doesn't cover.
8:18Build the smallest thing possible. Ask a
8:20focused question if unclear. Save a
8:23prompt, get approval, implement it, and
8:26share the test steps. This is what stops
8:28the AI from making a large independent
8:31decision in a situation that you didn't
8:33anticipate.
8:35And you'll see a full agent MD file for
8:38the news app we're building in this
8:40video line by line when we set it up.
8:43Every section with all the reasoning
8:45behind it. But yeah, once the agents MD
8:48has the project rules, we then add
8:50skills on top of it. A skill is a
8:54dedicated file for one tool. The clerk
8:56skill holds how to implement clerk
8:59properly. Superbase skill holds query
9:02patterns and rowle security. The
9:04scraping skill holds how that service
9:06works. So agents is about your project
9:11and the skills are about the tools. So
9:14when a library updates, you update one
9:17skill file. When you start a new project
9:19with the same stack, you bring the
9:21skills and write a fresh agents MD. In
9:25your prompts, you just reference what's
9:27relevant, like implement clerk
9:29authentication using the clerk skill.
9:32One sentence, agent MD has the project
9:34context, the skill has the tool
9:37knowledge, and you wrote neither into
9:39the prompt. And these skills are
9:41something that you can find for the
9:43tools you'll use or you can build them
9:45yourself. For the agentic engineering
9:47course, I'm building a full custom skill
9:50set that goes further skills that
9:52interrogate you about a feature before
9:54planning it. Push the grunt work to
9:57cheaper models to save tokens and review
10:00the finished code with a different model
10:02than the one that wrote it. That's going
10:04to ship with a full course. I'll leave
10:06the link down in the description if you
10:07want to join the weight list. But the
10:09thing that makes Vibe Engineering, the
10:11process that I'm going to teach you in
10:12this video, different is that AI writes
10:16the prompt before it writes the code.
10:19Before touching any file, the AI will
10:22write a prompt, save it as a markdown
10:25file in a prompts folder, and ask you
10:28whether it's good enough to execute. And
10:31it's not a vague summary. It's a
10:33specific document. It contains a goal
10:36which is a one clear sentence. What it
10:38read like which skills and existing
10:40files. What it assumes like small
10:43decisions that it can resolve on its
10:45own. What files will change containing
10:48the exact list. What it'll build based
10:50on specific requirements. The security
10:53requirements. So what's the server side?
10:56What the route rejects. Acceptance
10:58criteria checks to run like type check
11:01or lint. And finally, how do you verify
11:04it with step-by-step instructions? That
11:07last one matters most. For an API
11:10feature, it's exact curl commands. For a
11:13UI feature, it's what to click. Or for a
11:16database migration, it's the SQL code to
11:18verify. The AI isn't just telling you
11:21that it built a thing. It's telling you
11:23how to confirm it yourself. And it wrote
11:26that down before building it. And then
11:28it asks you just a single question. It
11:31says, "I prepared the implementation
11:33prompt in this folder. Is it good enough
11:36to execute?" You read it. If it's wrong,
11:39you say so. If it's good, you approve.
11:42And then it builds. You review the
11:45prompt, not the code, and you catch
11:47problems before they become features
11:49that need to be undone. And when the AI
11:52hits a decision that the agents MMD
11:55doesn't cover, something that could
11:57reasonably go two ways, the rule is
12:00always ask before assuming, not midbu,
12:04but before the prompt. That way, the
12:07decision gets made by you at the right
12:10time instead of by the AI in the wrong
12:13place. And that allows your everyday
12:15prompts to stay short. All of this setup
12:18has one payoff. Implement clerk
12:21authentication using agent skills.
12:23Implement the database schema and data
12:25access using superbase skills or
12:28implement the homepage from the attached
12:30design. One or two sentences each. The
12:33AI reads the context it needs, plans the
12:36feature, saves the prompt file, asks for
12:39approval, and finally builds it. That
12:42way you're not writing a long prompt for
12:45every feature, but rather you're
12:47maintaining a file that makes long
12:49prompts unnecessary. And then you simply
12:53build in order. Not because anything
12:55stops you from building out of order,
12:57but because the wrong order means
13:00building against uncertainty. UI
13:02decisions can happen before you know
13:04what the data looks like and
13:06integrations can happen before the
13:09database that they depend on actually
13:11exists. So in skew our news application
13:16the order is like this. Design system UI
13:20clerk o superbase schema oxyab scraping
13:24AI analysis PG vector for related
13:27articles and then theuler and cron that
13:31run it all automatically. One prompt per
13:34step each approved before
13:36implementation. That way you'll see how
13:38each one of these steps connects as we
13:41go. And if you've been following the
13:43channel, you know that there's also a
13:45third approach that I've taught you. The
13:48first one was frontloading everything.
13:51Six markdown files, long planning
13:53documents, full prompts before you write
13:56any code. That's lots of context, but
13:59most of the planning work was done by
14:01you. Then there's practical vibe coding
14:04with short prompts, agents MD, one task
14:07at a time, which is great for most apps,
14:10but messy on anything complex enough
14:12that a short prompt leaves out. And then
14:15this video is about vibe engineering,
14:18and it sits somewhere in between. You
14:20write the agents MD and a small set of
14:23skills. Your prompts stay short and then
14:26the AI writes the detailed prompt for
14:28you. That way the AI does more of the
14:31planning work and you control more of
14:33the decisions. The right things are
14:36written down once and used everywhere
14:38and that covers some more complex
14:40projects. But there's a level past this
14:43and it's the one the agentic engineering
14:46course is built around. It follows the
14:48same core idea. You own the decisions
14:51and the AI does the work but the skills
14:54do far more of the thinking. They
14:57interrogate you before they plan. They
14:59pull in each framework's own current
15:01agent skills, so you're not getting
15:04plausible but dated code. And they
15:06review the result with a different model
15:09than the one that wrote it. Plus, what
15:12this video can't fit, on boarding a
15:14legacy codebase that you didn't write,
15:16recovering a project that has gone
15:18sideways, or running agents in parallel
15:21without them stepping on each other.
15:23You'll feel where the by hand version
15:26runs out of room in this build. The
15:28course picks up right there. It's not
15:31out yet, but the wait list is linked in
15:34the description. The founding price will
15:36go to the devs that are on that list
15:38first, and I'd very much like to see you
15:41there because the official JSM skills
15:44are something that we've been working on
15:45for the past year. And in the course,
15:48I'll teach you how to use them to become
15:50an agentic engineer. But in this video,
15:53skew is the project. The workflow is
15:56what you're learning. Do this once and
15:59you can do it for any web app at any
16:01level of complexity. Define how the
16:03parts connect. Write the agents MD file.
16:06Add the skills and then work through it
16:09one prompt at a time. The project
16:12changes every time, but the way of
16:14working stays the same. and you'll
16:17direct the AI, but the AI still can't
16:21create your accounts or hand you your
16:23own API keys. So, before we build, let's
16:27get four of these ready. Each one is
16:29free, and each is what makes a specific
16:32part of our application work. We'll use
16:34Clerk for authentication, and it's more
16:37than just a login box. It's a complete
16:40user management and billing allowing you
16:42to drop in signin and signup components
16:45and a stack of OOTH providers. The free
16:48hobby plan covers 50,000 monthly active
16:52users. So you can build and launch
16:54without paying anything. So since we use
16:57it in this application, click the link
16:59down in the description, create your
17:00account, and then create a project that
17:03you're going to use for this app. We'll
17:05also use Superbase for the database.
17:08It's what I would call a built in a
17:11weekend backend that still scales to
17:13millions with Postgress, instant APIs,
17:17rowle security, storage, real time, and
17:20edge functions. All of it on a free tier
17:23that's great for building and testing.
17:25The app we're going to build today uses
17:27it as the source of truth for sources,
17:30articles, AI analysis, scrape logs, and
17:34scheduleuler runs. and its PG vector
17:36extension is what powers the related
17:38article search later on. So once again,
17:42click the link down in the description,
17:43create your free account, and then a
17:45project within it for the app we'll
17:48start building right now. We'll also use
17:50Oxyabs for scraping. It's an all-in-one
17:54web data platform with a web scraper API
17:57on a free trial and a freeuler for
18:00recurring jobs. New sites actively block
18:03scrapers. Oxyabs manages proxy networks
18:07and geocmplexity legally to deliver
18:09clean data and it's been awarded the
18:11best proxy service. We're going to use
18:13it in our app today to pull real
18:16articles off real news homepages and
18:19then keep them coming in automatically.
18:21It can collect real-time data from
18:23almost any public website, scale
18:26effortlessly without you having to
18:28manage those browsers, servers, and
18:30proxies, and integrates with tools that
18:33you already know like Puppeteer,
18:34Playright, or workflows like N8N. And it
18:38even has MCPs for Claude and Cursor. On
18:41top of that, it offers very low latency
18:44scraping, 24/7 support if you need it,
18:47and has been recognized as one of the
18:48best proxy services. Make sure to go to
18:51oxyilabs.io/javascript
18:54to get a free trial with up to 2,000
18:56scraping results free and no credit card
18:59required. Plus, you can use my code
19:02JavaScript for 20% off all of their
19:04plans. I'll leave the link down in the
19:07description so you can create an account
19:09and follow along nicely. And we're using
19:12Post Hog for analytics that act on
19:15itself. Post Hog will capture how people
19:18actually use your app, including
19:20analytics, session replays, and feature
19:23flags. But it's moving past dashboards
19:26into what they call self-driving agents
19:29that read your product data and open
19:32pull requests to fix what they find.
19:34Plus, a Slack app that you can tag to
19:37ask data questions or ship a change
19:40without even leaving the thread. It's
19:42free. up to 1 million events and 5,000
19:45recordings a month. We're gonna use it
19:47in our app today so you learn how to
19:50ship products, not just the code, and so
19:52you see where the whole industry is
19:54heading. So go ahead and create that
19:56account right now by clicking the link
19:58in the description. With the accounts
20:00created, as soon as a dedicated lesson
20:02on installing it comes in, we can just
20:05wire its own service at the right
20:07moment. So having them ready means that
20:09we never stop the build to go sign up
20:11for something. Once again, all the links
20:13are in the description. And with that
20:15done, let's dive right into building.
20:20Let's start from the bare beginnings by
20:23setting up a new empty project. Now, I
20:26created a new folder on my desktop,
20:28dragged and dropped it into my editor,
20:30and you can just open up an integrated
20:32terminal right here. The goal is to set
20:35up a base Nex.js JS application and you
20:38can do that by running the command mpx
20:41create next app add latest and use that
20:44slash if you're already in the
20:46repository you want to create the
20:48project within. So simply press enter.
20:51It'll ask you whether you want to
20:53install it using the recommended
20:55defaults uh using TypeScript ESLint
20:57React compiler. Yep, we definitely want
21:00to do that. And it'll now proceed with
21:02installing it. So give it about a minute
21:05and as soon as it is done you should be
21:07able to see a couple of files and
21:08folders right here. But let's go ahead
21:10and run the project first by running mpm
21:14rundev which will spin it up on
21:16localhost 3000. Now in the latest
21:19versions of VS Code you can also just
21:21click it and it'll open it up within an
21:23integrated browser. So in this video
21:26you'll see me use this as well as the
21:28browser. Either way works. Here we also
21:31get some additional functionalities like
21:33targeting specific elements and then
21:35opening them up within the agentic chat
21:37so you can directly change them. But
21:38yeah, this is the app the default NexJS
21:41screen. So now let me explain the
21:44generated project briefly. Most
21:46important stuff is contained around the
21:48app folder. So right here you'll see a
21:51page.tsx which is that default page. So
21:54for now we can remove this default
21:57content. Replace it with an RAFCE uh
22:00which is going to say home. This simply
22:02is a basic React arrow function
22:05component. And then if you go back to
22:07the homepage you should be able to see
22:09just a plain text that says home.
22:11Alongside the page there's also the
22:14layout which loads specific fonts and
22:16CSS and then just simply renders
22:19whatever we're passing into it. There's
22:20also the globals.css CSS which includes
22:22the Tailwind CSS configuration as well
22:25as some general theming and styles.
22:28There's the public folder for all of the
22:30assets and images, a package JSON for
22:34scripts and dependencies, and there's
22:36also a TS config for setting up
22:38different TypeScript options. But yeah,
22:41the best way to learn about a new
22:43codebase is using AI. whether it's a new
22:46project that you want to understand the
22:47structure of or even more importantly if
22:50it's an older project that you just got
22:52onboarded to. So I think this would be a
22:55nice point to set up our AI agent and
22:58that I tell you a bit more about the
22:59workflow that we'll be using for the
23:01rest of this project. See VS Code has a
23:04very nice built-in chat feature that you
23:08can open if you press command shiftB and
23:10then search for chat or you should also
23:11be able to see it right at the top right
23:13here. This is the default built-in chat,
23:16but you can also install different
23:18extensions. For example, I have Claude
23:21code installed right here. So, I can
23:23very quickly initiate new chat
23:24conversations with Claude. If you head
23:27over to the extensions tab and then
23:29search for something like Codex, you'll
23:32also be able to see a Codex or maybe
23:35like a Gemini extension if that's what
23:37you prefer to use. Either way is fine
23:39with me. The process that I'm about to
23:41show you works with all of the different
23:44agents. It's more about the tools that
23:47you put around the agent that make it
23:49work. So, whatever you find works best
23:52for you, whatever you're paying for
23:53already or whatever is within your
23:55budget, you can totally go for that. In
23:58this video, I'll be using Claude Code,
24:00and we're going to use it with a default
24:02model of Opus set to the effort of
24:05medium. And you'll mostly see me use it
24:08in a full screen mode. So let's test it
24:11out with a super simple prompt by
24:13telling it to explain the generated
24:16Nex.js project structure and keep it
24:18beginner friendly but technical. This
24:21should be pretty simple. And here we can
24:23see a bit about how the agents actually
24:26work. So it's actually using different
24:27tools like bash commands and reading
24:30specific files. It's thinking and then
24:32giving us a walkthrough of how
24:34everything works. So we get the big
24:36picture, the full information of all the
24:38files. The app folder is where the app
24:41router is. So this is most important
24:43here. We'll be using Nex.js's file-based
24:46routing, loading different fonts, and
24:48then displaying the homepage, the
24:50globals for the Tailwind, favicon, and
24:52here are some config files. And of
24:54course, we don't touch the node modules
24:57or the next folders. The typical mental
25:00model is that a request comes in for a
25:02URL. It next matches it to a page in the
25:05app folder. The page renders inside a
25:08layout and then server components render
25:10in the server. Send HTML and then
25:12Tailwind classes are handling the
25:14styling. Pretty simple stuff for now,
25:16but every good project needs a claude.md
25:20file. That's just an empty one, but if
25:23you want your agents to behave better
25:24within your project, then you have to
25:27have a proper agents and claw.md files.
25:31That way, you're just providing more
25:33information about the codebase and what
25:35you're doing to the agent so it can do
25:37better work. So, let's focus on that in
25:40the next lesson.
25:43In this lesson, we'll create the most
25:46important file in the project, and it's
25:49the agents.mmd file. Currently, it has
25:52just some basic Nex.js agent rules, but
25:55the one that we'll add is going to be
25:57significantly longer. So instead of
25:59typing it, I'll provide you with the
26:01full file. You can find it in the video
26:03kit link down in the description. And if
26:05you scroll a bit down, it is right here.
26:07So you can simply copy the agent MD file
26:10and then paste it right here over this
26:11one. I'm using an extension that allows
26:13me to preview the markdown files in a
26:15bit of a prettier format, but any
26:17markdown previewer will do. This file
26:20tells the agent what our application is,
26:23how it should be built, which tools are
26:26allowed, how features are planned, and
26:28when the AI is allowed to write code.
26:32It becomes the project's operating
26:34system that the AI must read before
26:37every implementation. So instead of
26:39repeating the same rule in every prompt,
26:42we write them once inside of the
26:43agents.mmd file. So let me explain
26:46what's really happening here. And this
26:49might sound simple, but it has a huge
26:51impact on the output quality. Most
26:54people treat AI like generic chatbot and
26:58then wonder why the output is
26:59inconsistent. But large language models
27:02respond heavily to context. So if you
27:05tell the AI it is a generic assistant,
27:08it'll optimize for being helpful. But if
27:10you tell it that it is a principal level
27:12full stack engineer responsible for
27:15implementation quality, architecture
27:17decisions, testing and security, it'll
27:20approach the task differently. This role
27:23establishes expectations before any
27:26features begin. We want the AI to think
27:28about the architecture, maintainability,
27:31testing, security, not about just
27:33generating the code it compiles. We also
27:35define the exact job we expect right at
27:38the top. and it is to understand the
27:40request, use the right project skills,
27:43create a clear implementation prompt,
27:46ask for approval, and then implement.
27:49Notice the order. The plan comes before
27:52the code and our approval comes before
27:55either. And then before the product
27:57section, there's also a reference to
27:59Nex.js docs. This is how Nex.js has
28:02started handling knowledge for agents.
28:04No separate skill for it right now. If
28:06you scroll to the product section,
28:08you'll see that here we're describing
28:10what the product is actually about.
28:12Imagine hiring a new engineer. The first
28:16thing they'd ask is, "What are we
28:18building?" And that's exactly what we're
28:20answering here. In our case, we're
28:22building skew. Skew is an application
28:26that collects real news articles from
28:28configured sources, analyzes them with
28:30AI, stores them in Superbase, and then
28:33displays reader friendly sentiment and
28:36framing insights. But an even cleaner
28:38way of saying what the app is is to tell
28:41it what to build. It needs to build a
28:44homepage where users browse articles as
28:46news cards. A news details page where
28:50users read the full article analysis.
28:52Clerk for authentication. Superbase for
28:55persistence. Oxilapse for scraping and
28:58scheduling. AI article analysis logs. PG
29:02vector similarity search. Versel cron
29:04for automatic scheduling and minimal
29:07responsive UI. Do not overbuild. The
29:10more clearly you define the project, the
29:12less likely the AI is to make
29:14assumptions. So this line, do not
29:16overbuild, might be one of the most
29:18important lines in the file. As you
29:20know, the agents love adding features
29:22that sound useful. Maybe it'll suggest
29:25to add comments, bookmarks,
29:27notifications, recommendation engines,
29:29but none of that is needed as it's not
29:32part of the MVP. You must draw the
29:35boundary. tell it what it needs to go
29:37into the project and what it must skip.
29:40Now, if you go to the workflow section,
29:42this is the main idea of how the AI
29:44should approach building features. You
29:47mention it to read the agents MD to read
29:50the relevant skills, inspect the code,
29:52ask a focused question if something is
29:54unclear. This way, the AI doesn't jump
29:57straight into coding. It first writes
29:59the implementation prompt for our
30:01feature, saves it, and asks us if it's
30:04good to execute. We approve the prompt
30:06before it touches the codebase and check
30:09out the last two steps. After
30:11implementing, it runs the available
30:13checks and then shares the exact steps
30:15to test or verify the feature. After
30:18that, in the skill section, we tell it
30:20which skills to use for the libraries we
30:22decided to include in the project.
30:24Thankfully, all of these modern tools
30:27support working with agents, so they've
30:29already created their agent skills. And
30:31finally, in the prompt sections, we're
30:34going to let the AI write the prompt and
30:37tell it to save each one into the
30:39prompts directory. Each one of the
30:40prompt needs to include a goal, a skills
30:43to be read, existing codebase,
30:45inspected, decisions or assumptions,
30:47files likely to change, and all the
30:49other stuff which I believe is important
30:51for it to execute the task properly. For
30:54the UI, it'll also include things such
30:56as visual interpretation, layout,
30:58typography, and more. And feel free to
31:00take this agents.mmd file because you
31:03can use it for any of the upcoming
31:04projects you create. In the architecture
31:06session, we keep the layers separate. On
31:09the website, we have the pages, cards,
31:11details, and so on. In the API, only the
31:14route handlers, database includes
31:16superbase reads and writes. For
31:18scraping, we use oxyabs calls anduler
31:21integration. UI must display store data
31:24only, not invented. And UI must not
31:27scrape, analyze, or mutate. pipeline
31:30state. Then I tell it a bit about the
31:32text stack that we're going to use to
31:34use superbase as the source of truth
31:37with specific tables that we're going to
31:39have and then how scraping is going to
31:41work. Then we tell it a bit about the
31:43correct scraping model, how the workflow
31:46should look like and a bit more info on
31:48how and what to scrape and how to
31:51display it on our website. There's even
31:53a list of likely environment variables
31:56that we're going to have. But we don't
31:57need to read through all of this
31:58together because this is specific for
32:00this project and we will get back to it
32:02later on as we develop some of these
32:04features. You can add other info, but
32:06the goal is to keep it simple so the AI
32:08knows what to do and what not to do. So
32:11in the next lesson, we'll set up the
32:13tool specific skills for Nex.js, Clerk,
32:16Superbase, and Oxyabs so the AI knows
32:19exactly how to use each service
32:22correctly. Let's do that next.
32:26In this lesson, we'll install official
32:28skills for all of the tools that we'll
32:30use to build our application. We'll
32:32install them from the tool team's own
32:34documentation. So, the AI agent gets
32:36up-to-date tool specific guidance.
32:40Basically, the agents file that we saw
32:43explains the tools. So, we'll use the
32:46skills from clerk, superbase, oxyabs,
32:49and AI SDK. And this will help us keep
32:51our setup clean. So, make sure that
32:53you're in the right repo right here at
32:56the root of your project. And let's
32:58first install clerk skills. The
33:01installation is super simple. You simply
33:03have to say mpx skills add clerk skills.
33:08Run it. And if you haven't installed the
33:10skills mpm package, which allows you to
33:12install all of those skills, you can
33:14install it now. And you can select clerk
33:17as well as I think we'll need clerk
33:18billing features here if you want to add
33:20billing to the project. And you can also
33:23select clerk nextgs patterns. Press
33:26enter. Here you can see that by default
33:28these skills will work for these agents.
33:31But if you want to add it for some
33:32additional ones like claw code, you can
33:34scroll the list, select it and press
33:37enter. Then you can choose whether you
33:39want to install it for this project or
33:41globally. For now, let's just proceed
33:43with this project through sim link. And
33:47that's it. Proceed with installation.
33:49And the first set of skills has been
33:51installed. Next, let's install
33:53Superbases agent skills. The
33:56installation command is once again super
33:58simple. You just run MPX skills add
34:01Superbase agent skills
34:04and select what you want to get. In this
34:06case, we're going to go both with
34:07Postgress best practices as well as
34:10Superbase and press enter a few more
34:12times to get it installed. Now the core
34:15of our application are of course the
34:17news which we have to scrape and to
34:20improve the integration of oxyabs which
34:23we'll use for scraping. We can install
34:25their agent skills. On top of the skills
34:27they also have specific plugins for claw
34:30code but in this case I'll proceed just
34:32with installing the skills by running
34:35mpx skills. Add github.com oxyabs agent
34:39skills. You can select Oxyabs. And in
34:42this case, we just need the web scraper.
34:44Press enter a few more times and the
34:48skills are installed. By the way, check
34:50out the OxyLabs agent skills official
34:52repo. Head over there, take a look, and
34:55drop the star to support the project.
34:57And finally, as we'll use AI to figure
35:00out the bias of specific articles, we'll
35:03be doing that through Verscell's AI SDK.
35:06And thankfully, they have the skills for
35:07that, too. So you can just run MPX
35:11skills add versel AI. You can select AI
35:15SDK and press enter a few more times.
35:19Now as I said in this video I'm teaching
35:21you the approach where the agent will
35:24generate the prompts for us. That's the
35:27workflow that we told our agent to use
35:29in this video. But over the last year
35:32I've been also working on my own set of
35:35skills. the skills for agentic
35:37engineering workflows that guide you
35:40from the idea to scoping a feature,
35:42architecting it, and then developing it.
35:45The video you're watching today is super
35:47useful because it was the first step to
35:50get me to develop these skills. I've
35:52developed a set of skills that do that
35:54for you. These skills spend more tokens
35:57and take more time to develop, but the
35:59output that you're getting with them is
36:02significantly higher. So, what I would
36:04recommend for now is maybe checking it
36:05out a bit, giving it a star because it
36:08really helps, and then continuing to
36:09follow this video as it is, as that's
36:12going to give you the under the hood
36:13look at how the skills came to be. And
36:16then after that is done, you can join
36:18the weight list. And then after you
36:20watch the video, you can download the
36:22skills and try developing something of
36:24your own by following the workflow. You
36:26can also join the weight list for the
36:28course in case you want to see me
36:29develop using the skill set. But with
36:31that in mind, after we install all of
36:33these skills, let's inspect the
36:36generated skill files in our agents
36:38folder. Here you should be able to see
36:41the agents for AI SDK, clerk, oxyabs,
36:46and superbase. Everything is here. The
36:49extra clerk skills like billing, orgs,
36:51and so on are fine to keep. They just
36:54support the main clerk one, and you can
36:56use them for future feature
36:57integrations. So now that the AI has
36:59project context from the agents MD file
37:02and the tool context from the installed
37:05skills, the next step is to put together
37:07a workflow. I'll show you how a simple
37:10prompt that we're going to write turns
37:12into an AI written implementation
37:14prompt. How you can then approve that
37:16prompt and how coding only starts after
37:20the approval.
37:23Now that the AI knows the project rules
37:26and tool specific skills, we can finally
37:29build the first real feature. But
37:32instead of just jumping into the
37:33homepage, we'll start with the design
37:35system. This is important because our
37:38app has multiple screens. The homepage,
37:41the news details page, the off screen,
37:44article cards, analysis section, badges,
37:47button states, and more. So if we let AI
37:51style each screen separately, the app
37:54will quickly become inconsistent. So
37:56first we give it a UI reference and then
37:59ask it to create the visual foundation.
38:01Open up your new chat. I'll just go
38:04ahead and open up a new Claude window
38:06and tell it to implement the app design
38:09system from this attached UI reference.
38:12And this reference is something that
38:14I'll provide to you in the video kit
38:16link down in the description. So you can
38:18just attach it and it's going to look
38:20something like this. Skew news balanced
38:23news coverage powered by AI. Here we can
38:25see some primary colors, spacing system,
38:28grid system, the font we wanted to use,
38:30the buttons, and the card example. It's
38:33all here and that's it. You can simply
38:35run it. And a pro tip, you don't even
38:38have to strictly tell it to read the
38:40agents MD file at the start of every
38:42prompt. Modern AI coding tools will load
38:45the project agent rules automatically.
38:47You just have to make sure that if
38:49you're within claude, it refers to the
38:51agents file that is right here. So, as
38:54you can see, it automatically said that
38:56it'll follow the project's workflow.
38:58Inspect the code, read the relevant
39:00guides, and then create a prompt file
39:02for approval before implementing it. It
39:05figured out that it's using Tailwind and
39:07the reference is a full design system
39:09spec. And then it might ask you some
39:12questions. How much of the design system
39:14should this task deliver? We can say the
39:16tokens, components, and a showcase. This
39:19is actually recommended so that we can
39:20immediately visualize everything that it
39:22did. So let's go ahead and submit the
39:24answers. I actually used a GPT image 2.0
39:28model. So if you're interested how to
39:29create these design images, let me know
39:32down in the comments and I can teach you
39:33in one of the upcoming videos. But yeah,
39:35the AI didn't start writing code. It
39:38read the agents. It read the name skill.
39:40It inspected the projects. And only now
39:43it'll create the implementation prompt.
39:45And there it is. You can find it right
39:47here under prompts design system. Since
39:51it's a markdown, we can preview it. And
39:53it looks something like this. The goal
39:56is to translate the attached skew design
39:58system reference into the codebase as a
40:01real reusable foundation. It includes
40:04design tokens that pop in type scale and
40:06a small set of UI primitives plus a
40:08design system showcase page that mirrors
40:11the reference so the tokens can be
40:13visually validated. It's saying which
40:14skills to read the parts of the codebase
40:16to inspect and finally files that it
40:19will change. So this looks good to me.
40:22Let's tell it to run it. If you go back
40:24to the agent, it'll say I prepared the
40:27implementation. Is this good to execute?
40:29And now you can tell it yes it is. So,
40:33let's give it a minute to implement it
40:34and then we'll check out its work on the
40:37forward slashdesign system showcase. And
40:40in about a minute, it came back with
40:42what it built. The full token layer is
40:45now within the app.globals.css.
40:48The color text primaries, background
40:50colors, left and right bias colors,
40:53radiuses, shadows, all of it is right
40:55here. It also implemented the fonts and
40:58created primitive components for
40:59buttons, chips, bias meters, badges, and
41:03article cards. And what I love about it
41:05is that it also created a showcase page
41:08where it actually rendered all of those
41:10components. So the only thing that we
41:11need to do to test it is run mpm rundev
41:14if we haven't already to spin it up on
41:16localhost 3000 and then open up
41:19localhost 3000 design system which
41:22should look something like this. skew
41:24news, balanced news, coverage, brand
41:27colors are right here. And note, this
41:29isn't just an image. These are actual
41:31components that we can use. Different
41:33headings, body elements, UI elements,
41:36and buttons with hoverable states, chip
41:38categories, bias meters, an example of a
41:41card, all of it is right here. That way,
41:44the whole codebase and its UI will be
41:46able to stay cohesive across all of
41:49these new features that we'll add. Now
41:50in this situation we approved the build
41:53directly from the existing chat and when
41:56you do it in the same conversation the
41:58prompt content is already loaded in the
42:00agent's context the fact that it wrote
42:03it a couple of moments ago. So it
42:05implements it from memory of what it
42:07wrote not a fresh read of the file. But
42:10there are two scenarios where the file
42:12and context can diverge. If you start a
42:15new session instead of continuing in the
42:17old one by simply creating a new clawed
42:21window like this that way the agent
42:24won't have the full prompt in the
42:26context. So without the explicit read
42:29the and now we can call it design system
42:32MD coming from prompts and implement it
42:36won't know what it needs to build. So to
42:37make this more reliable we'll modify the
42:40agents.mmd file. Head over into the
42:42agents MD.
42:44Head over into the second step which is
42:46the workflow. Right after seven, modify
42:49the number eight by adding on approval.
42:53Reread the approved prompt file in
42:56prompts folder and implement it
43:00strictly. Implement only author user
43:02approval. That way it'll always go back
43:05to the specific prompt and follow it
43:07strictly. Now, in the next lesson, we'll
43:09use the same workflow to build a
43:11homepage using the design system that
43:13our agent just created. So, let's do
43:16that next.
43:19In this lesson, we'll implement the
43:21homepage UI. So, open up a new chat
43:24window and feed the new homepage design
43:28into it. It'll also be in the video kit
43:30link down in the description, but I
43:32would recommend downloading all the
43:34design images as a zip and then
43:36unzipping them somewhere right here so
43:38that you can very easily drag and drop,
43:40for example, the homepage or one of the
43:42upcoming features.
43:44In this case, we'll tell it to implement
43:47the skew homepage from this attached
43:50app. And skew is just the name of our
43:53app. You can skew left, you can skew
43:55right, and this app is all about the
43:57news bias. So this is the only thing we
43:59need to do. Tell it to implement the
44:01page from the UI. Sounds super simple, a
44:05bit too simple. But then it'll read the
44:08agents MD file and understand that its
44:11goal is not to implement it right away,
44:13but rather understand the codebase and
44:16then create an implementation prompt
44:18that's going to be significantly more
44:20detailed. And it's possible that it'll
44:21ask you some additional questions in
44:23case it isn't 100% certain about
44:25something. What I love immediately is
44:27that it figured out that there's an
44:29established design system. So, it's
44:32going to read the existing UI components
44:34to understand the patterns to build
44:36upon. No questions. This time, it has
44:38enough data to write the implementation
44:40prompt. And the prompt is in. Let's
44:43quickly check it out. The goal is to
44:45build the skew homepage at app page.dsx
44:47DSX to match the attached reference, a
44:50utility top bar, a sticky side header
44:53with logo, primary nav, o buttons, and a
44:56horizontally scrollable category chip
44:58row, a top new section, and a responsive
45:01three column grid of vertical cards.
45:04Display only, no scraping, no analysis,
45:06no pipelation, no live superbase reads
45:09yet because the data layer is not built.
45:12We simply only want to render from typed
45:15mock data so the layout can be validated
45:17pixel close and then swapped to real
45:19superbase queries without touching the
45:21presentational components later on.
45:24Perfect. This is looking good to me. So
45:26let's go back to the chat and tell it to
45:29implement it. And in about a minute the
45:32build is in. The homepage is
45:34implemented. It replaced the placeholder
45:36app with full skew homepage composed
45:39from new presentational components. the
45:42top bar, sidebar, category bar, and site
45:45footer. And all of these components are
45:47developed up to a very high coding
45:49standards. They do contain fake data for
45:52now, but all of that will be updated
45:54very soon. So, the only thing remaining
45:56for us to do is to actually test it.
45:58Head over to localhost 3000 and verify
46:01it all looks good. And damn, it does.
46:05Already, I can see that it loads some
46:07fake images. A couple of images are
46:10broken, but that's okay because this is
46:11just the fake data, but it really does
46:13feel like a real news application. We
46:16have different categories at the top,
46:18the ability to subscribe and login, some
46:21chips that we can click to filter by
46:23specific tags, and then here are the top
46:25news with our app specialties, which is
46:28the bias. So, this is looking great to
46:31me. And we can also check it on mobile.
46:32So, I'll open up an iPhone 14 Pro Max
46:35view. And by testing it like this, I was
46:38actually able to notice a small issue.
46:40And that is that the top bar is jumping
46:43out. So you can see right here that the
46:46top bar jumps out of the view. So we can
46:49simply tell it that. And you can use the
46:51voice for these corrective prompts.
46:53Sometimes it's a bit easier. On mobile
46:55devices, the top bar jumps outside of
46:58the primary content. Fix it. And now
47:00I'll let it do its thing. I'll stay on
47:02local host in mobile mode so we can very
47:05easily see what is happening here. And
47:08when it implements the change, it should
47:10be able to just automatically refresh
47:12and we'll see it in action. There we go.
47:14Exactly as I was saying. That's perfect.
47:17So, we can scroll normally right now on
47:18mobile and we can get back to something
47:21like desktop and it still looks great.
47:24Wonderful. And with that, we have not
47:26only implemented but also tested our
47:29homepage on desktop and mobile devices.
47:33In the next lesson, we'll carry this
47:34forward and implement the news details
47:37page UI in the same way.
47:41In this lesson, we'll build the news
47:43details page. So, simply open up a new
47:46claude window and attach the details
47:49image that you were able to download
47:51from the media kit. It looks something
47:54like this. It still has the top bar and
47:56the navbar, but then it has the actual
47:59article right in the middle. Here, our
48:01users can inspect the full AI analysis
48:03for a specific story. So, we can simply
48:06tell it to implement the news details
48:08page from the attached UI.
48:11Give it a second to analyze everything
48:13it needs. And then, in case it has any
48:15questions, it'll ask us right here using
48:18the ask question functionality. And it
48:21looks like it's diving right in. It has
48:22a complete picture of the codebase
48:24patterns. Per the agents workflow, it'll
48:27create a detailed implementation plan
48:29before writing any code. This is still a
48:31display only UI task. So no feature
48:34skill applies. It's only going to build
48:36on the established design system tokens
48:38and primitives.
48:40So let's let it do its thing and write
48:42the implementation prompt. And very
48:44quickly, it came back with a detailed
48:46implementation prompt. Here's a summary
48:48of what it proposes, or you can just
48:50check out the full proposal right here
48:52under the news details page UI. The goal
48:55right here is to build the skew news
48:57details page to match the attached
48:59reference top shared bar site header, a
49:02two column layout, a related stories
49:05grid, a newsletter CTA, and the shared
49:07footer display only, no data for now.
49:10Later on, we'll get the data from the
49:12database. So, I'll tell it that this
49:16looks good to me. implemented and it'll
49:19just get to work. So, let's let it work.
49:22And two minutes later, the news details
49:24page is now implemented and verified.
49:26Here's what was built. Following the
49:28approved prompt, add prompts news
49:30details page. It added a new page using
49:34the latest Nex.js 16 params and 404 for
49:38unknown IDs. It created different
49:40article components and used different
49:42pieces of UI. It also created the layout
49:45data and then it ran the checks to
49:47verify that everything is actually
49:48working. So news one gave a 200 whereas
49:52news 999 gave a 404 and then as per the
49:55instructions is giving us the manual
49:57steps to test it out. So we can head
50:00over to localhost 3000 and then navigate
50:03to the first card. If we click it we
50:06should be able to see everything. So,
50:08I'll open up the terminal, reload the
50:11application, and rerun it. And then head
50:13over to localhost 3000. Here we can see
50:16the homepage. And then if you click on
50:18the first article, it's going to lead
50:21you to this new news details page that
50:23it created. Of course, all of this is
50:25just fake data right now. So, the bias
50:28being calculated here is not correct,
50:30but you can see all of the UI
50:32components.
50:34It gives you information about what is
50:36happening, the related stories at the
50:38bottom, and even a newsletter that you
50:40can sign up to. And then it's giving you
50:42different sources that are covering the
50:44story. Looks very realistic, like a real
50:47news application. And who's saying it's
50:49not? If you decide to upload it to the
50:51internet later on, you totally can do
50:54that. You can make further adjustments
50:55and really make it your own. Later on,
50:57I'll teach you how to do a real AI
51:00analysis based on the scrape news data,
51:03not just the temporary placeholders that
51:04you can see now. Specifically, you'll
51:07also learn how the page gets related
51:10articles powered by PG vector. And it's
51:12going to be different from a normal
51:14search because a direct search matches
51:16words. So, if you search election, it
51:19looks for the literal string election.
51:21So, only two stories about the same
51:22event come up. But, PG Vector works on
51:25meaning instead. So when the AI analyzes
51:28an article, we'll also turn it into an
51:30embedding, a long list of numbers that
51:32captures what the article is actually
51:35about, then it compares it against all
51:37the others and serves you the best fit.
51:39So all of this is looking good to me.
51:40Now we can also check it out on mobile
51:43view and it feels absolutely amazing and
51:46mobile native. So I'll call it a day
51:49with this feature. So far, we've been
51:51focusing only on the UI, but immediately
51:54in the next lesson, we'll start adding
51:56some logic.
51:59Right now, our app still has two pages,
52:02the homepage and the news details page,
52:06but the app still doesn't know who the
52:08user is. So, in this lesson, we'll add
52:11authentication with clerk. And I love it
52:14how mpm install got replaced with add
52:18clerk o to my app either through the
52:20agent directly or through CLI or the
52:23skills. Love it. That's the future of
52:26modern dev tools. And clerk is more than
52:29just a login box. It's a full user
52:31management platform. So instead of using
52:34something like superbase o which is a
52:36fair choice, we'll use clerk because it
52:39provides us all of these components out
52:40of the box. It provides session handling
52:43and a stack of different oath providers
52:46with almost no UI code needed from us
52:49and it also scales to organizations and
52:52billing later on in case you want to
52:54monetize your app. So the call for this
52:56app is to make clerk own the identity
53:00and superbase can own the data. That
53:03separation will keep both layers clean.
53:06So hopefully you already created account
53:07at the end of the crash course. And if
53:10you haven't already, you can also create
53:11a new project. I'll call it skew news.
53:15And here you can choose from all the
53:17different OOTH providers. Email, phone
53:20number, username, Google, GitHub,
53:22whatever you want. In this case, I'll
53:24proceed just with email and Google. You
53:27can totally just copy the prompt right
53:30here from clerk's onboarding process.
53:32But since we've already added the custom
53:35clerk skill for clerk o, we don't have
53:38to do it. It's going to be even simpler.
53:40Let me show you. In the clerk dashboard
53:42under configure, you can head over to
53:44your API keys where you can find the
53:46publishable key and the secret key.
53:49Simply copy them. Head back over into
53:51your application and create a newv.local
53:57file and simply paste them in. Then open
54:00up a new agent window. Even though this
54:03one didn't spend too much of context,
54:06you can see it's only 10%. But it's
54:08always better to create a new window for
54:11each new separate feature that you're
54:13developing. And you can simply tell it
54:16to implement the clerk authentication
54:19using the clerk agent skills that we
54:22installed before
54:24and press enter. So let's give it some
54:26time to create the prompt and then we'll
54:29check it out. And it might ask you some
54:31questions like should any routes require
54:33login or is off optional with a public
54:36new site with a login option. You can
54:39choose what you prefer, but in this case
54:41I'll go against the recommended
54:43approach, which isn't something that I
54:44often do. And I'll tell it to protect
54:47the news details page, which means that
54:49anyone can browse the home feed, but
54:52opening a full article analysis requires
54:54the signin. But again, you can choose
54:56what you prefer. It'll now write the
54:58implementation prompt and then we can
55:00execute on it. And the implementation
55:02prompt is in the key decisions that it
55:05baked in. So let's check it out. The
55:09goal is to add clerk o to our
55:11application including the sign in and
55:13signup flows session aware header UI and
55:17route protection so that the news
55:18details page require a signedin user
55:21while the home feed stays public. The
55:24goal is to wire up all the environment
55:25variables and keep the service servers
55:28boundaries intact. Key decisions that
55:30were baked in is that we're going to use
55:31the clerk next.js SDK with route
55:34protection, dedicated pages, and then a
55:37header placeholder where login and
55:39subscribe buttons become the signin and
55:42user buttons. So, this looked good to
55:44me. So, I'll tell it to implement it.
55:47And the O is in. The build registers the
55:50proxy middleware and both app routes.
55:54Oh, and I love this. Um, it says, "Note
55:56on the SDK, Cler V7 current SDK remove
55:59the signed in and signed out
56:01components." So, it automatically used
56:03replacements show when either signed in
56:06or signed out API. So, that's how skills
56:09always have the up-to-date info and it
56:11just proved it right here. Okay. So, to
56:14test it out, we can head over to
56:15localhost 3000. While signed out, try to
56:19access any kind of a details page, then
56:22sign back in and try once again. Open up
56:25your terminal, rerun the application on
56:27localhost 3000, and this time I'll spin
56:30it up in a real browser. I'll try to
56:33open up the first article, and it'll
56:36lead me to a clerk sign-in page. So,
56:39I'll try to log in,
56:42continue with Google, and as soon as I
56:44sign in, I'll be redirected back to the
56:47homepage, and now I can access the
56:51article, and I can even see some
56:53information about my profile. You can
56:55manage the account or even check out the
56:57security. Believe it or not, that's it.
56:59We've successfully implemented the O
57:01into our application. And if you want to
57:04have a better understanding of exactly
57:05what was done, you can ask the AI to
57:08explain each generated or modified file
57:10for this feature in detail, including
57:13the middleware, the sign-in and signup
57:14routes, and how protected routes work,
57:17as well as how the login button and user
57:19buttons are wired in the UI. AI is
57:22amazing for explaining specific parts of
57:25the code. So, you can see here how the
57:27proxy works. It's going to give you the
57:29code and then it'll provide you the
57:30explanation. Nex.js actually renamed
57:33middleware to proxy and it explains line
57:36by line exactly how clerk authentication
57:39was implemented. And if you've been
57:40paying attention, you saw that it
57:42implemented all of this without any
57:44mistakes or bugs. That's because of the
57:47workflow that we implemented. The
57:48workflow and the skills tell it how to
57:51implement it properly so that we don't
57:53have to have a lot of back and forth
57:54with the agents. So, if you were already
57:56typing in the comments saying, "This
57:59agentmd file is gonna spend a lot of
58:01tokens." Well, that's not really the
58:03case. If you take a look at the context
58:05usage, it took only 8% to implement the
58:08entire authentication feature. But
58:10imagine if it wasn't that detailed. You
58:12would have to have a lot of back and
58:14forth with the agent to try to fix it.
58:16So take a second try to understand
58:18really how this works verify the login
58:21behavior and in the next lesson we'll
58:23dive right into implementing the
58:25superbase database.
58:29Until now the application has UI and O
58:32but the product data is still just
58:34temporary fake data. So now we need a
58:37database and for this project and many
58:40other projects that I'm building we'll
58:42use superbase as the source of truth.
58:44That means that the sources, articles,
58:47analyses, scraping logs, schedules, and
58:50scheduled run history all live in
58:52Superbase. If you head over into our
58:55agents.mmd file, specifically if you
58:58head over to section number five,
59:01architecture, you'll see that the UI
59:03must display stored data only. UI must
59:06not scrape, analyze, or mutate the
59:09pipeline state. It only displays what is
59:12already there in the database. So
59:13scraping and analysis are separate
59:16layers we build later. This is where we
59:18use Superbase. So if you haven't
59:21already, click the Superbase link down
59:22in the description. Create your account
59:25and then create a new project. Call it
59:27JSM_SK
59:29SKU news and choose the region that is
59:31closest to you and generate a new strong
59:35password which you need to save because
59:37you'll need this database password later
59:39on. So save it somewhere safe and create
59:42a new project. It is completely free.
59:45Next, give Superbase a couple of minutes
59:47to provision your database and the rest
59:50of the stuff that you'll have access to
59:52directly by creating this project. Once
59:54it's ready, you can just pull up your
59:56project URL. You can find it right here
59:58under copy. So go ahead and copy the
1:00:00project URL. Head back over into your VS
1:00:03Code and head over into yourv.local.
1:00:07You can then paste it right here under
1:00:09next public superbase URL.
1:00:14We'll also need a next public superbase
1:00:19anon key. This is going to be the
1:00:21publishable anonymous key as well as a
1:00:24superbase service ro key.
1:00:30So let's go ahead and copy them one by
1:00:32one. Publishable key is right here. That
1:00:36is the add an add-on key that I've been
1:00:37talking about. And then finally the
1:00:40secret key which isn't here but rather
1:00:43under settings
1:00:45API keys and right here at the bottom
1:00:49secret keys. So you can copy it and
1:00:52paste it right here. With that in mind
1:00:54you have everything you need to just get
1:00:57the database developed. I'll simply open
1:00:59up a new claude window and simply tell
1:01:02it to implement the superbase database
1:01:04and data access for skew news by using
1:01:08the superbase skills. Again, it seems a
1:01:11bit wrong just telling it go ahead and
1:01:13do this without having any expectations
1:01:16or telling it what it needs to do or
1:01:18what it needs to avoid. But again, we're
1:01:20not really executing yet. We're only
1:01:22telling it to start analyzing the
1:01:24existing code patterns and the current
1:01:26data layer and the agents MD and then to
1:01:29come up with a final prompt that's going
1:01:32to be super detailed so it gets it
1:01:34implemented properly from the first try.
1:01:36So let's give it some time and
1:01:38potentially answer some questions if it
1:01:40asks us midsession. This time we do get
1:01:43a couple of questions. The first one is
1:01:45how far should this task go? Data layer
1:01:48only or should we also rewire the pages
1:01:51to read from superbase? Yep, that's what
1:01:53I'll tell it to do. We want to form the
1:01:55database and then additionally point the
1:01:58home feed and detail pages at the new
1:02:00query functions. And it's asking us
1:02:03whether we want to see the sources table
1:02:04with the agents example sources like
1:02:08these famous newspapers. And I'll tell
1:02:10it yes, go ahead and see them properly.
1:02:12till we already have some active
1:02:14sources. And in a few minutes, the
1:02:16implementation prompt is in. Before we
1:02:19read the quick summary, let's actually
1:02:21give it a proper read. The goal here is
1:02:23to implement the superbasis data layer
1:02:26as well as wire it to the UI. The schema
1:02:30thankfully was already provided by the
1:02:31agents MD. It's going to include
1:02:33sources, articles, article analyses,
1:02:36logs, oxyab schedules, and oxyilabs
1:02:39scheduled runs without the embedding
1:02:41column for now. It's going to see the
1:02:44database with some example sources. It's
1:02:46going to create handwritten database
1:02:48types. So, we have type safety from the
1:02:50get-go and then it'll create a superbase
1:02:53client on the client side as well as
1:02:55write queries so we can fetch those
1:02:58analyzed articles and sources. Finally,
1:03:01it'll wire it all together. All of this
1:03:04looks good to me. Uh, it is pointing one
1:03:06judgment call that it wants to flag. The
1:03:09mock models a story as a cluster of many
1:03:12sources, but the real schema is one
1:03:14article, one source. I map that
1:03:17honestly. Source count is set to one
1:03:19without redesigning the multi-source
1:03:21sidebar cards. This sounds good to me.
1:03:24Later on, if we need to fix it, we can
1:03:26always get back to it, but for now, this
1:03:27is looking good. So, is this good enough
1:03:30to execute? Yeah, just go ahead and
1:03:32execute it. And you can see based on
1:03:35that addition that we added to agents
1:03:37MD, it's actually going to reread the
1:03:39prompt and start implementing it
1:03:41starting with package installs and then
1:03:43schema files. In this case, we're using
1:03:45Superbase.js and it'll proceed creating
1:03:48the SQL files. And in a couple of
1:03:50minutes, all three checks pass. So, not
1:03:53only has Superbase database been added
1:03:55and database access connected, but it
1:03:58also did a couple of checks. It confirms
1:04:01the build on the homepage and the news
1:04:03detail pages which are now dynamic and
1:04:06server rendered as well as the addition
1:04:08of the serveron service client enforced
1:04:11by the build. So, it actually
1:04:13implemented the changes across a couple
1:04:15of different files, the SQL schema being
1:04:18a primary one. This is something that
1:04:20we'll have to manually apply in the
1:04:22Superbase dashboard. So, we're creating
1:04:24tables for articles, sources, analyses,
1:04:27and more. Then, in the seed, it's
1:04:29actually going to seed a couple of
1:04:31sources.
1:04:32There's the types that enforce
1:04:35everything is type safe. The server,
1:04:37which actually creates and exports a
1:04:40service client,
1:04:42and then the queries to actually fetch
1:04:44the articles and display them. So to
1:04:47test them, we first have to head over
1:04:49into the Superbase dashboard under the
1:04:50SQL editor and then copy and paste the
1:04:53schema and click run. We can do that
1:04:57easily by heading over here, heading
1:04:59over to SQL editor, pasting it, and
1:05:02clicking run. If it says something along
1:05:05the lines of success, no rows returned,
1:05:08we're good. The second step is to run
1:05:11the Superbase seed SQL function. Confirm
1:05:14five active rows in sources. This again
1:05:16you have to run right here. But make
1:05:18sure to override what you previously
1:05:20had. And this should run very quickly.
1:05:22So if you head over into the table
1:05:23editor under sources, you should be able
1:05:26to see five sources added to your
1:05:29database right here. There we go. Then
1:05:31the next step is to ensure that we have
1:05:33the three environment variables, which
1:05:35we do, and then run it on localhost
1:05:383000. Back in the browser, I'll run
1:05:40localhost 3000. And now under homepage,
1:05:44we won't have any analyzed articles yet.
1:05:47So what we can do is insert an article
1:05:51manually into the SQL editor. Or we can
1:05:54tell it to generate one for us. But
1:05:56first, it's asking us whether we want to
1:05:57save something to project memory. I'll
1:05:59say yes. Save it. And give me the dummy
1:06:05article I can paste into the SQL editor.
1:06:10AI is amazing for just giving you quick
1:06:13pieces of data that you can copy or
1:06:14paste somewhere. And there we go. It
1:06:17gave us something that we can very
1:06:19easily copy. It's a self-contained dummy
1:06:22article. We can paste into the SQL
1:06:23editor. So, head over into here and just
1:06:27paste it in. And it's asking us whether
1:06:29we want to run it without RLS or with
1:06:32RLS enabled. I'll go ahead and enable
1:06:34it. And we got a success. So now if you
1:06:37go to localhost 3000 you'll see a single
1:06:40article right here. It's a real article
1:06:43coming from the database. You can open
1:06:44it up, see what it is all about. And
1:06:47that's it. You can also see our real
1:06:50source. But this is the article that we
1:06:53added manually. Even though it's real
1:06:55data in a real database, it's still
1:06:57dummy data. So in the next lesson, we'll
1:07:00build the OxyLabs scraping pipeline.
1:07:03It's going to read the active sources
1:07:05that we just seeded, visit each
1:07:07homepage, and then pull real articles
1:07:10and insert them into the articles table
1:07:12automatically, the very table that this
1:07:15page is reading from right now. So,
1:07:18let's do that in the next lesson.
1:07:23This page reads from the database and we
1:07:26seated the sources, but there are a few
1:07:29articles. I mean, there's one, and that
1:07:31one that is here is just a dummy article
1:07:34that we added manually through the SQL
1:07:36editor. So, in this lesson, we'll build
1:07:38the engine that fills it. The Oxyabs
1:07:41scraping pipeline that turns our seated
1:07:44source homepages into real stored
1:07:46articles. But first, let me explain what
1:07:49this pipeline actually does, because
1:07:51it's more than just download a page. So
1:07:54head over into your agents.mmd and
1:07:56specifically head over into section 9
1:08:00correct scraping model and I can check
1:08:02this out in preview. At a high level the
1:08:05scraping will work like this. First we
1:08:08load the selected active sources from
1:08:10superbase. Then we obtain each sourc's
1:08:13homepage html and manual scraping
1:08:16fetches the stored homepage URL live
1:08:18through oxyabs. Scheduler processing
1:08:20uses completed Oxyab's job result and
1:08:24the goal is to never crawl into sublinks
1:08:26to find more listing pages. Then we
1:08:29extract the candidate links from visible
1:08:31homepage story cards. We reject anything
1:08:34on the non article reject list before
1:08:36detailed scraping. We then normalize and
1:08:39ddup candidate URLs and then skip URLs
1:08:42already stored in superbase. That way we
1:08:44don't have duplicate articles. We then
1:08:47scrape only article detail pages that
1:08:49pass the candidate URL check. We
1:08:51validate and clean each detail page
1:08:54because it must pass the article contact
1:08:56gate. More on that later. We insert only
1:08:59the valid articles, append them, and
1:09:01that's it. But the thing that I want to
1:09:03point out here is that Oxyabs won't do a
1:09:06raw fetch. That's because new sites
1:09:10actively block scrapers. Oxyabs manages
1:09:13proxy networks and geo complexity
1:09:15legally so we can get a clean homepage
1:09:17HTML back instead of a page that's
1:09:20telling us that we've been blocked. It
1:09:22can collect real-time data from almost
1:09:24any public website scale effortlessly
1:09:27without you having to manage those
1:09:29browsers, servers, and proxies and
1:09:32integrates with tools that you already
1:09:33know like Puppeteer, Playright or
1:09:36workflows like N8N. And it even has MCPS
1:09:39for claw and cursor. On top of that, it
1:09:42offers very low latency scraping, 24/7
1:09:46support if you need it, and has been
1:09:47recognized as one of the best proxy
1:09:49services. Head over to
1:09:51oxyabs.io/javascript
1:09:54where you can get a completely free
1:09:56trial, up to 2,000 results with no
1:09:59credit card needed. And if you upgrade,
1:10:01you can use the JavaScript code to get
1:10:0420% off any plan. We'll be using the web
1:10:07scraper API. So, let's get started with
1:10:09Google. As soon as you're in, you can
1:10:11see the testing that I've done. And when
1:10:12you're here, you can just get the API
1:10:14key by heading over into my account in
1:10:16the left sidebar. And then right here
1:10:18under web scraper API, you can start a
1:10:21free trial. Since I was doing a lot of
1:10:24testing while building this application,
1:10:26I got upgraded. But you can see even
1:10:28with all the testing that I've been
1:10:29doing, I still spent only like six
1:10:31bucks. and that is me testing it
1:10:33throughout the entire process of
1:10:34developing the app. So, you're going to
1:10:37be more than good on the free plan,
1:10:38which you can start right here. When you
1:10:40create an account, you'll get your
1:10:42username and password. Make sure to copy
1:10:45those and not lose them. Then, you can
1:10:47take them and head over into your envoy
1:10:53WSA username
1:10:56and you can paste yours. And you can do
1:10:58the same thing with oxy wsa password.
1:11:02And alongside that, if you head over
1:11:04into your agents.mmd and search for x
1:11:08skew admin secret, you can see that this
1:11:11is being mentioned. You can add another
1:11:14environment variable that says xskeq
1:11:17admin secret and make it equal to any
1:11:20kind of a strong secret that you choose.
1:11:22So this can be an automatically
1:11:24generated string of characters. Next, go
1:11:27ahead and open up a new claude window
1:11:29and tell it to implement the Oxyabs
1:11:32scraping pipeline using the Oxyabs web
1:11:35scraper skill as well as the Superbase
1:11:38skill to connect it all together and
1:11:40press enter. This is a big multi-part
1:11:43feature, so the prompt file matters more
1:11:46than usual. Let's give it a second to
1:11:48review all the relevant code and then
1:11:50create an implementation prompt. It
1:11:52might ask us a couple of questions on
1:11:53the way. And after inspecting the
1:11:55existing data layer, the oxyab superbase
1:11:58skills and the Nex.js route handler
1:12:00docs, it wrote the implementation
1:12:02prompt. The scope it planned is right
1:12:06here. The goal is to implement the SKU's
1:12:09news applications manual scraping
1:12:11pipeline, the post API scrape action
1:12:14route and the layered scrape to insert
1:12:16engine behind it on demand. It loads the
1:12:19selected active sources from Superbase.
1:12:22It fetches each sourc's HTML page
1:12:24through OxyLabs, extracts the visible
1:12:27story card links from the homepage,
1:12:29rejects any non-article URLs via the
1:12:32non-article reject list, normalizes the
1:12:35dupes and everything else we discussed
1:12:37so far. As I said, this is a big
1:12:40feature, so the prompt file matters more
1:12:42than usual. While you're reviewing it,
1:12:44check that it reflects the pipeline
1:12:46rules from the agent. Thankfully, in
1:12:48this case, it does. So if the prompt
1:12:50looks good, you can go ahead and confirm
1:12:52it by putting out the shortest answer
1:12:55that you can give. And I think that's a
1:12:57single letter Y. I think this is going
1:12:58to be enough for you to figure out as
1:13:00yes, go ahead and implement it. We're
1:13:02implementing one of the most difficult
1:13:04features with a single letter. And
1:13:07that's only possible due to the agentic
1:13:09workflow that we implemented before
1:13:10through the agents MD file. And of
1:13:13course, thanks to the tools that we're
1:13:14using such as Superbase and OxyLabs in
1:13:17this case for scraping because otherwise
1:13:19we'll just be getting denied while
1:13:21trying to scrape these types of
1:13:22websites. So now since this is a big
1:13:24feature, let's give it some time to
1:13:26implement it. And there we go. This is
1:13:28the biggest response we've gotten so
1:13:30far, which makes sense for the biggest
1:13:32feature. It implemented the manual
1:13:34OxyLabs scraping pipeline per the
1:13:37approved prompt. All checks pass.
1:13:39linting zero warnings and build also
1:13:42builds both the API scrape route and the
1:13:44API sources route both of which were
1:13:47registered. So what was built the
1:13:50scraping layer using this oxyaps file
1:13:53and the pipeline with scrape.ts which
1:13:56uses the run manual scrape orchestrator.
1:13:58This includes the source selection all
1:14:00active by default live homepage fetch
1:14:03extract filter ddup everything else we
1:14:06discussed so far and there are two new
1:14:08API routes both guarded with admin off
1:14:11how to test it set the keys which we
1:14:14have already done here it called it skew
1:14:16admin secret if I'm not mistaken we
1:14:18called it a bit differently x skw admin
1:14:21secret so you can do whatever the agent
1:14:24says right here make sure the name is
1:14:26correct
1:14:27and then run npm rundev. Inspect the
1:14:30active sources by making a curl to API
1:14:32sources and then make another post
1:14:34request to scrape the data and then
1:14:37successfully scrape all three resources
1:14:39bypassing your skew admin secret. Okay,
1:14:42let's go ahead and do it. I'll rerun the
1:14:45application.
1:14:46And the first one we can actually do
1:14:49using the browser because it's just a
1:14:51get request. And as you can see right
1:14:53here, when we go to localhost 3000 API
1:14:55sources, we immediately get back the
1:14:58five sources we have added so far. Now
1:15:01we can try to make an unauthenticated
1:15:03post request by copying the scroll
1:15:05command, opening up another terminal,
1:15:08and then pasting it in. You'll see a 401
1:15:12unauthorized. That's good. And then
1:15:14finally, we can try to scrape three
1:15:17resources by passing the admin secret.
1:15:20So do it like this. Go ahead and copy
1:15:23your admin secret, which is going to be
1:15:25this one right here. And just update it
1:15:28right here. X skew admin secret. Here I
1:15:33can see that it does contain that X up
1:15:35front. So we'll test it out with and
1:15:36without it. At the same time in our
1:15:39terminal, you can see that it scraped
1:15:41three selected resources. BBC MPR
1:15:45routers fetching the homepage. Homepage
1:15:47fetched 48 links. We got 118 links right
1:15:51here and it's inserting them right here.
1:15:53From BBC from 48 links, it got zero
1:15:56candidates. But from NPR, it got 57
1:15:59candidates and it's starting to insert
1:16:01them. This process will take some time
1:16:03as we're trying to scrape a lot of
1:16:05articles. So, let's let it run. And now
1:16:07it's taking a look at the final source.
1:16:1053 candidates, 47 rejected, and it's
1:16:13trying to parse them. At the same time
1:16:14while this is happening, we can head
1:16:16back over to Superbase under the table
1:16:18editor and take a look at the articles
1:16:20that we have. And you can see that
1:16:22slowly but surely we're building them
1:16:25in.
1:16:26There's the one that we implemented
1:16:29ourselves manually, which I believe is
1:16:31this one from the Unsplash photo. That's
1:16:33the fake one, which we want to delete
1:16:35now that we have all the real data. And
1:16:37as we're deleting it, new articles are
1:16:39going to keep coming in until the scrape
1:16:41finishes, which looks like it just did.
1:16:44So, manual scrape completed, status
1:16:46completed, three sources checked, many
1:16:48candidates found, 25 detailed pages
1:16:51scraped, and here we got per source
1:16:53information. So, if you head back, you
1:16:56can see that this completed. Wonderful.
1:16:58This means that we have gotten a couple
1:17:00of real articles each with a real source
1:17:04ID, a URL, a canonical URL, a title, for
1:17:07example, talking about the largest
1:17:09housing affordability bill in decades.
1:17:13This is coming from NPR and more. So
1:17:15now, if you head back to your browser
1:17:17and expect that if you head back over to
1:17:19localhost 3000, which is the homepage,
1:17:22that you're going to see the articles
1:17:24here, you'd be wrong. They're not going
1:17:26to be displayed here just yet.
1:17:29Thankfully, we know that they exist
1:17:31because we can see them in the
1:17:32Superbasis table editor, but on the
1:17:35homepage, we're filtering by the
1:17:37analyzed at field. And you can see
1:17:39analyzed at is currently set to null for
1:17:42all of these articles, which means that
1:17:44they're not yet accessible on the
1:17:46homepage. For them to appear there, they
1:17:48need to have the data about the AI
1:17:50analysis, which is what we're going to
1:17:52do in the next lesson. But for now, stop
1:17:55and appreciate what you've done in this
1:17:57lesson, as this one is the biggest
1:17:59payoff so far. From a single one-s
1:18:02sentence prompt, you build a real
1:18:05production scraping pipeline featuring
1:18:07robust uninterrupted data collection
1:18:09through Oxyabs homepage parsing, a
1:18:12reject list for non articles,
1:18:15URL dduping against the database, per
1:18:18article validation and text cleanup, and
1:18:20appendon inserts. All with fullr run
1:18:23logging with a summary. People spend
1:18:26days wiring this up by hand. But what
1:18:29you did now was our agent skills
1:18:32workflow doing its thing. Here we
1:18:34defined the tools and services that
1:18:36we're going to use and then AI wrote the
1:18:39detailed prompt covering all of it and
1:18:41implemented it. You then reviewed that
1:18:43detailed prompt and said yes. It's that
1:18:46simple and easy. So now that we have
1:18:49real articles in the database, we need
1:18:51some insights on them. I mean that's
1:18:53what our app is all about. We don't want
1:18:55to simply scrape and display. So in the
1:18:58next lesson, we'll build the AI analysis
1:19:01pipeline that reads from these scraped
1:19:03articles, generates a neutral summary,
1:19:06sentiment, and AI estimated framing,
1:19:09saves it, sets the analyze ad, and that
1:19:12is the moment that the homepage will
1:19:14finally fill [snorts] with real analyzed
1:19:18news cards. So let's do that next.
1:19:23This lesson is where skew news earns its
1:19:26name because we'll figure out toward
1:19:29which spectrum is the article leaning.
1:19:31We'll build the AI analysis pipeline
1:19:33that'll read each scraped article and
1:19:36produce its summary, sentiment, and AI
1:19:39estimated framing. If you head over into
1:19:41your agents.md and head over to 19,
1:19:46you'll be able to see a bit more about
1:19:48the analysis. It must process valid
1:19:51articles must be triggered via this URL.
1:19:54And here you can see what it'll save in
1:19:56the analysis. The neutral summary, the
1:20:00sentiment score, AI estimated political
1:20:03framing, left percentage, center
1:20:05percentage, right percentage, so where
1:20:07is it leading? Derived bias score,
1:20:11confidence, framing notes, load terms,
1:20:14disclaimer, and model name. Not sure
1:20:16where these extra characters are coming
1:20:18from. for you. They might not be there,
1:20:20but either way, that's okay. This right
1:20:22here is the heart of the product. The
1:20:25framing is AI estimated, not objective
1:20:27truth. And SK News is going to present
1:20:30the user with an estimate with a
1:20:33disclaimer saying that the AI judged the
1:20:35article based on the article's own text
1:20:38and never from the source name. For the
1:20:40analysis, we're going to use the OpenAI
1:20:42API key. So go ahead and sign in to
1:20:45OpenAI API and then create a new project
1:20:48which you can call something like JSM
1:20:52skew news
1:20:54and create a secret key
1:20:57which you can then copy. Head over into
1:20:59your env.local and add it as open AI
1:21:04API_key
1:21:06and then you can paste what you copied.
1:21:08Then open up a new chat and simply tell
1:21:11it to implement the AI article analysis
1:21:14pipeline using the superbase skills. I'm
1:21:17adding the superbase skills here, but
1:21:18the AI will also read the AI SDK skills
1:21:21on its own. So press enter and let's see
1:21:24what it comes back with. And very
1:21:26quickly, it prepared the AI analysis
1:21:29implementation prompt. The goal here is
1:21:32to implement the AI article analysis
1:21:34pipeline described in agents MD feature
1:21:3719. Given articles or the scraped and
1:21:41stored in Superbase, we want to detect
1:21:43articles that are pending analyses. Run
1:21:46each through an AI model to produce a
1:21:48neutral summary sentiment and AI
1:21:50estimated political framing and then
1:21:52validate the output with ZOD. And
1:21:54finally, save it append only to article
1:21:57analyses.
1:21:59Perfect. Sounds good to me. So let's go
1:22:02ahead and implement it. And there we go.
1:22:05The feature got built. This was the
1:22:08second biggest feature after scraping.
1:22:11So it did take a couple of minutes, but
1:22:13it is here. And we got the full
1:22:15analysis. We already added the OpenAI
1:22:18key and we have the admin secret key as
1:22:21well from before. So now we can just run
1:22:24a single request to analyze all the
1:22:27current articles or you can do it like
1:22:28this one by one. So let's go ahead and
1:22:31copy this curl request that you should
1:22:33have gotten as well. And it's so useful
1:22:35to have this way to test it. Again, this
1:22:37is also something that we manually added
1:22:39to the agent MD file. So it let us know
1:22:41how we can actually test the feature. So
1:22:44to give it a shot, you can open up your
1:22:47terminal, run mpm rundev, and then in
1:22:50the secondary terminal, you can paste
1:22:53this. But make sure to modify this admin
1:22:57secret key. Is it already pulling it
1:23:00from env? Maybe it is, but either way,
1:23:03I'll just manually type it here and
1:23:05press enter. Now, if you head back,
1:23:07you'll see some information. run started
1:23:11eight pending articles batch one out of
1:23:14two is doing it five per batch and it
1:23:16saved the analysis for this one this one
1:23:20this one and it'll continue doing it for
1:23:23the next four articles. You can already
1:23:25head back to Superbase and reload it to
1:23:28see that now the analyzed ad actually
1:23:31has a time stamp which means that now
1:23:33we'll have full analyses per article
1:23:36which are being stored in this table
1:23:38right here. You can see the sentiment
1:23:40score is different. We get the summary
1:23:42of what this article is discussing,
1:23:44whether it's negative, neutral, the bias
1:23:47score, whether it's mixed, unclear, or
1:23:50something else, and is it leaning left,
1:23:52center, or right. As you can see, the
1:23:54confidence score for most is actually
1:23:56pretty high besides this one where it's
1:23:58about 50%. Pretty cool stuff. So, with
1:24:01that in mind, if you head back to
1:24:02localhost 3000 and reload the page,
1:24:05you're now going to see real articles
1:24:07loaded right here. And it's possible
1:24:10that for you, you're going to get an
1:24:11error. If you do, that's fine. You can
1:24:13debug it using the agent. But most
1:24:16likely, the issue is that it's trying to
1:24:19render specific images. So if you head
1:24:21over to next config and if the images
1:24:24are being loaded from a source that you
1:24:26didn't allow, you can allow it for all
1:24:28sources by simply adding images remote
1:24:31patterns protocol https for all host
1:24:35names like this. Now if you revisit the
1:24:38homepage, you'll be able to see a list
1:24:40of all articles that have been scraped
1:24:42with their AI analysis. Now there are
1:24:45some what appear to be repeated
1:24:47articles. We're going to look into that
1:24:49later on. But the most important thing
1:24:51is that the articles that we have
1:24:53scraped, so we haven't entered them
1:24:55manually now appearing and each one
1:24:58actually has the analysis. For example,
1:25:01Ryionaire passenger partially sucked out
1:25:03of the window on a flight from Greece.
1:25:05And here we can see that this is mostly
1:25:07center as it doesn't really have to do a
1:25:09lot with politics. But with something
1:25:11like this talking about the white
1:25:13nationalist group Patriot Front, you can
1:25:16see that it's leaning significantly more
1:25:17toward left and you get the full post as
1:25:20well as the AI summary right here
1:25:22discussing what's happening. We're not
1:25:24going to get into politics on this
1:25:25channel, but it is good to know that you
1:25:28can develop apps that serve any purposes
1:25:30as well. Some of you are going to do
1:25:32apps that have to do something with
1:25:33healthcare or serve anybody in general,
1:25:36but from time to time it can also be
1:25:38news or politics. So yeah, it's all
1:25:40here. This is that satisfying moment
1:25:43which displays everything we've been
1:25:45working towards so far. Not just the
1:25:47homepage with the real data, but also
1:25:49the news detail pages. Everything we
1:25:52designed and wired in the earlier
1:25:54lessons just came alive with real data
1:25:57end to end. So with this, our skew news
1:26:00app is already a working product. It
1:26:03scrapes, analyzes, and displays real
1:26:06news with framing insights. And in the
1:26:09next lesson, I'll show you how you can
1:26:11go beyond just developing a codebase and
1:26:14turning this into a real product. You'll
1:26:16do that by understanding how your users
1:26:19are actually using the app.
1:26:23Before we do some tracking on how users
1:26:25are actually using your app, I want to
1:26:28implement a feature that's going to make
1:26:29the app so much more usable instantly.
1:26:32And that is when you open up a specific
1:26:34article, you can scroll down and then
1:26:37get access to some related articles.
1:26:40Articles not related just by keywords,
1:26:43but actual meaning. And we'll implement
1:26:46that feature using PG vector with
1:26:48superbase. As I explained before, the
1:26:51problem with normal search is that it
1:26:53matches words. So if you search for
1:26:56something like flood in this case, it'll
1:26:58just match that same word. But two
1:27:01articles covering the same event with
1:27:03different wording would never get
1:27:04connected. And PG vector solves that
1:27:07problem by searching on meaning instead
1:27:10of words. It's a Postgress extension
1:27:12that Superbase offers and it lets us
1:27:15store vectors right in the database and
1:27:18search by similarity. A vector or an
1:27:21embedding is just a long list of numbers
1:27:24that captures what a piece of text is
1:27:26actually about. text with similar
1:27:29meaning ends up with numbers that sit
1:27:31close together. We generate those
1:27:33embeddings with OpenAI's text embedding
1:27:36small model, one per analyzed article
1:27:39and then store it within the article
1:27:40analysis. To find the related articles,
1:27:43PG vector will then compare embeddings
1:27:46using the cosign distance operator and
1:27:49return the closest one in meaning. So
1:27:51related stops meaning shares a keyword
1:27:54and starts meaning closest in the
1:27:57meaning and that's how we surface the
1:27:59same story from another source even when
1:28:01the headlines share almost no similar
1:28:03words. Let me show you how to implement
1:28:05it. Superbase thankfully makes it super
1:28:07easy. So we can just open up a new
1:28:10clawed session and tell it to implement
1:28:13PG vector support and the related
1:28:15articles feature. Enable PG vector. Add
1:28:18the embedding column. Update the AI
1:28:21analysis pipeline to also generate and
1:28:23save embeddings and then add the related
1:28:25article section to the news detail page
1:28:28using the superbase skills. Run it and
1:28:31let's wait for a full detailed
1:28:33implementation prompt. And very quickly
1:28:35it said that it inspected the schema
1:28:37types analysis pipeline AI layer queries
1:28:41mappers and the news details page and
1:28:43read the superbase skill plus the memory
1:28:45conventions. It prepared the
1:28:47implementation prompt for implementing
1:28:49the PG vector related articles. So let's
1:28:52go ahead and check it out. This is going
1:28:54to be under PG vector prompt. And the
1:28:58goal in this case is to upgrade the
1:29:01analysis pipeline to also generate and
1:29:03store OpenAI embeddings and add related
1:29:06article section to the news details page
1:29:09powered by the PG vector cosine
1:29:12similarity. And I want to point out that
1:29:15this PG vector thing became so much more
1:29:17useful in the last couple of years when
1:29:20AI actually allows you to do proper
1:29:22matching. And that's what Superbase is
1:29:24all about. They allow you to very
1:29:26quickly spin up a new database and allow
1:29:29agents to just do their thing.
1:29:32Everything from database that you've
1:29:33seen in this video as well as storage,
1:29:35realtime edge functions, but also things
1:29:38like AI and vectors, which is what we're
1:29:40looking into right now. So this is
1:29:42looking perfect to me, strictly
1:29:45following the feature 20 of the
1:29:47agents.mmd.
1:29:48So let's go ahead and tell it to
1:29:50implement it and run it as it is. Let's
1:29:54give it some time to get it ready and
1:29:56we'll review it soon. And in a few
1:29:58minutes, the implementation is complete
1:30:00with all checks passing. It built a
1:30:04superbase schema which enables vectors.
1:30:07So specifically, we're talking about
1:30:08embedding vectors to article analyses.
1:30:12So if you take a look at this part right
1:30:14here, you'll be able to see a new schema
1:30:16where we create this extension using PG
1:30:19crypto, create a new table. And I
1:30:22remember the times where you had to
1:30:24write all of this SQL code manually, but
1:30:28those are no longer the times. And I'm
1:30:30very happy for that because development
1:30:33is not about just typing. Development is
1:30:36about thinking which problems you want
1:30:38to solve and what you want to achieve.
1:30:40And this is just the path to get there.
1:30:43So, Superbase now is handling a ton of
1:30:45that stuff for you. There's also the
1:30:47types, the embeddings layer using AI as
1:30:51well as the full pipeline and the new UI
1:30:53which fetches the embedding server side
1:30:55and then passes it and then displays the
1:30:58final articles. Uh, and for this one,
1:31:01you will need an OpenAI API key. So you
1:31:05can just head over to Google, search for
1:31:07OpenAI API, create a new secret key and
1:31:10add it to your environment variables.
1:31:13Before testing, head over to Superbase
1:31:15dashboard and run this SQL block right
1:31:18here that it provided you. So I'll head
1:31:20over to the dashboard to one of my many
1:31:23active projects and specifically head
1:31:26over to the SQL editor, override
1:31:30everything, and just run it. And this
1:31:32run failed. Uh, which I'm very happy
1:31:35about actually because I got to show you
1:31:36this debug with assistant feature. You
1:31:39could either debug it right here or you
1:31:42can take this message and then provide
1:31:44it to your agent which has more
1:31:46information about your full codebase and
1:31:48everything that's happening. So I'll
1:31:50provide it here. And it very quickly
1:31:52found the problem right here and gave us
1:31:54a function to rerun. So it updated the
1:31:57schema which you can now copy, override
1:31:59it here and just rerun it once again.
1:32:02This time it is running for longer which
1:32:04is always a good sign because we get a
1:32:06success with no rows returned. So if you
1:32:09take a look at the second step right
1:32:10here, it is to first run this then run
1:32:15the application on localhost 3000 and
1:32:18then trigger the analysis. The
1:32:20application already is running but I'll
1:32:22just rerun it once again.
1:32:25head over to localhost 3000 and then
1:32:29visit a specific article. And if you
1:32:31scroll down, you won't be able to see
1:32:33any articles yet. So the next step is to
1:32:36trigger the analysis. You can do that by
1:32:39copying this command and heading into a
1:32:41new terminal where you can paste it. But
1:32:45don't forget to get this skew admin
1:32:47secret which is in our env.
1:32:50And then just simply replace it.
1:32:54If you do this, it'll seem stuck, but
1:32:56then if you head over to your original
1:32:57terminal, you'll see that it's actually
1:33:00running the analysis on the current
1:33:03articles that we have so far. And for
1:33:05each one, it'll say saved analysis plus
1:33:08embedding. And these embeddings is
1:33:10what's going to make the related
1:33:12articles work. This is super useful for
1:33:14suggestions or recommendations of
1:33:17specific data pieces. It can be news, it
1:33:20can be podcasts, it can be information
1:33:22cards. Whatever it is, embeddings are
1:33:25the fastest and the most efficient way
1:33:28to show you recommended content. So,
1:33:30let's give it some time until it runs
1:33:32and then we'll test it out. But even
1:33:34while it's going already, some articles
1:33:37are in. So, I can see some top news
1:33:39appearing right here. And if you head
1:33:42over into it and scroll down, you'll be
1:33:45able to see the related stories section.
1:33:48And here you can also see the source
1:33:51that it is from. All of these are from
1:33:53Guardian. And you can see the narrative.
1:33:56In this case, it's all positive, but
1:33:58there is one negative one right here.
1:34:00So, for example, we can click on this
1:34:02one. And you can see it picks it up. Or
1:34:06maybe we can go with this one right
1:34:07here. So now this makes users stick
1:34:10longer with your app and explore more
1:34:12articles because they can just come to
1:34:14the end and explore the first article
1:34:17that is most related to that story.
1:34:20Wonderful. I'll let it proceed with the
1:34:22analysis but we're actually done and you
1:34:24could verify it on Superbase but
1:34:26thankfully we verified it already within
1:34:28the application itself. So that's it.
1:34:30Alongside the database storage, there's
1:34:33also these vectors which are becoming so
1:34:35much more important nowadays to match
1:34:38and show recommended content. And I
1:34:40remember the times where it would take
1:34:42weeks to set up something like this. But
1:34:44nowadays, as Superbase says, build in a
1:34:48weekend and scale to millions.
1:34:52You built the design system, the pages,
1:34:56o the database, scraping, AI analysis,
1:35:01but writing and shipping code is not the
1:35:04same as shipping a product. A product is
1:35:07code plus knowledge on what's happening
1:35:10inside it once real people use it. For
1:35:13years, analytics meant you look at a
1:35:16dashboards and decide what to fix. But
1:35:19Posthog is pushing past that into what
1:35:22they call self-driving. The idea is that
1:35:25Post Hog already has your product data,
1:35:28errors, session replays, funnels,
1:35:30support tickets. So instead of just
1:35:33charting it, agents can read it, cluster
1:35:36what's worth fixing into a report, and
1:35:38then open up a pull request with a fix
1:35:40already attached. You wake up to diffs
1:35:43to review, not the dashboards to
1:35:45interpret. And that's the same idea
1:35:48you've been running the entire time. You
1:35:50gave your agents the rules and it wrote
1:35:52its own prompts. Self-driving gives an
1:35:55agent your product data and it writes
1:35:57its own fixes. It's the same mechanism
1:36:00pointed at a live product instead of a
1:36:02codebase. Nothing ships on autopilot
1:36:05though. A human still needs to merge.
1:36:08You are the driver. And most of us will
1:36:11actually drive it through Posthog on
1:36:13Slack. You simply tag Post Hog in a
1:36:16thread with some kind of a data question
1:36:18like why did my signups drop last week
1:36:21or some kind of a change that you want
1:36:23to add and it answers from your own data
1:36:25or just opens up a draft PR for you
1:36:28without you having to leave Slack. So,
1:36:30let me show you how all of it works and
1:36:32how you can implement it for this
1:36:34project and all of your upcoming
1:36:36projects. First, click the Post Hog link
1:36:38down in the description and create a
1:36:41free account. If you're new, you'll most
1:36:43likely be redirected to the onboarding.
1:36:45And if you've developed some projects
1:36:47before, you can just manually head over
1:36:49to onboarding by changing the URL. It'll
1:36:52ask you what you want to use the post
1:36:53hawk for, like understand how users
1:36:56behave, find and fix issues, or collect
1:36:58feedback. Either way is fine because the
1:37:01best way to install and set up Post Hog
1:37:04is through their wizard. That way, you
1:37:07don't have to handle any ENVs or capture
1:37:09things on client and server. Post Hog
1:37:12Wizard will do all of that for you.
1:37:14It'll basically wire itself into your
1:37:16codebase. So, simply open up your
1:37:18terminal, stop it from running, and run
1:37:22MPXY post hog wizard latest. And if you
1:37:26have a specific region, you can add that
1:37:28too. Just run it. It might ask you to
1:37:31install the Post Hog wizard. So, simply
1:37:34press Y and then install it. And here's
1:37:36how it works. It automatically figured
1:37:38out that we are in this directory, that
1:37:41we're using Nex.js and that we want to
1:37:43integrate Post Hog. So, simply press
1:37:45enter to continue. It'll ask you to
1:37:48authenticate directly within the browser
1:37:50and choose a project that you want to
1:37:52develop within. I'll create a new
1:37:54project called JSM skew news. And you
1:37:58can do the same on your end as well.
1:38:00Simply authorize Post Hog Wizard and
1:38:02then that's it. You can return to your
1:38:05editor. The Post Hog wizard will now
1:38:07install the SDK, make it work on both
1:38:10client and server side, write the next
1:38:12public Post Hawk key into your ENV. It's
1:38:15public, so it's safe on the client, and
1:38:17create starter events and dashboards
1:38:19from your actual app. This means that
1:38:23it'll figure out what the app is all
1:38:25about and which events you might want to
1:38:28capture and then it'll proceed to track
1:38:30the entire application for you. So since
1:38:34we already have a lot of code in our
1:38:35codebase, let's give it a couple of
1:38:37minutes to analyze it all and
1:38:39self-implement itself. Then once it's
1:38:41back, we'll figure out exactly which
1:38:43events is it tracking. And there we go.
1:38:46It is done. On the previous screen, it
1:38:48showed me the full implementation and
1:38:50the link to the dashboard. I
1:38:52accidentally skipped it, so it led me
1:38:54directly through the MCP installation
1:38:56and post hog in Slack, but you were able
1:38:58to see it on your end. And we have the
1:39:00dashboard right here. It has been
1:39:02created for us. So, we can simply
1:39:04preview it. Check this out. This is
1:39:06pretty crazy. It pulled the analytics
1:39:08for the pipeline operations, the
1:39:11newsletter subscriber clicks, the
1:39:13articles views by bias label, so we know
1:39:16who is using it more or which articles
1:39:18are getting more clicked, as well as the
1:39:20article engagement wizard and article
1:39:22clicks over time. This data so far isn't
1:39:25real, but as soon as we start using the
1:39:27app, it'll start collecting real data,
1:39:29and we'll be able to see it right here
1:39:30within the dashboard. So, if you run the
1:39:33app on localhost 3000, reload it, and
1:39:36click over in a couple of articles,
1:39:38you'll be able to see some events are
1:39:40getting tracked. And back in the
1:39:42dashboard, if you head over under
1:39:43activity, you'll be able to see exactly
1:39:45what is happening. Post hog has been
1:39:48successfully wired up. But the next step
1:39:50that builds directly onto what we've
1:39:52done so far is to set up post hog in
1:39:54Slack. So we can ask it questions about
1:39:56product data, debug issues, and just
1:39:59generate PRs without even leaving the
1:40:01thread. So you can just open up the
1:40:03Slack one-click setup, which looks
1:40:05something like this. So you just have to
1:40:07allow it for your workspace. And that's
1:40:09it. We're connected. You can go back to
1:40:11Post Hog and keep using it through its
1:40:14dashboards. you know the general starter
1:40:16dashboard which is how we have been
1:40:18looking at analytics so far. You can
1:40:20also look at it through the wizard
1:40:22generated dashboard already much more
1:40:25valuable info or we can start asking it
1:40:28questions directly through our slack
1:40:30application. So let's keep using the app
1:40:32a bit more so that we can fully test it
1:40:34out. But before we do that I'm noticing
1:40:36at the bottom left corner that we have
1:40:38some issues. post hogjs exception
1:40:40autocapture failed to load script which
1:40:43I will copy and bring back to our cloud
1:40:46code. I'll say I just implemented post
1:40:50hog and this is the error that I see.
1:40:54And then we can just paste what we
1:40:55copied and I'll do the same thing for
1:40:57the second error right here talking
1:40:59about the surveys. And then I'll just
1:41:02paste both errors here and tell it to
1:41:04fix them.
1:41:06It'll now analyze the codebase and post
1:41:08hog's implementation and let's see what
1:41:10it makes of it. And after some fixing,
1:41:13it says for the exception auto capture
1:41:15error, there's a real trade-off since
1:41:17it's the feature you may want to keep.
1:41:19So, it's asking us the exception auto
1:41:21capture script is blocked by the add
1:41:24tracker blockers like some kind of a
1:41:26Ublock or Brave Shield. Our server proxy
1:41:29is fine. So, it's asking us how we want
1:41:31to handle it. And this is actually what
1:41:33I suspected will happen when we deploy
1:41:35the application. The console error won't
1:41:38be there because we're going to be in a
1:41:39real browser, not running it locally.
1:41:42And if the users are running it without
1:41:43a blocker, they're going to get full
1:41:45error tracking. So, yep, I think we're
1:41:47totally fine on this for now. We just
1:41:49disabled surveys and the rest works
1:41:51properly. This console log will be gone
1:41:53on the deployed version. So, now you can
1:41:55go through a couple of articles, explore
1:41:56the application a bit, maybe even log
1:41:59out and log back in. All of this data is
1:42:02now being tracked and I'll show you how
1:42:04we can make sense of it. So just try to
1:42:06use the application a bit more. Then if
1:42:08you head over to post hog you'll be able
1:42:10to see that all the data is properly
1:42:12getting picked up like even the identify
1:42:14where it identified our random account
1:42:17with this email and connected it. So now
1:42:19we have a bit more information about
1:42:21what this user is and what they do. So
1:42:23if you open up Slack, in this case I
1:42:25just created a new channel called Post
1:42:27Hog testing, but you can invoke Post Hog
1:42:30in any chat. You can simply ask it some
1:42:32kind of a question. For example, which
1:42:34articles got opened the most? And let's
1:42:38see if it can figure out what we're
1:42:39asking and deliver us a quality
1:42:41response. Since this is a private
1:42:43channel, we have to manually add Post
1:42:45Hog within it. And it already responded
1:42:48saying that we're connected to multiple
1:42:50projects. You can see that we're using
1:42:51this for the official jsmastery.pro
1:42:53domain. But now we can specify that the
1:42:55project we want to use it for is
1:42:57actually skew news. So let's invoke it
1:43:00like this. There we go. And we will
1:43:02restart the question. Immediately it
1:43:05flags it as in progress. And if you
1:43:07really want to eaves drop, you can see
1:43:08what the agent is doing. So it's taking
1:43:10all of this context figuring out what
1:43:13the query is all about using the MCP to
1:43:16extract the data and give you an answer.
1:43:19Or of course you don't have to do that
1:43:20and you can just stay within Slack and
1:43:22expect your answer back. And just like
1:43:24that the response is in within literally
1:43:27a couple of seconds. Here are the most
1:43:29opened articles by article viewed
1:43:31events. So this one which I clicked on a
1:43:34couple of times is the most opened
1:43:35article. It has been opened 13 times
1:43:38following by this one with 12 opens. All
1:43:41of this is early testing traffic. Now
1:43:44let's ask it a bit of a tougher
1:43:45question. something like hey post hog
1:43:49what's our drop off between the home
1:43:53feed and an article
1:43:56page I really want to know are we really
1:43:59activating the user to get them to check
1:44:01out specific articles and keep in mind
1:44:03that I didn't write the query or a
1:44:06tracking plan I simply described what I
1:44:09wanted and it gave me the answer from my
1:44:12own product data that is the
1:44:14self-driving that I was telling telling
1:44:16you about and the response is in. Right
1:44:18now, there's effectively no drop off to
1:44:20report because the project has just one
1:44:22person across all events. 29 article
1:44:26views, 13 article clicks from a single
1:44:28user. This is a testdev traffic. But you
1:44:31can see how meaningful this answer is.
1:44:34And as I told you, Post Hog doesn't only
1:44:36provide you the data that you have in a
1:44:38meaningful format. It can also draft a
1:44:41PR on its own. For example, you might
1:44:44want to track the amount of people that
1:44:46click the links at the bottom if that's
1:44:48not being tracked already. You can tell
1:44:49it to track that for you. Or maybe you
1:44:52want to know whether people are really
1:44:53interested in how we analyze bias. So
1:44:56you can tell it to add tracking to this
1:44:58button. I think this is a good test. So
1:45:01let's see if it can do that. I'll tell
1:45:03it, hey post hog,
1:45:06in the app we have a button that says
1:45:09how we analyze bias. I want to be able
1:45:12to track the amount of people that click
1:45:14on that button. If that's not being
1:45:16tracked already, can you make it happen?
1:45:20So, this is a pretty open-ended
1:45:21question, and I want to see how it'll
1:45:23react in this situation. And very
1:45:26quickly, Post Hog came back to me and
1:45:28said, "Or detracted. Uh, no code change
1:45:30needed. Uh, your app has auto capture
1:45:32enabled, so clicks on that button are
1:45:34being recorded automatically. 16 clicks
1:45:37so far, though all came from a single
1:45:39test user. And to make it easy to watch
1:45:41going forward, I set up two things.
1:45:43Clicked, how we analyze bias matches any
1:45:46autocaptured click on that button as
1:45:48well as the insight on the unique
1:45:50clickers. So it didn't just tell me,
1:45:52hey, this is already being tracked, but
1:45:55it also created the insights so that
1:45:57whenever I want to, I can come back on
1:45:59my dashboard and check that number out.
1:46:01And you can see that this was created
1:46:03for us. That's what the self-driving is.
1:46:05You tell it the destination, it drives,
1:46:08asks for your feedback if it's needed,
1:46:10and then it just brings you to the
1:46:11destination. Works very similarly to our
1:46:14current agentic engineering setup with
1:46:16the agents MD and the prompts where AI
1:46:19is technically self-driving itself, but
1:46:22you're still the architect behind it.
1:46:24And I embrace this change. With the
1:46:27self-driving mode, Post Hog is
1:46:29challenging the status quo where you're
1:46:31just looking at data points within a
1:46:33dashboard. Why do you need a dashboard
1:46:34if you can simply ask the agent what you
1:46:37want and preferably you can ask that
1:46:39question directly within the platform
1:46:41that you already use to communicate with
1:46:42the other teammates. Great stuff. So
1:46:45with this done, let's improve our
1:46:46application even further.
1:46:50SK News now works end to end, but every
1:46:53part is manual. We trigger scraping and
1:46:56analysis by hand. So this lesson is the
1:47:00last piece of the build because we'll
1:47:02automate the whole thing so that SK News
1:47:05pulls and analyzes fresh news every hour
1:47:08on its own. So if you head over to
1:47:10agentsmd and that's going to be under 18
1:47:14right right here we're going to talk
1:47:17about the oxyabsuler.
1:47:19We'll use this feature to run hourly
1:47:21scraping for active source homepages
1:47:24stored in superbase. scheduler should
1:47:27scrape source homepages only. And
1:47:29there's one distinction I want to make
1:47:30here. This is two separate one-time
1:47:33setups. We'll create oxyab schedules but
1:47:36also configure versel cron jobs as two
1:47:39independent one-time setups and neither
1:47:42one triggers the other. The postpi oxyab
1:47:46schedules tells oxyabs what to scrape
1:47:48hourly. It's done once per source. But
1:47:51then the versel cron config will tell
1:47:53versel to call api chron p p p p p p p p
1:47:55p p p p p p p p p p p pipeline at 15
1:47:56minutes past every hour and both must be
1:47:59completed for the pipeline to be fully
1:48:01automatic. Until Versel chron is
1:48:03configured, the process route must be
1:48:05called manually. So in simple words, the
1:48:09Oxyabuler tells Oxyabs to scrape our
1:48:12active source homepages every hour and
1:48:14then store the results. And then
1:48:16Verscelron tells Verscell to call our
1:48:18pipeline 15 minutes later, take these
1:48:20stored results, turn them into articles,
1:48:23and analyze them. That's also only set
1:48:25up once. So you're wondering why have
1:48:28two and not just one like scrape, store,
1:48:31and analyze in a single shot. There's a
1:48:34couple of reasons. First is that the
1:48:36Oxyabuler doesn't call our app. It runs
1:48:39the scraping on OxyLabs's own servers
1:48:42and stores the results there. It's not a
1:48:45web hook that hands us a finished
1:48:46payload. So, we have to always come back
1:48:49and fetch the completed results
1:48:50ourselves, which is what the Verselron
1:48:53does. That's also why it waits 15
1:48:55minutes. The scraping isn't instant, and
1:48:57at the moment the job kicks off, there's
1:48:59nothing ready to process yet. So even if
1:49:02we could do it in one request, we
1:49:04wouldn't want to. Scraping every source,
1:49:06parsing detailed pages, and then running
1:49:08AI analysis on them in a single HTTP
1:49:11call would blow straight past serverless
1:49:14function time limits, and it will just
1:49:16time out. But splitting it, OxyLabs
1:49:19handles the scraping on its side, and
1:49:21our chron processes the results and then
1:49:23analyzes them in batches. So to
1:49:26implement it, go ahead and open up a new
1:49:29agent window and tell it to implement
1:49:31the Oxyabuler with Verselron for
1:49:34automatic hourly processing and use the
1:49:37OxyLabs web scraper as well as the
1:49:39Superb Skills to do it. This is the
1:49:41biggest feature in the project and it's
1:49:44going to ship several parts together. So
1:49:47let's give it some time and then let's
1:49:49review the prompt it generates
1:49:50carefully. It thought a bit and told me
1:49:53that it has a complete picture. So the
1:49:55oxyab schedules and oxyilab schedule run
1:49:58tables plus the types already exist. The
1:50:02scrape pipeline or the exposes the
1:50:04process source HTML. The run analysis is
1:50:07a reusable step. Manual scraping uses
1:50:10the real-time endpoints and theuler uses
1:50:12the data API which means that a new
1:50:14client module is needed. It all makes
1:50:16sense. So based on it, it prepared a
1:50:19complete prompt to execute. We can
1:50:22review it here. The things that we care
1:50:24about the most is that it delivers all
1:50:26the pieces at once. The goal here is to
1:50:29implement the OxyLabuler and Verselron
1:50:32layer so that the active source
1:50:34homepages are scraped automatically
1:50:35every hour and then turned into analyzed
1:50:37articles with no manual intervention
1:50:40after one-time setup. What we care about
1:50:42the most is that it delivers all the
1:50:44pieces at once. All of this is looking
1:50:46good to me. So, we can go ahead and
1:50:49confirm it. I'll simply go ahead and say
1:50:51implement. And very quickly, the
1:50:53response is in. It built the Oxyabs data
1:50:57API client with create schedules,
1:51:00listing schedules, getting scheduled
1:51:02runs using the forward/runs with per job
1:51:05result status, never just jobs. It sets
1:51:08the schedule state and gets the job
1:51:09result HTML. The schedule ids are
1:51:12extracted from raw response via reax and
1:51:14kept as strings. DB layer wise, it
1:51:17created the queries for the schedules so
1:51:19that now we can fetch them. As far as
1:51:21the pipelines go, it'll happen on an
1:51:24hourly basis. And then after that is
1:51:26processed, we will analyze them. So
1:51:28here's how we can test it out. First,
1:51:30let's create and sync Oxyab schedules,
1:51:33one per active source plus deactivate
1:51:35orphans. We can do that by running the
1:51:38curl and we have to pass her admin
1:51:40secret. So let's give it a shot. I'll
1:51:43open up my terminal.
1:51:45I'll do it in a new terminal window and
1:51:49I'll manually provide the admin secret
1:51:52right here.
1:51:54Press enter and it'll start doing its
1:51:56thing. You can see that it is either
1:51:58creating new schedules or deactivating
1:52:01the orphan schedules. To be able to see
1:52:03this, I switched over to my terminal
1:52:06where we're running the application. And
1:52:07you can see that the sync has been
1:52:09completed. Sources checked five
1:52:11schedules created five. And that is it.
1:52:14We can see the schedules for all of the
1:52:16major newspapers. Then we can list the
1:52:18stored schedule rows. This should be
1:52:20simple. And you can see that we get back
1:52:22the data. And then after Oxyabs runs top
1:52:26of the hour, process the completed
1:52:28results. So I will run it even though
1:52:30Oxyabs didn't run yet. and press enter.
1:52:33And this is what we got back. After
1:52:35this, we can run the fourth command
1:52:37which shows us the runs. And then
1:52:39finally, for automatic pipeline locally,
1:52:41we can run the pipeline. This will give
1:52:43you some more information on what's
1:52:45happening. But for all this to work, we
1:52:47also have to set up versel cron. So if
1:52:49you head over to a versel.json
1:52:53file, you should be able to see a
1:52:55pipeline created automatically for us.
1:52:57The path points to a API cron pipeline
1:53:00and it's going to run 15 minutes past
1:53:02every hour. So with the last command we
1:53:05ran which was this one, we just tested
1:53:07it manually without waiting for it to
1:53:09get scheduled. To verify it, you can
1:53:11head over into your table editor and go
1:53:14under the Oxyab schedules.
1:53:18It has to have one active schedule per
1:53:20source. As you can see, we have five
1:53:22sources. So five schedules makes sense.
1:53:25And you can also check the OxyLabs
1:53:28schedule runs. Right now, this table is
1:53:31completely empty, but later on once it
1:53:34runs, we should be able to see them
1:53:35right here. We ran these manually, so I
1:53:38expected that maybe something is going
1:53:39to be there already. So, what I can do
1:53:41is copy all of the outputs that we got
1:53:44from running these commands. And then we
1:53:46can paste it over into our agent and
1:53:48tell it these are the commands that I've
1:53:51ran. Paste them right here. and then
1:53:53specify that oxyabs schedule runs is
1:53:56still empty. So we can say but oxyilabs
1:54:00schedule_r runstable is still empty.
1:54:06And let's see what it says. And very
1:54:08quickly it says that everything checks
1:54:09out. The empty run table is correct, not
1:54:12a bug. Oxyabs itself confirms it. Here's
1:54:15the BBC schedule reports. The schedules
1:54:18were created at 11:05 and the cron is 0
1:54:21hours. top of every hour, which means
1:54:23that nothing runs until 12 on my end. So
1:54:26there's genuinely nothing to process
1:54:28yet. The parser verified it against real
1:54:30data. So the 1200 hours pipeline will
1:54:33find the completed run, record it, fetch
1:54:36the homepage HTML, scrape it, and then
1:54:38mark the job as completed. So in this
1:54:40case, there's nothing left for me to do.
1:54:42then just wait, which I guess is the
1:54:44whole point of the application for me to
1:54:46not go ahead and manually browse the
1:54:48newspapers, try to figure out the bias
1:54:50and and political agendas, but rather
1:54:53wait on the dashboard and potentially be
1:54:55notified when the results come back in.
1:54:58So, while we're waiting for this, I
1:54:59actually want to go a step forward and
1:55:02deploy the application. We want to put
1:55:04it online, and there's a couple of
1:55:06reasons to deploy it. The first one is
1:55:08that deploying it will make the
1:55:10application public. So other users will
1:55:12actually be able to use it. And the
1:55:15second is that it's the only way for the
1:55:18automatic hourly pipeline to actually
1:55:20run because Versel cron will not fire on
1:55:24your laptop. Rather, it runs only on a
1:55:27deploy project. So at this point in
1:55:30time, our application is finished. We're
1:55:33scraping the articles and analyzing them
1:55:35for bias. So let's go ahead and deploy
1:55:38it. push all of the changes you have so
1:55:40far onto GitHub. You can do that easily
1:55:42by heading over to github.com/new.
1:55:46Then you can choose your repo name such
1:55:48as skew news and simply create a new
1:55:52repo. Then you can copy some of these
1:55:54commands.
1:55:56So first we run git init
1:56:02commit-m first commit.
1:56:05We can also branch over to the main
1:56:08branch from master and then add a remote
1:56:11origin to this remote repo. Finally, we
1:56:15can push it over to origin main, which
1:56:18will bring all of our code right here to
1:56:20GitHub. Once the repo is right here, you
1:56:23can head over to versel.com and add a
1:56:26new project. If you head over to your
1:56:28account, you should be able to see skew
1:56:30news just now. So, you can very easily
1:56:33import it. And we then have to set up
1:56:35the environment variables. And this is
1:56:37the step that most people get wrong. You
1:56:39want to head back here and copy all your
1:56:42local environment variables. Then simply
1:56:44paste them here. And there's an
1:56:46additional variable that you have to add
1:56:48for cron jobs to work. And it is called,
1:56:52let's add it right here, cron
1:56:55secret. And this has to be a string of
1:56:5916 or more characters. That's the
1:57:01recommended value. and make sure to copy
1:57:03it in case we need to use it later. For
1:57:06now, click deploy. And let's give it
1:57:08some time. It's first going to
1:57:10initialize the dependencies and then
1:57:12build the project. And that's it. We
1:57:15just deployed a new project to JSM Pro
1:57:17repo. For you, it's going to be your own
1:57:19name. But yeah, here we are. You can
1:57:22open it up and you can see that that
1:57:25post hog warning is now gone. And we can
1:57:28see the app right here. The next and
1:57:30optional thing is to point clerk at the
1:57:32deploy domain because the test keys work
1:57:34only for local development. But for the
1:57:37live site, you'll need to configure your
1:57:38clerk instance for production domain and
1:57:41then add the versel URL to your clerk
1:57:43application. For now, it is still using
1:57:45the local one. So I'll proceed to sign
1:57:47in and we're in. So now we have to
1:57:50confirm that the cron job got
1:57:52registered. Head back over to Verscell.
1:57:54Open up your project. You can use the
1:57:56find right here to search for the cron
1:57:58jobs under observability and you can see
1:58:01whether it picked up on it. For me, it
1:58:03isn't here. So, we can further debug it.
1:58:05I'll simply copy this page, at least the
1:58:08most relevant information,
1:58:10head back into my VS Code, and open up
1:58:12my conversation with the agent. In this
1:58:14case, I'll open up the conversation that
1:58:16we've had before because what we're
1:58:18doing is still relevant to this
1:58:19implementation.
1:58:21and I'll tell it I deployed the project
1:58:24over cell and this is what I see on the
1:58:27cron settings page. This should give it
1:58:29enough insight into what I'm trying to
1:58:31fix. I was assuming that because we have
1:58:33this forc cell JSON, it'll be able to
1:58:35automatically pick it up. In this case,
1:58:38no data found for the last 12 hours
1:58:40doesn't necessarily mean anything is
1:58:42broken. That panel shows cron executions
1:58:45which only appear after the job has
1:58:47actually fired. But it's checking the
1:58:49other gotchas right here so we can
1:58:50verify everything. Specifically talking
1:58:53about the Verscell plan limits as
1:58:56different plans might allow you to run
1:58:58chron jobs at different time points. For
1:59:01example, it says right here that our
1:59:03pipeline is set to 15,
1:59:07which is likely a blocker because it
1:59:09depends on the versel plan. So the hobby
1:59:12plan hourly limit which is most likely
1:59:14what you have limits the cron jobs to
1:59:17once per day. So the cron expressions
1:59:20that would run more frequently will fail
1:59:22during deployment. Our schedule is set
1:59:25to hourly and I can run that because I
1:59:27have a versell pro plan. But for you you
1:59:29might need to switch it over to every 12
1:59:32hours. So it's asking us which versel
1:59:34plan is the project deployed on. Since
1:59:37I'm on pro I will select pro. And again
1:59:40to fix it for you, you can select hobby.
1:59:43Considering that I'm on the pro plan,
1:59:45everything on the code side is correct
1:59:47and shipped. We just have to wait a bit.
1:59:50Confirm registration, not observability.
1:59:52So go to settings, cron jobs. If we go
1:59:55here once again, you can search for cron
1:59:57jobs under settings, not observability.
2:00:00And you can see that our API cron
2:00:02pipeline had been added here. Then
2:00:05verify its production deployment. That
2:00:07seems good to me. set the cron secret
2:00:09which we already have and then run a
2:00:12quick manual test. Then wait for the
2:00:14next 15 which means that everything is
2:00:16working and that in about half an hour
2:00:18from now I'll be able to get all the new
2:00:21articles scraped and analyzed right here
2:00:24on my landing page. So take a second to
2:00:27look at what you've built. SK News
2:00:30scrapes real news from multiple sources,
2:00:33analyzes each story for sentiment and AI
2:00:37estimated framing, and shows it all in a
2:00:40clean authenticated UI, and even
2:00:42refreshes itself every hour,
2:00:46deployed and running on its own. That's
2:00:48a real production style product. But
2:00:52this is just the starting line.
2:00:53Everything we build is a foundation that
2:00:55you can keep building on. And the way
2:00:57you'd add to it is by using the same
2:00:59loop that we've used the whole time. You
2:01:02can further let signed in users bookmark
2:01:05or save articles. The clerk and
2:01:07superbase combo should make that super
2:01:09simple. You can add search topic filters
2:01:12or trending session. You can implement
2:01:14connected related articles by using PG
2:01:17vector or even send a daily or a weekly
2:01:20email digest on top of the analyze
2:01:22stories. or just niche it down and sell.
2:01:26With Clerk billing, you can super simply
2:01:29add a pricing plan to your application.
2:01:31And let's be honest, that serious
2:01:33scraping that we're doing with Oxyabs
2:01:35really makes it a paid product. So,
2:01:38don't just sit here and watch me do
2:01:40things. Actually go on and try to
2:01:43implement some of these. It's the same
2:01:44loop every time. Decide what the feature
2:01:47is and write the rule into the agents
2:01:50MD. Once that is done, open up a new
2:01:52window and give a quick prompt
2:01:54explaining it. The loop doesn't care
2:01:57whether the feature is small or large.
2:01:59It scales with you. The AI didn't
2:02:02replace us here. It simply amplified
2:02:04what we can do. It took our decisions
2:02:07and turned them into working software
2:02:09fast. So, the skill that matters isn't
2:02:12coding. It's knowing what to build and
2:02:15being able to direct it well. and the
2:02:17full agentic engineering course that
2:02:19takes the same approach and pushes it
2:02:21even further with skills that plan the
2:02:24reviews themselves, legacy code bases,
2:02:27recovering broken projects, and agents
2:02:29running in parallel. It's not out yet,
2:02:32but I'll leave the link down in the
2:02:34description that gets you on the wait
2:02:35list and the best price once it goes out
2:02:38before the public launch. So, it's worth
2:02:41grabbing now while it's open. So, thank
2:02:43you for building this amazing
2:02:44application with me and I'll see you in
2:02:47the course. Have a wonderful day.

About this transcript

This page contains the full transcript of Learn AI Engineering in 2026 | Build a Full Stack AI Websites with Agents by JavaScript Mastery, generated from the public captions YouTube serves with the video. The transcript has 20,575 words across 2,963 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.