YT YouTube2Text

YouTube2Text → Transcripts → Spec-driven development: the AI engineering workflow at Notion | Ryan Nystrom

Spec-driven development: the AI engineering workflow at Notion | Ryan Nystrom — Transcript

by How I AI · 8,848 words · 1,337 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00One line that I've been putting in my
0:01prompts lately is I literally don't know
0:03what I'm doing here. You got to explain
0:05it like I'm a five-year-old. I didn't
0:06start with writing code. I didn't start
0:08with anything. I just started with an
0:10empty markdown document. I actually just
0:12opened up Whisper and just started
0:14yapping about how this feature should
0:16work. I gave the yap session to Codex
0:18and was like write us back. I then
0:20opened up Codex again, pointed it at
0:22this spec file, and I said build it. And
0:26basically one-shotted this. I've been in
0:28software engineering for 20-plus years.
0:30>> We were writing these documents and we
0:31were sitting in meetings with other
0:33engineers debating the merits of one
0:36implementation versus another. And now,
0:38no more waiting for the meeting, no more
0:40waiting for review.
0:41>> I'm not a CI expert, but I kind of know
0:44what I want. And so, other folks were
0:46kind of like, "Can you just bring some
0:48of your like puppy dog energy to like CI
0:51and just see what we can do?"
0:53>> Your AI, your agent is never going to
0:54complain when you ask it to do this five
0:57minutes before the meeting starts.
0:58>> It is more relaxing and it's more fun
1:01and I feel like I'm getting more done.
1:02It's weird to have this like
1:03win-win-win.
1:04>> They do the triangle and they're like
1:05pick two and you're like, "No, I want to
1:07pick all three."
1:07>> Give me the whole triangle.
1:08>> Give me the whole triangle.
1:13Welcome back to How I AI. I'm Clara
1:16Bell, product leader and AI obsessive
1:18here on a mission to help you build
1:19better with these new tools. Today we
1:21have Ryan Nystrom from Notion and he's
1:23going to show us as an engineering
1:25manager how you can never prep for a
1:28stand-up again. We're also going to see
1:29how you can get a background agent to
1:31write code for a fix your friend texted
1:33you and how spectrum and development
1:36really works in a code base at scale.
1:38Let's get to it. This episode is brought
1:41to you by WorkOS. AI has already changed
1:44how we work. Tools are helping teams
1:46write better code, analyze customer
1:48data, and even handle support tickets
1:50automatically. But there's a catch.
1:52These tools only work well when they
1:54have deep access to company systems.
1:57Your co-pilot needs to see your entire
1:59codebase. Your chatbot needs to search
2:01across internal docs. And for enterprise
2:04buyers, that raises serious security
2:06concerns.
2:07That's why these apps face intense IT
2:10scrutiny from day one. To pass, they
2:12need secure authentication, access
2:14controls, audit logs, the whole suite of
2:17enterprise features.
2:18Building all that from scratch, it's a
2:20massive lift. That's where WorkOS comes
2:23in.
2:24WorkOS gives you drop-in APIs for
2:26enterprise features, so your app can
2:28become enterprise-ready and scale
2:30upmarket faster. Think of it like Stripe
2:33for enterprise features.
2:35OpenAI, Perplexity, and Cursor are
2:37already using WorkOS to move faster and
2:40meet enterprise demands.
2:42Join them and hundreds of other industry
2:44leaders at workos.com.
2:47Start building today.
2:49Ryan, welcome to How I AI. I am really
2:53excited because you're going to show us,
2:56I think,
2:57start,
2:59medium, advanced mode on some AI coding
3:03stuff. And so, before we jump in, how
3:06has AI just changed how you live your
3:09life at work?
3:11>> I mean, it's it has completely upended
3:14the way I work. Like, I I've been doing
3:16this, I did a lot of like mobile iOS
3:19work in my past, and I worked the same
3:22way
3:23every day for like 12-plus years. And
3:26then, in the last year, I have changed
3:29IDEs, terminals,
3:32tools, like whatever, like
3:3510-plus times. Um
3:38so, it's like really weird and scary to
3:40be like changing this stuff so much, but
3:42it's also I feel that like
3:44I've been doing this for a while, and
3:45I'm feeling like so much like joy and
3:47freshness and like newness in this that
3:50like I wake up every day super excited
3:52to like tinker and build things.
3:55And I'm also like working faster and
3:57harder than I feel like I ever have, but
3:59like in a in a good way. I know people
4:01are like
4:02kind of freaked out about like
4:03everything's changing, the pace is up,
4:05but like it's it's really energizing for
4:07me.
4:08>> Well, you are not alone. I think every
4:09how I I guess has come on and said I'm
4:11having more fun. I'm working faster and
4:15everything is different. And what I love
4:16about what you're going to show us is
4:19it's not just the set of tools, which I
4:21think has changed or how we write code
4:23has changed. How you like run a team has
4:26changed. So, and I think for the better.
4:29You know, so I would love to see how
4:32you're using Notion AI to actually run
4:35teams differently.
4:36>> Yeah, for so for context, I I manage a
4:39team
4:40like six, seven people. I'm like an
4:43engineering manager
4:45technical engineering tech lead manager
4:49or whatever we call it. I manage people
4:51and I write code
4:52is like my role, which I love. And I've
4:56run a bunch of different projects here
4:57at Notion. The the one I'm going to show
4:59today we've we've nicknamed Afterburner.
5:03Like kind of a quick backstory is I've
5:06been kind of vocal about our like DevEx
5:09CI for a while. I've just
5:12I worked at places where it's really
5:13slow. I've worked at places where it's
5:15really fast. And I came here and I was
5:16like we're kind of like in between.
5:18But I feel like we're like slower than
5:21we need to be.
5:22And eventually this like caught up to me
5:24and somebody was like could you just
5:27come fix it and like come work on it?
5:29And I'm not like a infra expert. I'm not
5:31a CI expert. But I I kind of know what I
5:35want. And I think also most importantly
5:38the the group that I manage and the the
5:41org that I work in
5:42we're a little notorious for being like
5:45really fast and like very, very AI
5:47piled. And so, other folks were kind of
5:49like, "Can you just bring some of your
5:52like puppy dog energy to like CI and
5:54just like see what we can do?" And
5:57that's what we've That's what we've
5:58done. So, we had this really aggressive
5:59goal to cut our CI into like a quarter
6:01of what it is. Um we're on the path to
6:05to doing that, but so, I what I want to
6:07show you is a little bit about how we
6:09run projects in Notion and how I'm using
6:13um AI to kind of streamline
6:15those projects. So, what we're looking
6:17at here is like our like project hub
6:20called Afterburner. And so, in here,
6:23I've got all this documentation. I've
6:24got databases. Um we'll look at some
6:27meetings. I have like a automation that
6:29looks for any sort of like little wins
6:31if we knock off seconds from different
6:33jobs or whatever and we keep it we keep
6:35track of them in here.
6:37But, what I want to show you is
6:39basically how we run our meetings. Our
6:42small group, we run a stand-up every
6:44single day.
6:46And
6:47doing stand-ups where everyone just like
6:50is kind of like dead-eyed and going
6:52around being like,
6:54"I did this. I shipped this change." Or,
6:57you know, "No updates for me. Thanks."
6:59Is like
7:00painful. And in my opinion, like a huge
7:03waste of time. I want to like get to the
7:05the meat. So, we have this kind of like
7:08automated meeting template that shows up
7:11every single day that we run our
7:12meetings, which is basically every day.
7:14And it starts blank. And then, what I
7:17have set up behind the scenes uh is a um
7:21custom agent. So, we shipped this Notion
7:23AI custom agent stuff a little bit ago.
7:26And this runs right after the meeting
7:29template gets generated. And it looks
7:32through all of our like Slack
7:33conversation in the last 24 hours, any
7:37tasks in Notion that we close, any
7:40um
7:41pull requests that we've like merged,
7:43just like all sorts of contexts. Oh, and
7:46it looks at like yesterday's meeting
7:47transcript as well. And then it compiles
7:50a basically like a pre-read. And um this
7:53is a this is one from a week or so ago.
7:57And it Yeah, it shows it pulls metrics.
8:00It can show us like what our latest CI
8:02time is. Uh it shows some of the things
8:05we've decided. Shows like progress on
8:07different like projects or like
8:09different things that we're trying to
8:10like make faster.
8:12Bugs, feedback, open questions, ta- like
8:16anything that's like of concern. And
8:18like
8:19I
8:21can basically work up until like the
8:24minute of our meeting
8:27without having done a bunch of like
8:29prep. And then we all get on a video
8:31call and we look at the screen and we're
8:34like,
8:35"Okay, here's what we need to talk
8:36about."
8:37And we'll like hit each bullet. We have
8:41like meeting notes that we run down here
8:43um within Notion.
8:45And like all the context is basically
8:46like captured. All of the like agenda is
8:48like
8:49set up for me. So, we spend the entire
8:51time talking about like problems,
8:53decisions, wins, findings, like what are
8:56we going to work on next? And it's less
8:58the like
8:59"Oh, I did this thing."
9:00>> Yeah, what I want to call out for folks
9:02that are maybe listening and not
9:03watching is this is a very detailed
9:07meeting like kind of like pre-read /
9:10status update. Which if you're a you
9:13know, TLM or an EM
9:15running good meetings unfortunately is
9:17part of part of the job and there's a
9:20really big difference between a good
9:21stand-up and a bad stand-up. And I think
9:24the the ones you describe, these like
9:25wrote like, "I wrote this PR today. I'm
9:28going to start on this." And that
9:30basically like a notes document that's a
9:33very high-level because some human is
9:35putting it together, it stinks and
9:39you start to like do I the other thing
9:41that I found and maybe I'm I'm curious
9:43if this has impacted how you work or
9:45just made it easier, which is like you
9:47start to have those meetings less
9:48because the updates aren't super rich.
9:52People don't feel like it's a good use
9:53of their time and I think you lose
9:55something by reducing the frequency, but
9:57if you can have high bandwidth,
9:59high quality meetings
10:02with high frequency without the
10:04overhead, I think you can get better,
10:07more detailed work done collaboratively,
10:10especially if you're running like a
10:12remote team or not everybody's in the
10:14same same room.
10:16So, I'm curious, do you feel like just
10:18being able to get to this level of
10:19detail in your stand-up has improved how
10:21you can actually build this product and
10:23do these do this work, basically?
10:25>> I've been in way too many meetings where
10:27I can tell like everybody's eyes are
10:29glazed over, nobody's paying attention
10:32and I like I have
10:34run enough projects at this point where
10:37like to me that's like pretty dangerous
10:40cuz like everybody in that room has
10:42ideas, they have insight, but like
10:45because we've made it like so
10:47non-productive and like not engaging
10:49that like we're not we're just like we
10:51need the point of the meeting is to like
10:53exchange ideas and information and if
10:54we're like not doing that, like
10:56literally what's the point? The whole
10:57like this could have been an email, like
10:59that's what the meme is from. Um
11:01and so I I found that this is basically
11:05a good conversation starter.
11:08Um
11:08and
11:10a key thing to me that that tells me
11:13that this is working is when I'm like
11:16I'm running these meetings and kind of
11:18going through like bullet by bullet and
11:19then I see like oh, somebody like
11:22fixed our mock server environment and
11:24our our just tests and we're seeing a
11:26test improvement by like up to 13%. Like
11:29oh,
11:30I'd miss that, you know, Like
11:32that's super cool. Like tell like let's
11:34talk about it. And then from that maybe
11:36like oh, there's additional head room
11:38that we can make on this. And so all of
11:40a sudden like let's drill into it. And
11:43so just like
11:44I also think it sort of democratizes
11:46that like sharing so that you've got
11:48some people that um
11:51I I could talk for an entire 30 meeting
11:53without shutting up. And then other
11:55engineers are like you know, kept
11:57keeping to themselves but like brilliant
11:59and super talented. And I think it's
12:00great. This kind of like raises
12:03everything up to the top. Um it makes it
12:05very visible.
12:06>> I was I was thinking last night truly
12:08this is a little bit of a sidebar but I
12:09was thinking last night how people don't
12:11know this. It's very very confusing as
12:14Claire Bow the podcast host. I am such
12:16an introvert. I'm a crazy
12:19introvert. I will avoid humans with
12:21every fiber of my being.
12:23And I find like AI as a proxy for me to
12:27get over my own anxiety communicating
12:29with the
12:29It's so insane. But I really feel it.
12:33Whenever I like message my open claw and
12:36I'm like hey Polly, can you email
12:38so-and-so blah blah blah on my behalf?
12:41My anxiety is gone. But when I sit in
12:43front of my Gmail and start typing an
12:45email my introvert starts showing and
12:47I'm like I can't I can't do it. So I
12:49think this is a real point which is like
12:51you got to pull information out of
12:52everybody.
12:53>> Yeah.
12:54>> And there's like spikes and valleys of
12:56who is going to come and show off their
12:58work, how detailed they're going to be.
13:01It does not mean the work is not good.
13:02It does not mean they are not talented.
13:04They just have different skills in
13:06different environments. And so I love
13:08this call out of like we're going to
13:09pull equal amounts of information out of
13:12everybody because the playing field is
13:15is equal. I also want to go back to
13:17something you said which makes me think
13:18about burnout. Which you said I can like
13:20work up until this meeting starts. And I
13:24I know so many people who feel like
13:27they're in meeting after meeting after
13:29meeting one because when they're not in
13:32meetings they're preparing for meetings.
13:33So you're getting rid of that.
13:34>> Yeah.
13:35>> And then two I think managers being able
13:39to for example code
13:42up until the stand-up is such a burnout
13:45like protection mechanism which is you
13:48would much rather your managers hands-on
13:51the code hands-on doing work filling a
13:53creative impulse
13:55than prepping for meetings. Sure they're
13:57great at running the meetings but let's
13:58do that in like this just-in-time
14:01mechanism where they're supported um
14:04through these automations and so I just
14:06imagine and I used to feel so stressed
14:08as a leader and as a manager being like
14:11if I'm not in a meeting I'm prepping for
14:12a meeting and then I'm in a meeting and
14:14if I can carve back more of my time to
14:16just do real things
14:18that feels so much so much so much
14:21better. So I'm guessing you're having
14:23more fun just through the balance of the
14:25kind of work you're trying to do.
14:27>> I'm yeah I'm having so much more fun
14:30working this way. I I have done the like
14:33run a big engineering group where yes
14:35you're spending half of your time just
14:38compiling information synthesizing it
14:40writing reports like doing all this
14:43stuff and I I hated it. Like it's so
14:47draining. And now I feel like I'm in a
14:49sweet spot where I can support and work
14:52with a team of like very talented
14:55individuals
14:57but also not have to yeah be like doing
15:00like paperwork the entire time. Like I I
15:02don't want to do like the tedium. I get
15:04a lot of joy out of working with people.
15:06It's like why I like managing. I like
15:08talking with them having fun. I love
15:10like solving hard problems together.
15:13And then I like building stuff and
15:17yeah I think that we're maybe at an
15:20inflection point where
15:23like
15:24I I don't maybe this is controversial or
15:25not, but like if you're like a line
15:27manager, you like
15:30write write code, you know, get in there
15:32and like
15:33stay stay close. Maybe don't do the the
15:35the P0 hero projects, but like help your
15:38team fix bugs, like make optimizations,
15:40like whatever, you know, I it's just
15:42it's so easy now.
15:44>> I mean, I'm going to pull that thread
15:46all the way up, which is like directors
15:48of engineering, VPs of engineering, like
15:51CTOs.
15:52>> Yeah.
15:52>> CPOs, it write write some code. Now is
15:56the time. And I say this all the time on
15:58this podcast, this is the era of the
16:00hard skill. This is not how do I get
16:03better at my my soft skills and managing
16:05people? This is literally like how do
16:07you write code? How do you write
16:08automations? How do you learn these new
16:09tools? How do you understand what models
16:11do do what for your own skills?
16:14That I think is super important. Okay,
16:15we can go on forever on this topic. I do
16:17want to show folks how you built this.
16:20>> Yes.
16:21>> Because we haven't seen actually haven't
16:23seen we've had a couple of Notion folks
16:24on the podcast, we haven't seen Notion
16:26AI in action and I just want to see kind
16:29of your thought process on how you build
16:31something like this out.
16:32>> So I'm going to flip over to this is our
16:35custom agent. Don't ask me why we got
16:37this like potato theme for like the
16:38entire project. I think it was kind of
16:40something about like CI is just this
16:42like cobbled together like mess and so
16:44we're going to like make the potato like
16:46a rocket ship. I don't know. I don't
16:48even know if that makes any sense, but
16:50like now we're having fun with it and we
16:51have
16:52reactions and like agents and it has
16:56spun off into its own thing, which is
16:57fun.
16:58But this is our hot potato agent. So you
17:01can see I have this set up to run at
17:039:00 a.m. every single day. It's also
17:05set up for chat. It's set up if the
17:07agent's mentioned, but we never really
17:09use any of that. And probably the
17:11important part is the actual
17:13instructions. So in this instruction
17:16page
17:17giving context on like what the purpose
17:19of this agent is, I'm telling it to run
17:23Yeah, look back at 24 hours. So,
17:25basically telling it, "Your job is to
17:27run every single day, and I only want
17:29you to look back for the last like 24
17:31hours of activity." I'm explicitly
17:33telling it to use sub-agents, which is
17:35kind of a sleeper feature in Notion AI.
17:38Like, this exists, but we don't we don't
17:40really push it to use it very often yet,
17:43cuz it's one, it's very expensive, and
17:45two, it can be kind of finicky
17:47sometimes, but um
17:49I helped build it, so I'm I know how
17:51this works.
17:52Uh
17:53And then I ask it to kind of like fan
17:55out and do like a map reduce, where I'm
17:57saying, "Go use the honeycomb MCP to
17:59figure out what the latest metric is.
18:01Look in our project channel and like
18:04find updates, feedback, questions." I
18:07tell it where the task uh databases and
18:10how to look for tasks within this
18:12project, and then how to find
18:14yesterday's meeting.
18:15And then
18:17I give it a template in the
18:18instructions, where I'm like, "This is
18:20your format. I care about CI speed,
18:22decisions, progress, changes, bugs,
18:25questions, risks."
18:27A little bit of guidance on writing.
18:30And then when it's done, uh I have it
18:32post to Slack, and I like emphasize this
18:34like I want it to be brief and fun, and
18:36sometimes it's really corny, and then
18:38sometimes it's like
18:39really good, uh and it'll be very
18:42quirky, and just post this like link in
18:45our Slack channel, and it's like, "Hey,
18:46here's your your pre-read." Some little
18:48quibble about like whatever, you know,
18:51like, "Hey, you guys are not making
18:52enough progress." Uh and then that's it.
18:55And then it's we have our meeting note,
18:57it's updated.
18:59What I like the most about this one, you
19:01can show you some of our like internal
19:04settings. So, this is like I give it
19:06access to all of these things. I'm like,
19:07"You can only view all of this stuff,
19:09cuz I don't want it going and like
19:11modifying our task database, our project
19:13database. Like everybody at Notion uses
19:15those. But this meetings database in
19:18particular, I'm like, "Yo, you can edit
19:20content cuz this is the one you're going
19:21to like write and update the page."
19:24It can read from our Slack channels,
19:28respond to our project one.
19:31And then this was new to me actually
19:33when I set this agent up is our MCP. So,
19:37we've had MCP and
19:39>> They have this
19:40>> other thing called workers which is like
19:42kind of like writing code. I haven't
19:44used them very much within custom
19:46agents, but in this one in particular,
19:47I'm like, "I know exactly where this
19:49metric is. It's in Honeycomb."
19:52And so I like just configured the MCP in
19:55Notion.
19:56Uh by the way, I like I like used the
19:59agent like set itself up cuz I was like,
20:01"Here's the query." I literally gave it
20:03a screenshot of the Honeycomb query. I
20:04was like, "I don't know how this works.
20:06Can you just like update your
20:07instructions?" And
20:08>> that you screenshot it. You didn't even
20:10copy and paste it. You're like, "Please
20:12OCR this screenshot."
20:15>> Too lazy. I'm like, "Here it is. Just
20:17take it.
20:18Figure it out." And it it kind of it got
20:20it mostly. Like most of the way there. I
20:23had to fiddle it with it a little bit,
20:24but yeah.
20:26>> What what I appreciate about this, and
20:28again, for anybody trying to just
20:29brainstorm
20:31workflows where AI can actually have a
20:34huge impact on your productivity at work
20:35or life, I just feel like write down
20:38what you would do if you had time. If
20:40you had time every morning at 9:00, you
20:42would sit down and you with your
20:44eyebrows would go through Slack. You
20:45would go through Honeycomb. You would
20:47ask people what's going on. You would
20:49look at GitHub. And then you would
20:50compile it. And then you would try to be
20:52very fun in Slack. Like it's it's just a
20:55description of what you would do. And
20:59>> Yep.
21:00>> it's it doesn't have to be that
21:01complicated. You can iterate so quickly
21:03on it. What I appreciate about this,
21:06you know, versus
21:07old era of more deterministic like
21:10workflow style builders is the updates
21:13are so easy to make. Just change the
21:15natural language, redo the order, change
21:18the trigger, give it more
21:20access to data, and then it's it's ready
21:23ready to go.
21:24>> Yeah, and you know, I think the other
21:26thing I I've gotten hung up on when
21:29trying to think about like these
21:30automations and I I've seen others do is
21:32that like
21:33you get this like
21:35you start giga braining it and you're
21:37like, well, how am I going to save like
21:395 hours of work a day? And I what you
21:42just said made me realize too that like
21:46the tedium that this removes for me is
21:49not like
21:50world changing.
21:52But it's like 20 minutes a day.
21:56And that's like 20 minutes I can spend
21:57doing other stuff. And that like it's
21:59not even just about like saving that 20
22:01minutes, but it's like
22:03protecting my brain from like having to
22:05context shift about all this stuff and
22:07like ingest it. And instead yeah, it's
22:09just I know that the information will be
22:11there when I'm ready to like read it and
22:13I'm ready to like shift gears to this
22:14project. Um rather than yeah. I I hate
22:18doing the like read this update, copy
22:21all this information, like put it into a
22:23like an update board. Like it's soul
22:25sucking.
22:25>> I mean, you and I have been doing this
22:27for a while. That's like that was I feel
22:30like 70% of my job. At some point 70% of
22:33my job was just like what's going on?
22:36>> Mhm.
22:37>> And how do I massage it
22:39into a format appropriate for the
22:41audience at hand? And it's always the
22:43same information. It's just like, what's
22:46the executive version of it? And what's
22:48the team version of it? And what's the
22:49full team version of it? And like
22:52I don't know. Like my shoulders dropped
22:53out of my ears when I realized we don't
22:56have like we just don't have to do it
22:57anymore. And the other thing that I
22:58think people maybe underappreciate about
23:01AI and this like just-in-time delivery
23:03is
23:04your AI your agent is never going to
23:06complain when you ask it to do this 5
23:08minutes before the meeting starts.
23:10>> I know. It's so It's so great.
23:13>> It's so great. It's just like when you
23:14have it, drop it, get it done and out of
23:18your brain. I think is Again, I go back
23:21to like burnout and enjoying your work
23:24and reducing toil and it just feels like
23:27a more relaxed way to work.
23:30>> Yeah.
23:31>> Yeah, we're we're happy.
23:33>> It's so funny cuz it is more relaxing
23:35and it's more fun and I feel like I'm
23:37getting more done.
23:38It's weird to have like this like
23:40win-win-win.
23:41>> Yeah. You know, they they do the
23:43triangle and they're like pick two and
23:44you're like, "No, I want to pick all
23:45three."
23:45>> Yeah, and I'm like give me the whole
23:47triangle.
23:47>> Give me the whole triangle. That Stamp
23:50it for the YouTube thumbnail. Give me
23:51the whole triangle. All right.
23:53>> This episode is brought to you by Orkes,
23:56the company behind open source
23:57Conductor, which powers complex
24:00workflows and process orchestration for
24:02modern enterprise apps and agentic
24:03workflows. Legacy business process
24:05automation tools are breaking down.
24:08Siloed low-code platforms, outdated
24:10process management systems, and
24:11disconnected API management tools
24:13weren't built for today's AI-powered
24:15world. Orkes changes that. With Orkes
24:18Conductor, you get a modern
24:20orchestration layer that scales with
24:21high reliability and brings humans, AI,
24:24and systems together in real time. It's
24:26not just about tasks. It's about
24:28orchestrating everything. APIs,
24:31microservices, data pipelines,
24:33human-in-the-loop actions, and even
24:34autonomous agents. So, build, test, and
24:37debug complex workflows with ease, all
24:40while maintaining enterprise-grade
24:41security, compliance, and observability.
24:44Orkes, orchestrate the future of work.
24:47Learn more and start building at
24:49orkes.io.
24:51>> Let's talk about So, we're talking about
24:53how meetings happen. Love this. You do
24:55write code, though.
24:57>> I do write code.
24:58>> Sometimes sometimes with your fingers
24:59and sometimes with your my favorite
25:02harness of the moment. Um so let's go to
25:04how you get
25:06coded.
25:07>> Yeah, I I want to show the little bit of
25:10a new workflow that we have going on in
25:13Notion. Um
25:15I honestly don't think this is
25:17necessarily a big feature that we're
25:18going to ship or some we might ship some
25:21version of this feature. So this is all
25:22basically internal only at this point.
25:25But prior to this, the way one I love
25:29Codex. I've been a Codex stand for like
25:33I don't know, 6-7 months now. And we
25:37started we started building this like
25:38Codex integration into Notion. Prior to
25:41this, it was like I mean obviously
25:44you're using the CLI to like write your
25:46prompt and then they created the Codex
25:47app which is which is nice, but I'm
25:49still like writing my prompt in this
25:51thing.
25:52So I started actually writing prompts in
25:54Notion pages where I can be a little bit
25:56more like free form and structured and
25:59you know I'm in like the CLI I don't
26:01have to worry about like hitting enter
26:03and then like oh like I sent my
26:04prompt. I can actually like write a
26:07document in Notion.
26:10But then of course I'm like cop or like
26:12highlighting all of the text copying it
26:14going into my terminal hitting paste and
26:16like letting it go. It's like fine.
26:18Um there's like MCPs and other stuff
26:20that I could be using but I'm I'm a very
26:22simple person. It's like too much too
26:24much stuff.
26:25Um so we built this thing that
26:29we're kind of like calling it
26:31I think we're calling it both software
26:32factory but I like it's internal project
26:34name is Boxy
26:36cuz it's like all these little VMs that
26:38we install Codex and Cloud Code on. It's
26:41our little boxes.
26:42Where now we can actually invoke them
26:44from
26:46like tasks within Notion. And so
26:49literally happened this morning a friend
26:51of mine,
26:53um, who's a Notion fan, texted me. He's
26:55like, "Hey, I like the tab block that
26:57you built, um,
26:59which is this thing. He's like, "But I
27:01really wish I could like, you know, I
27:02can like copy." I'll show you now. If I
27:05click on the dot dot dot thing next to a
27:07block, there's like copy link to block.
27:10He's like, "Man, I really wish I could
27:12just like copy link to a tab and then
27:14like send it to somebody."
27:16I was like, "Oh, yeah, that That sounds
27:18really easy." So, I opened up this task
27:21and I just took some notes and I dropped
27:23in this screenshot showing them, uh,
27:26where it could live. You know, this is
27:28like on the the tab block. If I right
27:30click on it, you get this little flyover
27:32menu.
27:33And I just described the task. I was
27:35like, "Yeah, let's put a copy link
27:37button here." I also noticed that
27:39hovering over the delete button didn't
27:41change to red. So, I was like, "Yeah, we
27:42should fix that." And I was like, "I
27:44know one edge case is like
27:46you're going to have to like if you land
27:48on a URL where where like linking to
27:52this block
27:53and this tab, like it's got to like
27:55switch to the tab on like fresh
27:57refreshes. So, I like kind of outlined
27:59all the the cases." But, you can see
28:01here, this is one, two, three
28:04paragraphs, four like four sentences and
28:06a screenshot. It was like, "Not a Not a
28:08lot."
28:09And then, uh, this new thing that we
28:11built, I can actually mention Codex from
28:14within our comments.
28:16And this triggers are like boxy. Uh,
28:18this is all our like internal dev
28:20tooling stuff.
28:21And then, it got to work. And
28:24I was looking at the timestamps earlier
28:26and I think 10:40, 10:51
28:29started the implementation. And then,
28:33another 10 minutes later, it replies
28:35with a pull request link and a preview
28:37URL cuz we like we do the, um, like
28:40preview environment stuff.
28:42And
28:43uh, it had like built the entire thing.
28:45And if I switch actually over to
28:49here's the pull request.
28:50It built like the entire thing. And it
28:52was like, here's how I tested it. And
28:54then this this was actually the coolest
28:56part to me is like it actually uploaded
28:57screenshots of it doing its own like UI
28:59verification. And there was like a CI
29:03failure in it. So I was like, hey.
29:06Replied down here like
29:08Oh, yeah. This part of this code change
29:10I was like, I don't know what is going
29:12on here. This doesn't make sense.
29:14And there's some type check things. And
29:16it was like, cool. Here's why we did
29:18this change. And I fixed your types.
29:22And then there was a merge conflict.
29:24>> Let let's also talk about just all world
29:26do world.
29:29The emotions around code review.
29:31>> Uh-huh.
29:33>> Like I love that you're like, I don't
29:34get this. Not I'm going to try to get
29:37it. Or I've done my best to investigate
29:39and I'm pretty sure this is wrong. Like
29:41literally it just I don't get it.
29:43I don't get it.
29:43>> Yeah.
29:44>> Like I mean I can like send send that to
29:46somebody.
29:47>> Oh, yeah.
29:48>> it with teammates? They're like, no man.
29:50Look it up. It makes it So that's one of
29:52my one of my code review mechanisms as
29:55well. As I'm just like, I don't think
29:57this is right.
29:59>> I yeah. One one one uh line that I've
30:02been putting in my prompts lately is I'm
30:04like
30:05I literally don't know what I'm doing
30:06here.
30:07You need to explain this to me.
30:09Especially doing all this CI stuff. I'm
30:11like, I'm in over my head. And like you
30:13got to you got to like explain it like
30:15I'm a five-year-old. And sometimes it's
30:17like
30:18literally like caveman style. Like here
30:20are the things. And I'm like, I needed
30:22this.
30:23>> Well, what I appreciate about this and
30:24we'll talk about it in in a little bit
30:26later I think is we're not getting any
30:28of these Claude code warm fuzzies from
30:30Codex. I love it.
30:32But when it talks to me I'm like, I feel
30:34real dumb when you're talking to me
30:36Codex. Can you
30:38>> Like, okay. Okay, dumb guy. Here's
30:40what's going on.
30:42>> So I think that sort of like I don't
30:44understand what I'm doing, please
30:46explain it to me is I love it, but I
30:48think it is a a Codex specific
30:50experience.
30:53Claude Code shows up and is like, "Hey
30:55buddy, guess what I made for you."
30:59Okay, so stepping back before we go too
31:02deep on the personalities of all these
31:04coding models,
31:05you have built just at mention, and I
31:08think people miss this, but it's
31:09important for particularly our
31:11engineering team and engineering leader
31:13listeners to pay attention to, which is
31:15everybody that I know that's smart it
31:17has some background agent hooked up to a
31:20virtual machine that they kick off so
31:22that you're not spinning up your local
31:24environment and doing all this stuff on
31:26your machine. Um
31:28>> Yeah.
31:28>> And it just I think the velocity where
31:30we'll we'll talk about this, but the
31:31velocity plus your dev ex plus your CI I
31:34think is
31:35a huge piece of AI adoption in
31:38engineering. So, if you were not if you
31:40don't have a like VM strategy and
31:43background agent strategy in your large
31:44engineering org,
31:46time time to get one. Time to do it.
31:49>> And we'll just ask Codex to build it. Be
31:50like,
31:51I don't know what I'm talking about, but
31:53I think I think this is a Claire said
31:55this is a good idea. Please build. Make
31:56no Make no mistake.
31:58Um okay, last last one. We have a last
32:02use case you were going to you were
32:04going to walk us through.
32:06>> Yeah, so we recently rebuilt our entire
32:11agent harness. The quick TLDR is just
32:14like everybody else we reached like this
32:16point of tool and instruction fatigue
32:18where you had this bloated system
32:19prompts. We're like, "Okay, we need to
32:21dumb this down." And we borrowed the the
32:23kind of like skills and progressive
32:25disclosure concept from coding agents
32:27and brought that to Notion AI.
32:29And when we were doing this big rewrite,
32:31we were asking ourselves like,
32:33you know, it only been like 6 months
32:35since we shipped the last rewrite. And
32:39this time we were like, what would what
32:40would we do differently like seeing the
32:42state of the art with coding agents? And
32:44someone had this really great idea to
32:48like let's not start with code. Like
32:51let's just start with specs.
32:53And what we've ended up building
32:57is we have this like in our checked into
32:59our code base, you see this we have this
33:01we're looking at this is a notion repo,
33:03but we have this agent specs subfolder.
33:05And within this subfolder we have all of
33:08these markdown documents. This is one
33:10that that I worked on. We have this
33:12thing in our
33:13AI called ask mode where we basically
33:16ban all of the like mutating tools. So
33:19it can only just like read and answer
33:20questions. And so when I was building
33:23this, I didn't start with writing code,
33:25I didn't start with anything. I just
33:26started with an empty markdown document.
33:29And
33:30I mean I actually just opened up like
33:32whisper and just started yapping about
33:36how this feature should work. And then
33:38at the end of it, I gave that I gave the
33:41yap session to Codex and was like,
33:43here's our other like spec library,
33:46learn the format, take my information,
33:49write a spec.
33:51And then it spikes the first version. I
33:53did a couple revisions on it.
33:56And I ended up with this markdown
33:57document. Now the markdown document is
33:59like it's nice, but what we did with it
34:02next in my opinion is is I I kind of
34:05think that this is like the future of
34:06software engineering where I then opened
34:09up Codex again, pointed it at this spec
34:12file, and I said,
34:15build it.
34:16And it basically one shotted this
34:19because the entire spec file is so
34:21comprehensive with code pointers with
34:23down at the bottom we have verification.
34:27And it was like, here is how you're
34:29going to verify all of this stuff works.
34:31And we've even built our own CLI tools
34:34so that you can run Notion AI from the
34:36CLI and it could, once it's done seeing
34:39that all the tests pass, it can actually
34:40just spin up Notion AI
34:43itself, send it queries, send it
34:45questions, enable ask mode, disable ask
34:47mode, and then see the transcripts and
34:50like see what actually happens.
34:52And I think the first shot of this took
34:54a couple hours, but I came back to
34:56whatever, couple thousand lines, did
34:58some code review, played with it myself,
35:00and I was like, it's right. It's like
35:02done.
35:03And
35:05you know, since then, we've made I can
35:07the other beauty of this is like this is
35:09in version control.
35:10So, I can go to the past changes of the
35:13spec file and I can see how the the spec
35:16has evolved. And I it I could go look
35:19through all of the code changes, which
35:20also have their own like history, but
35:22this is now the sort of like source of
35:25truth for how this part of Notion AI
35:28works. And it's just in plain English
35:30that can then be verified and
35:32implemented by agents.
35:34>> Well, I think the other thing that
35:35people don't appreciate is, you know,
35:36taking this outside of the engineering
35:38flow is this plain English can be
35:41ingested by other parts of the business
35:43that need this information. So, let's
35:46say you need to then release this
35:48feature via some sort of marketing. This
35:51is actually like a pretty good asset
35:54that explains how it works, that can be
35:57translated into another another thing
35:59that in a way that like code itself
36:01>> Yeah.
36:02>> is still a little intractable. And so,
36:04this idea of spec-driven development,
36:06but what I like about what you said, I
36:07don't want people to miss is the way you
36:09make these updates is you update the
36:11spec and go go go look, make the make
36:14the update, change the date.
36:15>> Exactly.
36:16>> Um and so, like the spec is the source
36:17of truth, the spec as the the change
36:20log, I think is a really interesting
36:23model. And
36:25for people that are watching, it's very
36:28detailed. It's very technical. So, it's
36:29not like there is not code
36:32in the spec. There's just not all the
36:35code in in the spec. And I think that's
36:38uh a really kind of good hybrid model
36:41for experienced engineers to start to
36:44bridge into what would it look like to
36:46have an agent do more of your coding
36:49work while you still do architecture
36:51work, while you still do design work,
36:52while you still make sure that the thing
36:54is going to scale.
36:56>> Yeah, exactly. I
36:58I view our job as like engineers
37:01evolving into like systems thinkers and
37:05architects. And not And not even just
37:07necessarily writing like the spec and
37:10thinking about the behaviors, but most
37:12importantly is like the verification
37:14loop. Like, is it a Like, how should it
37:16verify correctness of this feature um or
37:19this change? And honestly, it's like if
37:22it can't or if like the verification's a
37:24little hazy, it's like that's the first
37:26thing you actually should be going and
37:27doing. It's like, do you have a tool to
37:30let the agent uh run itself? That's like
37:32one of the first things we did with this
37:33project was like, we we should actually
37:35build a CLI so that I can tell Codex
37:38like, send this prompt and like see what
37:39happens. And now that we have that, then
37:42we can take these specs and actually
37:43just like go deeper and deeper and
37:45deeper. And so, it's like we're still
37:47doing engineering but I'm not doing the
37:50like plumbing work of like wiring up
37:52this ask mode feature.
37:53>> Well, one thing is really funny is uh
37:56me, I've been in software engineering
37:57for 20 plus years. Like, we were writing
38:00these documents anyways. We were writing
38:03technical design documents, spec design
38:05documents anyway. And we were sitting in
38:07meetings with other engineers debating
38:10the merits of one implementation versus
38:12another.
38:13And then we still had to go write the
38:14code.
38:15>> Yep.
38:16>> It has like really It It hasn't added
38:18work to to go into this model. It's
38:20maybe like shifted the emphasis of where
38:23the human attention goes, but these were
38:25documents that at least I was in every
38:28org that I've ever been in writing
38:30before. And the other thing that I think
38:32has changed so much is those meet those
38:34docs
38:35then waited for review.
38:37And they waited for a meeting. And now,
38:41no more waiting for the meeting, no more
38:43waiting for review. Ship it, have a
38:45verification loop,
38:47debate it on the merits of it being live
38:49and working versus the theoretical
38:51merits of it sitting in a document
38:54waiting for everybody's calendar to open
38:55up for an you know, a live argument.
38:59>> Yep. Yep. Couldn't agree more.
39:01>> Let's let's do it.
39:02This has been my I I just you and I
39:05could talk all day about this. Just to
39:06recap for everybody cuz I know I got to
39:08get you out of here. Three use cases.
39:10One, never prep for a meeting again.
39:11Hook it up not only to your Slack, but
39:14to your meeting notes, your GitHub, your
39:15telemetry, and build the best stand-up
39:18meeting. So, no one has to stand there
39:20glassy-eyed being boring giving updates.
39:23Second use case, background agents, at
39:26mention from wherever you work, Notion
39:28being a great place to do that, kicking
39:30off virtual machines, getting PRs done,
39:32just saying yes when your friend texts
39:34you, can you ship this feature? And then
39:37the last one,
39:38putting all your specs in your repo,
39:40using them as a source of truth for a
39:43more autonomous coding agent like Codex.
39:46Let it cook for a couple hours, review
39:48the code, and then when you update,
39:50update the specs, don't update the code.
39:52Did I get it right?
39:53>> Nailed it.
39:54>> Uh okay, let's do a couple lightning
39:55round questions and I'll get out you get
39:56you out of here. You and I love Codex.
39:59Why Why do you love Codex? I'll tell you
40:00why I love Codex, but you go first.
40:02>> Um okay. I Well, so I first I first fell
40:06in love Oh my god, did I just say that?
40:09>> You did.
40:09>> I first I first fell in love with Codex
40:12so because
40:14when I was like evaluating both like
40:16Cloud Code and Codex. Um
40:18I found anytime Claude Code like
40:22filled up its context when you window,
40:23it would just kind of like lose the plot
40:25really quickly. And Codex, I don't I
40:28don't know exactly what it's doing, if
40:31it's the model, if it's the compaction,
40:33if it's like both, but it can it can
40:35grind for like hours. Um and
40:38with with the way that I work, both the
40:41systems and things that I work on and
40:43just like I like to be able to fire off
40:45a bunch of them like at the same time
40:48and then like go to a meeting or go do
40:50something else or like
40:52uh spend my time kind of like round
40:54robin managing like all of these agents.
40:56Like I I I don't necessarily want to I'm
40:59not the person that is like sitting with
41:00the browser open and like um some agent
41:02next to it and like iterate, look at the
41:04browser, iterate, look at the browser.
41:06I'm like
41:07the more the closer I can get to
41:09one-shotting solutions, the better uh
41:11cuz that frees me up to do other stuff.
41:13Um and so I found that like Codex was
41:16was pretty good about that. I also just
41:18feel it's like
41:19pretty simple. Like it's there's not a
41:23lot of bells and whistles. It's not like
41:25too fancy necessarily. Um I'm happy with
41:29the addition of like MCP and skills um
41:32and some like other stuff coming out. I
41:34also really love GPT-5 for um I think
41:38it's a great model. So I'm like all of
41:39those things together, it's just
41:42it really like matches my working style
41:45uh and type of work a lot.
41:47>> Yeah, I'll I'll tell you why I love
41:48Codex and I sent this to somebody. I
41:50said
41:51work trees everywhere.
41:53Port 3 3000 through 3009 spoken for.
41:56Like we're just we're just going across
41:59across the board. Um I do think it's
42:01good at long running tasks. I like its
42:03concept of projects cuz I run a lot of
42:05different projects. I think this is like
42:07a very helpful
42:08mental model. And then it's great at
42:11code review. Honestly,
42:13it's just like a really good code
42:15reviewer. It's a really good security
42:17reviewer. Um tireless, uncomplaining, um
42:21with the most tedious of things. And so
42:24I I find it's one of my one of my daily
42:27drivers. I just I really really like it.
42:29It's good. Okay, second question because
42:31you and I also agree on this.
42:33Give people the reason. This isn't even
42:35a question. This is a a demand for you
42:37to share my point of view, which is why
42:40bang on developer experience and CI
42:43speed right now.
42:45>> A couple a couple things. To me, CI was
42:47like
42:48super important prior to agents, um
42:51because in my opinion
42:53I mean, it's not even an opinion. It's
42:54like fact. That like the faster your CI
42:57completes, the faster you get signal,
43:00um the faster your like the more
43:03comfortable your engineers will feel
43:05with like making changes and doing
43:07things because they know like I can make
43:09a change and get it pushed into dev or
43:12into production really quick because I
43:13know my CI is fast. If it's slower, then
43:15you're building up these like monster
43:17changes and you're going to be even
43:19slower about like judiciously like
43:22reviewing every little like tiny thing
43:25when I I'm like a
43:26learn-through-like-doing
43:28sort of person. And if I can close the
43:31iteration loop like even tighter, then
43:34I'm going to be putting out a change.
43:36People are going to be using it. I will
43:37take that feedback. I will make another
43:39change. And I won't have to worry about
43:41um
43:42you know, it taking another day before
43:43like the deploy train is ready to go. I
43:46just want to like I want to crank really
43:47really fast. And that was all pre-agent.
43:49And now that we're like in the agent
43:51land it's it's that but like on steroids
43:55because agents don't get tired. They can
43:58work on a VM. They can work like while
44:01I'm sleeping. And you know, if I'm take
44:03if I've got a CI loop that takes an hour
44:07to run,
44:08that your agent is going to sit there
44:10and spin for like an hour waiting for
44:11results to like do something. If it
44:13takes 3 minutes to run, like holy crap,
44:15how much more stuff are you you as a
44:18human and then especially as your like
44:19little swarm of agents going to be able
44:21to get done? Like so so much more. Um
44:24and so I
44:25super important.
44:26>> I I agree and we just had Steve from
44:28Stripe on and they're doing like 1,300
44:32agent PRs a week. And you cannot do
44:35>> Yeah.
44:35>> Yeah. Yeah, you like cannot do that if
44:37your CI is slow. It's just you might as
44:40well be throwing all those PRs in the
44:41trash. And so I do think there is just a
44:45true mathematical limit
44:47>> Yep.
44:47>> on your capacity to ship code to
44:49production
44:50that is a reflection of how fast your CI
44:53pipeline is. And so if you are again,
44:55engineering leaders, if you are not
44:56spending time on making that fast and
44:59you are tolerating something slow there,
45:01you're not going to get the the benefits
45:03of AI that you could. And honestly, like
45:04good for AI, good for humans.
45:06No engineer wants to sit waiting for
45:09their stuff to hit hit production. It's
45:11miserable on the engineer side, lots of
45:13downsides. So allocate time
45:17to your pipeline, please.
45:18>> Yep. Yep.
45:19>> Ryan Ryan and Claire say so. All right,
45:21last question.
45:23When AI is not listening,
45:27it's writing bad specs,
45:29it's not being funny in Slack, what is
45:31your prompting strategy? Do you yell?
45:35>> Yeah. I can I can
45:37>> I can be a little bit of a diva
45:39sometimes if it like really goes off the
45:41rails. The
45:43the the other the other prompting
45:45strategy that has like saved my ass
45:47working on the CI stuff lately is
45:50because like even the best models I feel
45:52like can sometimes be a little
45:53sycophantic.
45:54I'll be like
45:56I will just like be like you're wrong.
45:58Like you need to defend your argument.
46:01Because I like I want it to defend it in
46:03the way that I like I want but I'm like
46:05I just need
46:07I just need to see the evidence that if
46:09I push counter to what it has done that
46:11it can like back up with like good
46:14pointed reasons rather than just be like
46:16are you know are you sure this like
46:18change looks okay? It's like oh boy it's
46:19like totally fine. I'm like no no no I
46:22need like the cited hard argument
46:25against it cuz like in a lot of times
46:28like with the CI stuff I'm like I
46:30don't know what I'm doing. I know
46:32generally what I'm doing but like the
46:34specifics are a lot more nuanced and I
46:37like I need to get this right. Um and
46:39that's been super helpful and then when
46:40it goes off the rails I'm like
46:42I can be such a diva too.
46:46>> Escape all caps no.
46:49That's what I do.
46:51Interrupt and no. Steer the
46:52conversation. Well arrow button has
46:55never gotten so much work. Ryan this was
46:57so much fun. Where can we find you and
47:00how can we be helpful?
47:02>> Uh you can find me on X. Um I'm out
47:06there shilling for Notion right now a
47:08lot. Um but that's that's basically
47:11where I spend my time. Yeah I my DMs are
47:13open so if you have Notion problems I
47:15like to try and fix them. I
47:17I'll send it to Boxy and we'll get it
47:19done.
47:20>> Perfect love it. Well thanks for joining
47:21the podcast.
47:22>> Thanks for having me Claire.
47:25>> Thanks so much for watching. If you
47:27enjoyed the show please like and
47:29subscribe here on YouTube or even better
47:31leave us a comment with your thoughts.
47:33You can also find this podcast on Apple
47:35Podcasts, Spotify, or your favorite
47:37podcast app. Please consider leaving us
47:40a rating and review which will help
47:42others find the show. You can see all
47:44our episodes and learn more about the
47:46show at howiaipod.com.
47:50See you next time.

About this transcript

This page contains the full transcript of Spec-driven development: the AI engineering workflow at Notion | Ryan Nystrom by How I AI, generated from the public captions YouTube serves with the video. The transcript has 8,848 words across 1,337 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.