Mann - My Codex Workflow & Swe best practices | OpenAI Build Week Community Meetup - San Francisco — Transcript

Full transcript

1. 0:00Like all the other people who spoke
2. 0:01before me, I'm not actually associated
3. 0:02with the startup yet, but
4. 0:05I was actually participating in the
5. 0:08OpenAI build week. I don't know if
6. 0:11everybody else in this room is also
7. 0:13doing that, but also show of hands if
8. 0:15you've used DevPost before. Like this
9. 0:17way. Okay. Quite a few of you.
10. 0:20I'll So what I'll do is I'll walk
11. 0:22through the project I built using Codex
12. 0:24and I'll walk through my workflow
13. 0:27with Codex and just some software
14. 0:29engineering best practices.
15. 0:30My project, I call it HackStack. It's
16. 0:33supposed to be a replacement for DevPost
17. 0:35because
18. 0:36honestly DevPost looks like something
19. 0:37built in the 2010s
20. 0:39and it looks pretty outdated. Doesn't
21. 0:42have a lot of cool AI features that it
22. 0:44probably should have. So um
23. 0:47So yeah,
24. 0:48this is my alternative to DevPost. It
25. 0:51obviously has a much nicer UI, but
26. 0:53nobody cares about that. What does it
27. 0:54do?
28. 0:56It generates analytics for the
29. 0:58hackathon. Basically have an AI agent
30. 1:00that ingests all the submissions from
31. 1:03all the users. I mean all the hackers
32. 1:05and all the teams. It looks at all their
33. 1:07Git repos, ingest that
34. 1:09and produces
35. 1:11hackathon-wide insights. You know, you
36. 1:13can see what kind of
37. 1:15tools people use, what kind of coding
38. 1:16agents people use
39. 1:18and and just like code like other stuff
40. 1:20about their code bases. You can see
41. 1:22stuff about the people who participated,
42. 1:24but the most interesting thing that I
43. 1:26think is if you go to a specific
44. 1:28project, right? The Yeah, maybe not this
45. 1:31one.
46. 1:32Yeah, what what the what agent does is
47. 1:35it looks at it generates a list of
48. 1:38of
49. 1:39features that the project claims to have
50. 1:41and then it it it hosts
51. 1:45It hosts an agent in a sandbox that
52. 1:47clones that code base and starts to
53. 1:49verify if implementations for those
54. 1:51claimed features actually exist. There I
55. 1:53just making
56. 1:54judges' life easier. So that's that's
57. 1:56that's the project I built. Um
58. 2:00So for the sake of the demo, as I think
59. 2:01George mentioned, Grill me a lot
60. 2:04uh and walking through my workflow of
61. 2:06building with Codex, I thought what I
62. 2:07would do is I would build a feature
63. 2:09using Codex on the Grill me scale. Uh
64. 2:11and obviously instead of doing it live,
65. 2:12I I did it before uh for a couple of
66. 2:14reasons. It's it's way faster and you
67. 2:16don't have to see the the thinking
68. 2:18spinner. Um also because
69. 2:20uh
70. 2:21it's it's scary doing a live demo with
71. 2:23LLMs cuz they're so unpredictable
72. 2:24sometimes. Um
73. 2:27So I I I asked it to add a really simple
74. 2:29feature for judges when they look at a
75. 2:30particular hackathon. They or look at a
76. 2:32particular project at a hackathon, they
77. 2:34want to maybe write some notes or their
78. 2:35thoughts about um
79. 2:38what they felt like or what stood out to
80. 2:39them or something like that. So I
81. 2:41described my feature with the Grill me
82. 2:42scale. And the reason I use Grill me, I
83. 2:45personally like it more than plan mode
84. 2:47simply because it's it's the loop of it
85. 2:49asking you questions and generating and
86. 2:51gathering context of what you're trying
87. 2:53to build is more intricate and I in my
88. 2:56opinion more in-depth uh compared to the
89. 2:59the plan mode where it just asks you a
90. 3:00few questions initially and then it's
91. 3:01just doing its own thing. But here it
92. 3:03sometimes especially if your feature is
93. 3:05complicated enough. This is a pretty
94. 3:06simple feature for the sake of the demo.
95. 3:08But if your feature is um
96. 3:10complicated enough, it could ask you a
97. 3:12way over 20 30 questions trying to be
98. 3:14exactly sure about um what you're trying
99. 3:17to build.
100. 3:18And and then it asks you specific
101. 3:20questions about Oh, is your
102. 3:22uh is the notes for every project
103. 3:26only for anonymous users or signed in
104. 3:28users and things like that. And as you
105. 3:30can see, my responses are usually pretty
106. 3:32brief and usually saying yes, yes. You
107. 3:35know, and because most of the time it's
108. 3:36done a pretty good job of
109. 3:39um
110. 3:40trying to understand what I'm asking it
111. 3:41to build. Also because this is a simpler
112. 3:44uh feature I was trying to add. Uh every
113. 3:46now and then you do have to uh correct
114. 3:48it and and and be like, you know, don't
115. 3:51do that. So I I personally really like
116. 3:53the grill me skill. Um It comes from
117. 3:56this very popular repository called um
118. 4:00Well, it's called skills, but it's by
119. 4:01this guy called
120. 4:03Matt Pocock and you can see 180,000
121. 4:06stars. I highly recommend the other
122. 4:08skills in this as well. But, I
123. 4:10personally only use um uh the grill me
124. 4:13and there's another skill called code
125. 4:15review or something like that, which is
126. 4:16pretty cool, too. Uh another thing I do
127. 4:18in my workflow is the agent.md file and
128. 4:21especially if you're building a
129. 4:23hackathon project and you have a team of
130. 4:24like five or six people and you're
131. 4:26building this high pressure like two
132. 4:27days, you know, just just white coding
133. 4:30re- like basically you're not reviewing
134. 4:31any like any of the code, right? You're
135. 4:33just You're just asking the agents to
136. 4:35build the features and then you just
137. 4:37test it out manually to see if those
138. 4:39features were implemented. Uh the
139. 4:41agent.md is
140. 4:43very useful. I don't usually recommend
141. 4:46uh fully white coding your agent.md and
142. 4:49I do I do think you should manually
143. 4:51write it.
144. 4:52Uh especially if you have a a database
145. 4:55in your
146. 4:56in in in your application and especially
147. 4:59especially if you have a production
148. 5:00database, I think you should be very um
149. 5:04you should be very careful then if you
150. 5:05have live user data and I actually I
151. 5:08severely recommend against using um MCV
152. 5:10servers for your uh live live database
153. 5:14and connecting your agent to that simply
154. 5:15because I think that is uh that is way
155. 5:17too risky. Uh so, what I have done is my
156. 5:20workflow for working with um
157. 5:23databases is something like this. I
158. 5:24think this is just good software
159. 5:25engineering practices where I have a
160. 5:27local local Superbase instance, which I
161. 5:30actually do give full control to the
162. 5:32agent. You know, they can do whatever it
163. 5:33wants to do to the local database, but
164. 5:35for a staging and production databases,
165. 5:37uh I I'm given I have given uh not not
166. 5:40just this, but all the stacks you see
167. 5:42before all these bullet points are also
168. 5:44alluding to the fact that do not ever
169. 5:46touch the staging and production
170. 5:48databases. And only when I am happy with
171. 5:50the migrations it wants to apply to the
172. 5:52local database, do I manually push code
173. 5:55to the staging and production DBs.
174. 5:58Um
175. 5:59So yeah, that's my
176. 6:01that's my workflow with that. Uh and as
177. 6:03you can see, the feature that I added or
178. 6:05wanted to add to the uh
179. 6:08to my to my hackathon project was pretty
180. 6:09simple and it did a really good job
181. 6:11uh adding it.
182. 6:13Um
183. 6:15And
184. 6:16yeah. Uh
185. 6:17I think that's about it.
186. 6:19Woo!
187. 6:21>> [applause]