100% Open Image Model Lets You Place Text With JSON — Transcript

Full transcript

1. 0:00Okay, so all three of these images came
2. 0:02out of a model that's just 5 billion
3. 0:04parameters. It's open weight and most
4. 0:06importantly, it's got an Apache 2.0
5. 0:09license, meaning that basically you can
6. 0:11do whatever you want with it. The
7. 0:13model's called Moss image and in this
8. 0:14video I'm going to go through some
9. 0:15examples of what it can do, what it
10. 0:17can't do, what its strengths are, and
11. 0:19how you can use this very small model to
12. 0:22produce really fantastic images out. So
13. 0:25let's take a look at what you can
14. 0:26actually get from a 5B image model
15. 0:28nowadays. All right, so Moss image comes
16. 0:32from Open Moss. This is the team that
17. 0:34put out Moss TTS and the Mova video
18. 0:36model. And this is their first image
19. 0:38model. It came out last week. And
20. 0:40basically under the hood, this is a 5
21. 0:42billion parameter diffusion transformer
22. 0:45that they trained from scratch. Now to
23. 0:47get it to work, they're basically using
24. 0:48the text encoder, which is Qwen 3.59B.
25. 0:52So you've actually got a 9 billion
26. 0:53parameter language model reading your
27. 0:55prompt. And that prompt can be up to
28. 0:56around 4,000 tokens. Older image models
29. 1:00used much smaller text encoders. And
30. 1:02that's a big part of why they struggled
31. 1:04with long context and complicated
32. 1:06prompts. Moss basically has two
33. 1:08checkpoints here. There's a base model,
34. 1:10which is really for fine-tuning and
35. 1:12training LoRAs. And then there's an SFT
36. 1:14model, which you can actually use to
37. 1:16generate with. But like I mentioned in
38. 1:17the intro, the big thing here is that
39. 1:19this is all Apache 2, both the weights
40. 1:23and the code. If you look at some of the
41. 1:24other strong image models that I've
42. 1:25looked at in the past, things like Qwen
43. 1:27image, et cetera, while the weights are
44. 1:29open, the licenses on those are not.
45. 1:32They're often under a non-commercial
46. 1:34research license. Things like Flux 3
47. 1:37client is gated under the Flux community
48. 1:39license. And often that just limits what
49. 1:42you can actually do with this. This one,
50. 1:44because it's Apache 2, basically you can
51. 1:46do what you want. You can make money out
52. 1:48of it. You can put it in your own
53. 1:49products. And you can pretty much make
54. 1:51any kind of images that you would want
55. 1:52to do. Amazingly, both safe for work
56. 1:55images and NSFW images as well. So out
57. 1:58of the box, the model is actually up on
58. 1:59Hugging Face already. It runs natively
59. 2:02in ComfyUI, which I think most people
60. 2:04are going to use. You can also use it in
61. 2:06SG Lang and VLM Omni, which is basically
62. 2:10how I'm running it here today. All
63. 2:11right, so let's start with just general
64. 2:13images. These are all on the SFT model
65. 2:15at default settings. We've got 50 steps,
66. 2:18and we've got a CFG, which is a
67. 2:19classifier-free guidance of three in
68. 2:22here. So, you'll notice that
69. 2:23photorealistic portraits are really
70. 2:25solid here. The skin and the details
71. 2:27really hold up well. Product shots come
72. 2:30out really nice and clean, and the
73. 2:31model's perfectly good at generating out
74. 2:33bouquet effects for the background, etc.
75. 2:36And I'll talk about that as we go
76. 2:37through. I also find that the lighting
77. 2:39effects and lighting sources here, the
78. 2:41model seems to handle those very well,
79. 2:44both for realistic images, but also if
80. 2:47you want to make something like a
81. 2:48watercolor or some kind of artistic
82. 2:51interpretation as well.
83. 2:53Styles work, too. If you want to do
84. 2:55something like anime or some particular
85. 2:57kind of artist, etc., the model seems to
86. 2:59be really good at being able to do that.
87. 3:01Also, being able to set it to actually
88. 3:03do different kinds of aspect ratios
89. 3:06means that it's not just locking you
90. 3:07into square images. You can really go
91. 3:09into wide banners, sort of portrait
92. 3:11phone wallpapers, and can do that with
93. 3:14anything from about a quarter of a
94. 3:15megapixel up to 4 megapixels. With what
95. 3:18I would say is 1 to 2 megapixels is
96. 3:20being the sweet spot there. So, the
97. 3:22model can definitely make decent images
98. 3:24for this model size. But I think the
99. 3:26real reason to look at this model is
100. 3:28text. So, we start with something
101. 3:30simple. You'll see that just a single
102. 3:32title, well, that's no problem. Three
103. 3:35lines, title, date, venue for a poster
104. 3:38or something, can handle that kind of
105. 3:40thing really well. It also can generate
106. 3:42the text both in English and Chinese for
107. 3:45the same scene. And if you wanted to
108. 3:46have something stylistic, like having
109. 3:49something written on a chalkboard menu
110. 3:50and stuff like that, it has no problems
111. 3:52with doing things like that. Even for
112. 3:54things like infographics, where you
113. 3:56basically want to call out different
114. 3:58steps, that kind of thing, this model's
115. 4:00doing really well at this. And you can
116. 4:02see if we look at these scores here, MOS
117. 4:04image is scoring way above the other
118. 4:07open models for this. In fact, it's only
119. 4:09just behind fraction of Nano Banana and
120. 4:13only a few percentage behind GPT image
121. 4:15two. So, clearly at the lab's own long
122. 4:17text benchmark, it's really doing
123. 4:19fantastically well for one of the open
124. 4:21models. I guess though, do keep in mind
125. 4:23that that's their own benchmark. And so
126. 4:26far, I haven't really found any
127. 4:27third-party evals there. Now, that
128. 4:29doesn't mean that it's perfect at text
129. 4:31all the time. There are times that it's
130. 4:33going to break if you push the limit to
131. 4:35be too many words on a page, something
132. 4:37like that. It can start to fall apart.
133. 4:39And perhaps when compared to bigger
134. 4:41models like the Korea 2 model, etc., you
135. 4:43are going to find that perhaps this
136. 4:44model is not quite as good. That said,
137. 4:47it's got some really nice features just
138. 4:49like those other systems have. So, the
139. 4:51whole idea of sort of JSON layout. So,
140. 4:54along with a regular prompt, you can
141. 4:56give MOS image a layout where you
142. 4:58describe the whole scene and then you
143. 5:00list out elements, text or objects, each
144. 5:03with a bounding box. And they basically
145. 5:05train the model to respect this sort of
146. 5:08layout JSON and the captions that go
147. 5:10along with it. So, it seems to me that
148. 5:12this is not something that's just bolted
149. 5:14on afterwards. You've basically got the
150. 5:16opportunity to have it where you can
151. 5:19describe what it is that you want and
152. 5:22then use JSON sort of bounding boxes to
153. 5:25show where you actually want it on the
154. 5:27actual image. This can be really cool if
155. 5:30you've been trying to do things with
156. 5:31in-painting in the past and you just
157. 5:33haven't been able to get what you wanted
158. 5:35where you wanted it, that kind of thing.
159. 5:37And it's also kind of cool trick of how
160. 5:40you can generate something and then look
161. 5:42at the output and then decide, "No,
162. 5:43actually I wanted that on a different
163. 5:45part of the image." Now, the cool thing
164. 5:47here, too, is that don't forget you can
165. 5:48get an LLM to actually write that JSON
166. 5:51out. So, you can go from a very
167. 5:53descriptive brief to have an LLM
168. 5:55actually write out the prompt with the
169. 5:58JSON to then have MOS image actually
170. 6:01generate this in a pipeline. And they've
171. 6:04got their own layout prompt enhancer,
172. 6:06which you can play around with and does
173. 6:07exactly that. All right, so what does
174. 6:09the model not do? Well, first off,
175. 6:12there's no editing here. You cannot put
176. 6:15in image to image here. You can't do any
177. 6:17sort of editing of images. It's purely
178. 6:19text to image. You can do sort of stuff
179. 6:22like the standard in painting in
180. 6:24ComfyUI, but that's pretty much about
181. 6:26it. You're not going to be able to do
182. 6:28something where you take a real-world
183. 6:29photo and then edit it, like you can
184. 6:31with Nano Banana and Quant Image Edit,
185. 6:33etc. And who knows, maybe that's going
186. 6:35to be the next model that they actually
187. 6:36release, which would be pretty cool to
188. 6:38see, especially if that's also Apache 2.
189. 6:40Now, one of the cool things is because
190. 6:42they've given you the base model, we can
191. 6:43actually do sort of LoRAs and fine-tunes
192. 6:46ourselves to actually get very specific
193. 6:49styles or get certain people in there,
194. 6:52that kind of thing. Now, at the time I'm
195. 6:54recording this, I'm not sure all the
196. 6:55code examples are out for that, but I
197. 6:58imagine that's something that the
198. 6:59community is going to be able to work
199. 7:00out themselves as well. Out of the box
200. 7:02itself, it's pretty slow at these stock
201. 7:04settings. You're about 50 steps there,
202. 7:07and I imagine if you're doing all of
203. 7:09this with a smaller, older GPU, it is
204. 7:11going to take time for you to generate
205. 7:13that out. Now, that said, I think the
206. 7:15community has already released some
207. 7:17distillations of this and modifications
208. 7:20for versions that you can run in
209. 7:21ComfyUI. I wouldn't be surprised that we
210. 7:24actually see a sort of turbo version of
211. 7:26this, and I think the community will put
212. 7:28out a lot of different versions of this.
213. 7:30In just under a week, the community has
214. 7:31already put out like an eight-step and
215. 7:33four-step LoRAs, quantized versions, and
216. 7:36even a full fine-tune on the base model.
217. 7:39Not to mention that the Most team
218. 7:40themselves have said that they're
219. 7:42actively working on a few-step
220. 7:43distillation. So, that itself might even
221. 7:46lead to an official turbo version on the
222. 7:47way. So, let's take a look at some of
223. 7:49the images you can actually do with this
224. 7:51in this little demo that I've created.
225. 7:53Okay, so if we come into this sort of
226. 7:55gallery app that I built here, just go
227. 7:58through and show you a number of
228. 8:00different things in here. First off,
229. 8:02you've got lots of things that you can
230. 8:03generate. Now, obviously, you can
231. 8:04generate with text. The big issue is
232. 8:07that sometimes the text is going to
233. 8:09work, sometimes not. The way to actually
234. 8:11make it work is to use layout control.
235. 8:15So, you see that something like this, I
236. 8:17basically had in there as just a
237. 8:19straight-up generation where I'm not
238. 8:20guiding it in any way, and it will start
239. 8:23to get different parts of the text wrong
240. 8:26over time. Whereas, if I come in and do
241. 8:28something like this, you can see here
242. 8:31I've basically got different layout
243. 8:33boxes. So, I can actually draw a box and
244. 8:37then fill it in. Right? So, this is
245. 8:40using the JSON generation in here. Once
246. 8:43I I've put it in there, I can then
247. 8:45basically tell it what do I want to
248. 8:47appear in that box. We've got, you know,
249. 8:50a whole bunch of different things. If
250. 8:51you want to Let me just turn on the
251. 8:53turbo version. So, this is going to be a
252. 8:54bit quicker. We can go even deeper in
253. 8:57here. So, you can see here we've
254. 8:58basically got the actual layouts there.
255. 9:02If we want to change, you know, the
256. 9:03couch, we could come in and just change
257. 9:05the color in here. We can change it
258. 9:08here. I can then click generate, and
259. 9:12you'll see now we'll get a green couch
260. 9:13instead of a red couch there. We can
261. 9:16also do things like we can change, you
262. 9:18know, things on the the poster at the
263. 9:20back there. If I generate that now, we
264. 9:23should see we're going to get Ready
265. 9:25Player One instead of just Player One
266. 9:28there. And so, this is sort of the key
267. 9:30way if you want to actually play with
268. 9:32text, you want to actually control where
269. 9:34it's going to appear, and that also
270. 9:37seems to make the text a lot more
271. 9:38accurate in here. Not just text though,
272. 9:40we can also change, you know, the cat to
273. 9:42a tiger. Let's see how that's going to
274. 9:44come out. And you can see now we've got
275. 9:47our tiger. Now, we will find sometimes,
276. 9:49you know, things like that changed, you
277. 9:50know, him to putting the hoodie on. And
278. 9:52so, if we had something really specific
279. 9:54that we wanted, we would basically go
280. 9:56through and tell it, "Okay, change these
281. 9:59different things." We would go through
282. 10:02and actually tell it to not put the
283. 10:04hoodie on his head, keep it at the back,
284. 10:06or something like that. We can also do
285. 10:08really nice sort of styles and effects
286. 10:11in here. So, these are just straight out
287. 10:13of the model, no sort of Laura
288. 10:15pre-trains or anything like that, and it
289. 10:17does a really nice job of those. It can
290. 10:19also do things like in-painting, so you
291. 10:21can see here we've got the same picture,
292. 10:23we're just changing what's going to
293. 10:25appear there. And you can see in this
294. 10:27case the books got cut off, but for the
295. 10:30cat it works well, for the dog perhaps
296. 10:32not as much. So, you do want to
297. 10:33experiment with it. Back to the styles
298. 10:36here, it really is able to capture some
299. 10:38really nice styles of different kinds of
300. 10:42pictures and stuff like that. You can
301. 10:44then basically use Laura's with it. So,
302. 10:46one of the Laura's that's already been
303. 10:48made is this eight-step Laura. So, this
304. 10:51just speeds up generation. So,
305. 10:54generation without it for me is around
306. 10:57about 30 seconds, but if I use this,
307. 11:01let's just come in and remix this. If I
308. 11:03use this and I actually put on this
309. 11:05Laura here, I can actually come in here
310. 11:08and change that. So, we had him wearing
311. 11:11a yellow top now, we change it to
312. 11:13basically an orange one. And you can see
313. 11:15that that's generating a lot quicker now
314. 11:18than 30 seconds. We've got the time down
315. 11:20much quicker. So, you'll see in regards
316. 11:23to sort of famous people, certain people
317. 11:26it will know very well, for example,
318. 11:28Brad Pitt, etc. it knows really well.
319. 11:31But if I ask it to make a picture of Sam
320. 11:33Altman, it really doesn't know who he
321. 11:35is, right? For whatever reason, there's
322. 11:37just not enough of him in the weights
323. 11:39and stuff in it. You'll see that like
324. 11:41each of these, it doesn't really work.
325. 11:43But if I create a Laura, and you can see
326. 11:45basically I've made a custom Laura of
327. 11:48him here, trained up on not a huge
328. 11:51amount of images, and probably not the
329. 11:53best quality images, and actually many
330. 11:55of the images when I scraped them were
331. 11:57actually from things that he did when he
332. 11:59was much younger at Y Combinator, etc.
333. 12:01So, that's why they tend to come out
334. 12:03looking a bit younger. But if you put
335. 12:04the effort in, you can actually train up
336. 12:06a Laura pretty quickly that will put
337. 12:09someone into the photos, even though
338. 12:11they weren't in the original training
339. 12:13data set. Just to sort of compare that,
340. 12:16this is one that I did with nano banana
341. 12:19with him. And you can see that, yeah,
342. 12:21okay, it's gotten better quality on
343. 12:23certain things, but it still hasn't got
344. 12:25his face perhaps as good as I would like
345. 12:28here. And that's coming about here
346. 12:30because we're not doing image editing.
347. 12:32We're not conditioning on a picture of
348. 12:34him to actually generate. We're actually
349. 12:37generating from the weights of the model
350. 12:39out there. So, that really shows where
351. 12:41the weakness of this model is, is that
352. 12:43because you can't do image editing, you
353. 12:46either have to fine-tune through a Laura
354. 12:48or something like that to actually
355. 12:50inject a person or a topic or a style
356. 12:53into the actual weights itself, rather
357. 12:56than just using it in context and doing
358. 12:59some kind of editing here. So, I do hope
359. 13:01that they would go on to make a version
360. 13:02of this that actually can do image
361. 13:04editing and can condition on images, not
362. 13:08just condition on text, etc. All right.
363. 13:10So, if you need text-heavy images or you
364. 13:13need things placed exactly where you
365. 13:15want them, and you want a license that
366. 13:17you can actually build a product on,
367. 13:19this is really an interesting model for
368. 13:21its size. If you need editing, yeah,
369. 13:23it's not quite there yet. You'll have to
370. 13:25wait for a newer model or a newer
371. 13:26version of this as it comes. Anyway,
372. 13:29I've put the links in the description.
373. 13:30Let me know in the comments what you
374. 13:32think if you're playing with this, what
375. 13:34you're sort of noticing this to be good
376. 13:35at, what you're noticing it to be bad
377. 13:37at. As always, if you found the video
378. 13:38useful, please click like and sub-
379. 13:40scribe, and I will talk to you in the
380. 13:41next video. Bye for now.