He Spent $1M on A/B Tests. Here's What Won. — Transcript

Full transcript

1. 0:0040% of people who install this app from
2. 0:02an ad end up paying for it. This
3. 0:05astrology app spent over a million
4. 0:07dollars on AB tests and made $20 million
5. 0:11with no investors. All the common advice
6. 0:14we hear about onboarding, pay wall,
7. 0:16pricing, and even giving free trials,
8. 0:18all of it failed those tests. So, I
9. 0:22asked the founder to come on the channel
10. 0:23to share all his experiments that won
11. 0:26and failed in the last 6 years. The
12. 0:29usual advice is to keep onboarding
13. 0:31short, but based on what we found in our
14. 0:34previous episode, studying over a
15. 0:35thousand onboarding flows, most of the
16. 0:38top crossing apps have one of the
17. 0:40longest onboarding flows. Moonly has a
18. 0:43similar approach. VR on boarding is 30
19. 0:46screens long. Over the last 6 years, we
20. 0:49had three major iterations of
21. 0:51onboardings. We focused mostly on
22. 0:53[music] features, but not the users
23. 0:55problem. The latest version shifted the
24. 0:58focus to users jobs to be done or the
25. 1:00pain behind it. And we offer an
26. 1:02activational insight. So we give you
27. 1:05small bite-sized information that
28. 1:07unlocks some insight such as [music] how
29. 1:10lunar cycles can affect your mood. And
30. 1:13finally, we introduce the feature by
31. 1:15showing how it solves your specific
32. 1:17problem. [music]
33. 1:18Since the app has 40 features and not
34. 1:20everyone uses each of these features,
35. 1:22they tailor the onboarding flow based on
36. 1:25the ad you came from.
37. 1:27>> We just finished developing our own
38. 1:29attribution engine. Today we can connect
39. 1:31user from ad network to specific custom
40. 1:34product page on app store and then to
41. 1:36specific [music] custom onboarding flow
42. 1:39and even a specific app experience.
43. 1:41Someone who simply search only will
44. 1:43[music] see the standard page but
45. 1:45someone who search specific feature like
46. 1:48to they will see fully customized page
47. 1:50and set of screenshot. We now have
48. 1:52around 10 different onboarding flows in
49. 1:54production each built around a specific
50. 1:57user job even anxiety or ADHD.
51. 2:00>> By customizing their onboarding flows
52. 2:02their customers lifetime value doubled
53. 2:04which means their customers are worth
54. 2:06twice more than before. This led them to
55. 2:08double their ad spend in a single month.
56. 2:10>> Our cos are now profitable from day zero
57. 2:13just because custom on boardings.
58. 2:15[music] During the onboarding, we
59. 2:17collect as much information as we can
60. 2:19including birth details and personalize
61. 2:22the first session.
62. 2:23>> So your user finishes onboarding and
63. 2:25lands on the homepage for the first
64. 2:26time. What do they see?
65. 2:28>> They see a daily guidance card,
66. 2:30moonface, personalized daily program.
67. 2:32Plus, they provide something called
68. 2:34personal transits,
69. 2:35>> which works as a perfect hook because
70. 2:37[music] you keep thinking about your
71. 2:40transits and you remember about this and
72. 2:42you expecting when your transit will
73. 2:44finish, when the next transit will
74. 2:45start. Quick assess buttons on main
75. 2:48screen are also customized [music] based
76. 2:50on their on boarding answers.
77. 2:52>> Customizing their whole experience
78. 2:54worked wonders for their app retention
79. 2:56numbers. Our day one retention now is
80. 2:5941%. And day 7 37%.
81. 3:03>> If you're trying to grow an app, you
82. 3:04would want to be featured as app of the
83. 3:06day. Only has been featured on the app
84. 3:08store multiple times. Did this move the
85. 3:11needle for you?
86. 3:12>> It worked greatly 15 years ago, but in
87. 3:15reality, being feature stopped
88. 3:17delivering the same benefits today. It
89. 3:19tends to bring a lot of irrelevant
90. 3:21traffic. Irrelevant traffic shows worse
91. 3:24conversion rates. So it just hurts your
92. 3:26statistics. The app store algorithms
93. 3:29analyze it and may rank you lower in
94. 3:32search result. It's a problem sometimes
95. 3:34actually. So featuring can still work
96. 3:36well for universal generic apps like
97. 3:39calculators or flashlights, [music]
98. 3:41but it isn't particularly useful for
99. 3:43niche apps.
100. 3:45>> When designing the onboarding flow, we
101. 3:47always debate when to introduce the
102. 3:49login. Is it before the customer arrives
103. 3:51at a homepage or after? On Mobin only
104. 3:5412% of apps have no sign up or login
105. 3:57step anywhere only has 10 million users
106. 4:00and no login screen at all.
107. 4:02>> I always recommend to start with
108. 4:04question why why do we need this?
109. 4:06[music] It's easy to go on afterpilot.
110. 4:09You see same elements and dozens of
111. 4:11other apps and assume we need it too.
112. 4:14[music] Sometimes login even requires an
113. 4:16SMS which doesn't always arrive in
114. 4:18[music] seconds. Just like that you
115. 4:20create a UX nightmare. So they asked why
116. 4:23do they even need login at all?
117. 4:25>> Was it made to let users keep their
118. 4:27data? No, because their data was already
119. 4:30stored in Apple keychain and [music]
120. 4:32even after installing the app on a new
121. 4:34iPhone, you will be instantly locked in.
122. 4:37>> Was it to collect emails?
123. 4:39>> Entering a real email creates
124. 4:40significant cognitive friction [music]
125. 4:43and distracts user from what they
126. 4:45actually came to do. Besides, users are
127. 4:47used to signing in with Apple and hiding
128. 4:50their real email address.
129. 4:51>> Was it for email marketing?
130. 4:54>> But building an [music] effective email
131. 4:55marketing is a huge and complex
132. 4:57challenge. And we haven't yet managed to
133. 4:59make it work efficiently despite the
134. 5:02fact we spent years on this. Even with
135. 5:04strong open rates, sending more than a
136. 5:06million emails over months cost
137. 5:08significant [music] amount of money. So
138. 5:10why did we need login at all? it was
139. 5:13better to remove it and reduce the
140. 5:14cognitive load on our users.
141. 5:16>> When designing pay walls, we also
142. 5:18believe that we should show the right
143. 5:19payw wall at the right moment. [music]
144. 5:21We call this contextual pay walls.
145. 5:23Moonly used to do this as well. They
146. 5:26would show users the relevant payw wall
147. 5:28that matches what they're doing.
148. 5:29>> For instance, taro feature taro payw
149. 5:32wall bus chart screen chart payw [music]
150. 5:34wall.
151. 5:34>> In theory, this sounded like a great
152. 5:36idea, but every single contextual payw
153. 5:39wall failed their AB test. [music] When
154. 5:42people see this pay walls one by one,
155. 5:45they have perception that you have to
156. 5:47buy all these features separately to use
157. 5:50the product.
158. 5:50>> So in the end, you went back to the
159. 5:52single consistent payw wall.
160. 5:54>> Yep.
161. 5:54>> Orbit's payw wall makes it really clear
162. 5:56that if you pay today, you unlock all
163. 5:59features. There are two types of pay
164. 6:01[music] walls. The soft payw wall is a
165. 6:03payw wall where you can close it and
166. 6:05keep using the app. Whereas the hard
167. 6:07payw wall, there's no way past it
168. 6:09without paying. You've tested both soft
169. 6:12payw wall and a hard pay wall. Which pay
170. 6:14wall won?
171. 6:15>> Across all our tests, the soft pay wall
172. 6:17won. The close button only appears after
173. 6:205 seconds, but we call it soft pay wall.
174. 6:22Anyway, in the US, users are much more
175. 6:26accepting of aggressive monetization.
176. 6:28>> Soft payw wall globally whereas hard
177. 6:31payw walls won in the US. But out of all
178. 6:34the paywall experiments, what moved
179. 6:36their numbers the most? I feel like
180. 6:39images are the biggest lever. We run
181. 6:42hundreds of experiments on every element
182. 6:44on the pay wall. [music]
183. 6:45Just a few days, a single winning image
184. 6:47delivered us 2x uplift. And the winner
185. 6:50was not the image that we bet on.
186. 6:52>> Could you guess which payw wall image
187. 6:54won the AB test?
188. 6:58>> The woman [music] on the sunset. Yeah,
189. 7:01>> that was the first image that was in
190. 7:03realistic style that won the AB test.
191. 7:06For some reason, we fell in love with
192. 7:08Disney style, but we failed to consider
193. 7:11that the audience most willing to pay
194. 7:14for our product is over 35 and they
195. 7:16couldn't recognize themselves in
196. 7:18cartoonish 3D characters. They need
197. 7:21something more realistic.
198. 7:23>> Since realism worked, they pushed it
199. 7:25even further.
200. 7:26>> We once decided to [music] personalize
201. 7:28users avatar based on their zodiac
202. 7:31signs. It makes product feel more
203. 7:34personal. But then we decided to take it
204. 7:37further and personalized every image
205. 7:40with user's [music] own face around 50
206. 7:42images in the product which is very
207. 7:44complex and expensive but we build it
208. 7:46anyway. The result aren't always
209. 7:49consistent [music] and some of them are
210. 7:51simply cringe so people are sensitive to
211. 7:54anything involving their own face.
212. 7:56[music] In the end thousands of users
213. 7:58avoiding open the app so we completely
214. 8:00removed this feature. They also tested
215. 8:03eight variants of titles and subtitles
216. 8:05on the pay wall. [music]
217. 8:07>> When we only change the words,
218. 8:09conversion increased by 12%. And I
219. 8:12recommend to always add social proof
220. 8:14like number of downloads, reviews.
221. 8:17>> They had a similar finding with our
222. 8:18previous guests.
223. 8:19>> These social accolades are really nice
224. 8:21to kind of show credibility and
225. 8:23authority.
226. 8:23>> Even mentioning a fine print. [music]
227. 8:25>> This no commitment cancel anytime
228. 8:27subtitle always seems to bump things up
229. 8:30incrementally. If you compare our pay
230. 8:31walls year by year, [music] it might
231. 8:34look like we launched only a few major
232. 8:37versions over last six years. But what
233. 8:39you see today is the result of small
234. 8:41incremental improvements made almost
235. 8:44every day. We started [music] with a 10%
236. 8:46conversion rate in 2020. It was our
237. 8:49baseline and today we reached 40%
238. 8:51conversion on paid [music] traffic.
239. 8:53>> Every element on the payw wall earned
240. 8:55its place. What about the price? [music]
241. 8:58Pricing tests are the most important
242. 9:00experiment you can run in a product. It
243. 9:02comes down [music] to the price and
244. 9:04perceived value.
245. 9:05>> In the past year, Moonly ran 40 [music]
246. 9:08pricing experiments.
247. 9:09>> For example, a highriced lifetime plan
248. 9:12can increase the product perceived value
249. 9:14and niche users toward ano over weekly
250. 9:17plan. So, you don't mean to sell a
251. 9:20lifetime plan.
252. 9:21>> We call this price anchoring. [music]
253. 9:23On Mobin, 4% of iOS apps offer a
254. 9:26lifetime plan and it's usually priced at
255. 9:29about twice the annual plan. How do you
256. 9:32decide which pricing test to run next?
257. 9:34>> People usually don't compare prices to
258. 9:37value, they compare prices to each
259. 9:39other. [music] So, we kept the number
260. 9:42and changed the units. For instance, 8
261. 9:45bucks [music] per month became 8 bucks a
262. 9:47week. 30 bucks per year became 30 bucks
263. 9:50per month. After we rebalance for weekly
264. 9:53[music] payments, money now arrive
265. 9:55weekly instead of monthly. You recover
266. 9:58your ad spend faster. You can scale
267. 10:00acquisition with a shorter feedback
268. 10:02loop.
269. 10:02>> Did the users who converted at $8 a week
270. 10:06churn harder?
271. 10:07>> Yes, but the final interview is bigger
272. 10:10as you have more rebuilds before the
273. 10:12scoreboard will die.
274. 10:13>> There are also a lot of advice around
275. 10:15free trials. They had one. It worked for
276. 10:18a year.
277. 10:19>> But most of the time the unit economics
278. 10:21were much stronger without them. People
279. 10:24make purchase and then in few seconds
280. 10:27they cancel it and it's now a very
281. 10:29common pattern.
282. 10:30>> The birth chart feature is one of the
283. 10:32most valuable features in the app and
284. 10:34trial users got it for free. After that
285. 10:37they feel that they already received 90%
286. 10:40of value from our product and they are
287. 10:42not interesting to pay it
288. 10:44>> and that's why they removed the free
289. 10:45trial completely.
290. 10:47On Mobin 59% of iOS paywall screens do
291. 10:52not offer a free trial. So instead of a
292. 10:55free trial moonly dropped a one-time
293. 10:57offer. They have different mechanics in
294. 10:59place but in this case it's usually on
295. 11:01day three. If price is the blocker, we
296. 11:05don't lose this user. The single payable
297. 11:07lifted revenue between day three and day
298. 11:1030 about 15%.
299. 11:12>> Now, here are four things that works
300. 11:14really well for Moonly. The first one is
301. 11:17sharing in one tap.
302. 11:18>> People don't share features, they share
303. 11:20insights about themselves. If someone
304. 11:23takes a screenshot in Moonly, we offer a
305. 11:26readym made flow that lets you share
306. 11:29this in one tab. 23% of users share
307. 11:32something from the app and as a result
308. 11:34they got millions of free views. Apart
309. 11:37from 40 features in their app, the one
310. 11:39that moves the needle was their AI
311. 11:41astrologer Luna.
312. 11:43>> We now have about 1 million
313. 11:45conversations after feature was launched
314. 11:476 months ago and it grows exponentially.
315. 11:50[music]
316. 11:51We experimented extensively with her
317. 11:53style, vibe, tone of voice, visual
318. 11:55identity. They tested cartoon graphics,
319. 11:58a Disney-like style, and a faceless
320. 12:00approach. But none of these appealed to
321. 12:03their users.
322. 12:04>> When people receive meaningful personal
323. 12:07guidance, they need the entity to feel
324. 12:10trustworthy. So, realism works better
325. 12:12here.
326. 12:13>> On top of subscriptions, they offer
327. 12:15add-ons.
328. 12:16>> We even developed our own viculations
329. 12:19engine. We've created around 10 in-depth
330. 12:22astrology reports that go far beyond the
331. 12:24core experience like soulmate report,
332. 12:27career, astrotography, numerology. This
333. 12:30report is also added to the AI
334. 12:33astrologers context. We just started few
335. 12:35months ago. We already generate around
336. 12:38500k per year on adons and it doesn't
337. 12:42subscription [music]
338. 12:43>> to sell add-ons. They built a smart
339. 12:46system for inapp announcements. It
340. 12:48analyzes each user's context and
341. 12:50anticipates what they might need next.
342. 12:53The system maintains a dynamic queue of
343. 12:56offers that updates every second.
344. 12:58>> The AI astrologer also recommends
345. 13:01add-ons to the [music] users
346. 13:02>> but only when a specific one can really
347. 13:05help user at the moment. And when it
348. 13:07comes to payments
349. 13:09>> with native payments, 50% of
350. 13:11cancellation happened in 3 minutes from
351. 13:14purchase because iOS made cancelling a
352. 13:17onetop habit.
353. 13:18>> In the US, they moved payments of Apple
354. 13:21entirely.
355. 13:22>> So we added Stripe payments as an
356. 13:24alternative in the US and our LTV
357. 13:26doubled overnight.
358. 13:28>> Now about 80% of their US customers pay
359. 13:31with [music] Stripe.
360. 13:32>> We provide small discounts for Stripe
361. 13:34and it's a win-win. I was very skeptical
362. 13:37of in the beginning, but now on iOS, it
363. 13:40works absolutely great.
364. 13:41>> Does that work on Android too?
365. 13:44>> You can only offer alternative billing
366. 13:46with a Google native SDK, which means
367. 13:48you don't control anything. The
368. 13:51interface is designed to scare users
369. 13:53away and it also doesn't support Google
370. 13:55Pay, which kills almost all conversions.
371. 13:58[music] No one wants to enter the cards
372. 14:00details manually. So I don't recommend
373. 14:02to try alternative building conundrate
374. 14:05>> since they run hundreds of AB tests and
375. 14:06[music] experiments each month. How do
376. 14:09they manage it?
377. 14:10>> There is no way we could manage it
378. 14:12manually. [music]
379. 14:13We use Coror and Codex for this. AI
380. 14:16helps us connect the dots and design
381. 14:18[music] experiments around areas of
382. 14:21inefficiency. AI automatically matches
383. 14:24numbers across all our data analytic
384. 14:26sources like post hog adaptive drive
385. 14:29[music] and delivers a clear
386. 14:30recommendations like to ship not to ship
387. 14:33or collect more data.
388. 14:35>> They tested things that people would
389. 14:37probably not even question in the first
390. 14:39place.
391. 14:39>> Every element, every icon, every text
392. 14:42[music] earned its place by winning a
393. 14:44test. Uh in 2026, it's very hard to get
394. 14:48positive unit economics in application.
395. 14:50>> [music]
396. 14:50>> The common mistake is to spend millions
397. 14:53of dollars trying to build amazing
398. 14:55product only after that realize that
399. 14:58market didn't exist for this product or
400. 15:00your approach is incorrect. [music]
401. 15:02>> And he recommends this instead.
402. 15:04>> Start with a web funnel because you have
403. 15:06much more freedom and you will instantly
404. 15:10get the signals from the market. You can
405. 15:12do it alone with AI agents or maybe with
406. 15:15one front- end developer. It's better to
407. 15:17be as lean as you can so you can go step
408. 15:20by step. I recommend to move slowly and
409. 15:23cheap.
410. 15:23>> So, six years, $1 million worth of AB
411. 15:27tests. In our previous episodes, we
412. 15:29studied thousands of onboarding flows,
413. 15:32dashboard designs,
414. 15:34pay walls to find out what works and
415. 15:37what doesn't. If you're into design
416. 15:39databacked [music]
417. 15:40breakdowns on apps and websites that
418. 15:42have already been shipped, this channel
419. 15:44is for you. Thank you for watching and
420. 15:46I'll see you in the next one.