Hardware Testing Guide — Transcript

Full transcript

1. 0:00Hello.
2. 0:00>> Hello. Hey, everyone.
3. 0:01>> Hello.
4. 0:01>> Hello.
5. 0:02>> Wait, you're doing the intro?
6. 0:04>> No, no, I'm sorry. You go for it.
7. 0:06>> Hello. This video is about how and why
8. 0:08to test Roblox games on a variety of
9. 0:10devices, what to expect, and what to
10. 0:12look for.
11. 0:13Roblox runs on everything from phones to
12. 0:14desktop computers, which means different
13. 0:16screen sizes, resolutions, primary and
14. 0:18secondary control schemes, thermal
15. 0:20characteristics, power usage,
16. 0:21performance levels, and more. Let's get
17. 0:23into it.
18. 0:24>> Hey, everyone. It's Andrew from Roblox.
19. 0:26You just heard from my friend and
20. 0:27colleague, Han. The two of us dug up a
21. 0:29handful of test devices, and today we're
22. 0:31going to look at all of them. Along the
23. 0:32way, we'll cover how to think about
24. 0:33device testing on Roblox in general.
25. 0:36Here's the agenda.
26. 0:43>> Why test on hardware?
27. 0:44>> [music]
28. 0:45>> While Roblox has a ton of tools to test,
29. 0:47monitor, and analyze performance,
30. 0:49getting your hands on the actual
31. 0:50devices, especially devices that mirror
32. 0:52those of your target user base, is
33. 0:53essential to understanding how your game
34. 0:55actually plays.
35. 0:56>> We're not just talking about FPS, memory
36. 0:58usage, crash rates, but also user
37. 1:00experience, input and controls, visuals,
38. 1:03and even just figuring out if your game
39. 1:05is enjoyable in a different form factor.
40. 1:07The device emulator in Studio is great
41. 1:09for checking user interface and
42. 1:11controls, but it doesn't simulate
43. 1:13performance and has [music] dramatically
44. 1:15inflated memory numbers. It is no
45. 1:17substitute for testing on a real device.
46. 1:19>> Absolutely. Device testing must be an
47. 1:21ongoing process throughout development.
48. 1:23[music] If you wait to test on a real
49. 1:24device until your game is feature
50. 1:26complete, you'll almost assuredly have
51. 1:27to redo work that you were happy with.
52. 1:29The best path is to build and test,
53. 1:31build and [music] test, and build and
54. 1:32test.
55. 1:36>> Most Roblox players are on mobile, and
56. 1:39in nearly all games and experiences,
57. 1:40[music] low end mobile devices are a
58. 1:42huge chunk of the ecosystem. Let me pull
59. 1:45some numbers from various real-world
60. 1:46games.
61. 1:48In both popular and growing games, the
62. 1:50most prevalent devices are Android
63. 1:51phones and tablets with between 2 and 4
64. 1:54GB of RAM.
65. 1:56Here are a handful of examples from some
66. 1:58games with over 1,000 concurrent
67. 2:00players.
68. 2:01These aren't exact figures and vary by
69. 2:03game, but roughly speaking, 65% of a
70. 2:06game's players might be on Android with
71. 2:0860% of those players having between 2
72. 2:11and 4 GBs of RAM.
73. 2:13Then maybe 35% have between 4 and 8 GBs
74. 2:16of RAM and 5% have more than 8 GBs.
75. 2:19>> The remaining players are typically iOS,
76. 2:21followed by Windows, and then other
77. 2:23platforms. There are exceptions, of
78. 2:24course, such as highly competitive games
79. 2:26or games built specifically for VR or
80. 2:28console, but these are useful
81. 2:30generalizations.
82. 2:31>> For today, we have the following array
83. 2:32of devices: a low-end Android phone,
84. 2:35mid-range iOS phone, high-end iOS
85. 2:37tablet, Windows gaming handheld, and a
86. 2:39high-end macOS laptop.
87. 2:45The first stop when testing performance
88. 2:47is the overlay. On mobile devices, you
89. 2:49can see that the overlay is quite
90. 2:51limited, though.
91. 2:52On Windows and macOS, seen here, we've
92. 2:54got some additional detail, particularly
93. 2:56control shift F2, which shows not just
94. 3:00FPS, but draw calls. For more
95. 3:02information on draw calls, see our
96. 3:03performance optimization video.
97. 3:06As some people are incredibly sensitive
98. 3:07to drops in frame rate, whereas other
99. 3:09people barely notice, so even the basic
100. 3:12overlay is useful for finding problems
101. 3:14in your experience.
102. 3:15If you get persistent dips below your
103. 3:17target frame rate in one area after
104. 3:19performing a certain action or even just
105. 3:21when looking in a particular direction,
106. 3:23you should probably investigate further.
107. 3:25>> But, of course, a frame rate dip only
108. 3:27tells you that there might be a problem.
109. 3:29It doesn't tell you where the problem is
110. 3:31or what the problem could be. To
111. 3:32investigate further, you need more
112. 3:34granular information. You need to know
113. 3:36each task that ran as a part of a frame
114. 3:38and how long it took to run.
115. 3:39>> That's right. You need my favorite tool,
116. 3:41the micro profiler.
117. 3:42>> But first, using the micro profiler
118. 3:45requires an understanding of the
119. 3:46difference between frame rate and frame
120. 3:48time.
121. 3:49Frame time is how long it took a frame
122. 3:51to render on screen, generally measured
123. 3:52in milliseconds. Frame rate is the
124. 3:55number of frames rendered on screen,
125. 3:56generally measured in frames per second,
126. 3:58FPS.
127. 4:00Lower frame time means higher frame
128. 4:02rate. Divide 1,000 milliseconds, 1
129. 4:04second, [music]
130. 4:05by frame time to get frame rate. The
131. 4:07micro profiler deals entirely with frame
132. 4:10time.
133. 4:11In general, you want the bars to be a
134. 4:12nice even height. Tall bars indicate
135. 4:15frame time spikes, where you'll feel a
136. 4:17hitch [music] when you play.
137. 4:18>> We're looking at Beyond the Dark here
138. 4:20largely because it's challenging to run.
139. 4:22In fact, the developers explicitly
140. 4:24recommend against running it on mobile
141. 4:26devices, and you can probably see why.
142. 4:28This Samsung Galaxy S8 from 2017 is
143. 4:32well, struggling.
144. 4:33Frame rate tends to be between 10 and 20
145. 4:35FPS, which means frame times of around
146. 4:3750 to 100 milliseconds.
147. 4:39>> When we check the micro profiler from a
148. 4:41powerful laptop, it's night and day.
149. 4:43It's basically locked at 60 FPS, which
150. 4:45means frame times of 16.67 milliseconds.
151. 4:48>> If I can interject quick
152. 4:49>> Interject away, Andrew.
153. 4:51>> To be clear, we're not dunking on this
154. 4:52phone. It was state-of-the-art at time
155. 4:55of release, and its screen still looks
156. 4:56terrific to this day.
157. 4:58We're playing a demanding game at
158. 4:59maximum settings, and the phone is using
159. 5:01less than 10 watts of power.
160. 5:039 years later, it still feels very
161. 5:05modern and very capable.
162. 5:07>> Unlike mobile devices, you can run the
163. 5:08micro profiler directly on Windows and
164. 5:10Mac OS with control F6 or command F6.
165. 5:13Rather than pausing and trying to
166. 5:15navigate it here, though, consider using
167. 5:16the dump menu and examining the HTML
168. 5:18files at your leisure.
169. 5:20Within these files, you can find device
170. 5:21information in the info menu. But, we
171. 5:24recommend organizing your micro profiler
172. 5:26dumps in folders and hanging on to them
173. 5:27for a while. Maybe rename them to
174. 5:29indicate the scene. And after you make
175. 5:31an optimization or change something
176. 5:33about your game, you can drag and drop
177. 5:34the old file into the new one to compare
178. 5:36the two.
179. 5:37>> Wait, wait. Uh
180. 5:38say more about that.
181. 5:39>> Andrew, didn't you show me this? When
182. 5:41you compare two files, you get a flame
183. 5:42graph that shows the difference between
184. 5:44CPU and memory usage between two [music]
185. 5:45scenes. So, it's easy to see if your
186. 5:47change helped or hurt your frame times.
187. 5:49>> Yeah.
188. 5:50>> You had a whole video on this.
189. 5:51>> I did, yes. Just making sure you watched
190. 5:53it.
191. 5:53>> Despite the name, the miscellaneous
192. 5:55section of the info menu has some very
193. 5:57interesting information, specifically
194. 5:59display size and draw size.
195. 6:02Display size indicates the output
196. 6:04resolution, which is usually the same as
197. 6:06the device's screen resolution, but not
198. 6:07always. And draw size indicates the
199. 6:10resolution that Roblox is rendering at.
200. 6:12In addition to graphics quality, Roblox
201. 6:14automatically lowers its internal render
202. 6:16resolution to improve performance.
203. 6:19See here how the iPhone 13 Mini is
204. 6:22rendering at 1218 by 562, [music]
205. 6:25which is a little more than a quarter
206. 6:26its display resolution, indicating that
207. 6:29it, too, is struggling to stay at a
208. 6:30consistent frame rate.
209. 6:32>> Most players use the automatic graphics
210. 6:34quality setting, which scales graphics
211. 6:36up and down quite drastically to try to
212. 6:38maintain device performance. Lighting,
213. 6:40shadows, draw distance, texture quality,
214. 6:42and particles can all take a big hit
215. 6:43during these swings. If you're trying to
216. 6:45get an apples-to-apples as you make
217. 6:47changes to your game, choose a specific
218. 6:49manual quality level. Otherwise, you
219. 6:51might make changes, test, and think the
220. 6:53[music] changes didn't improve
221. 6:54performance when the reality is that
222. 6:56your performance gains were obscured by
223. 6:57the automatic improvements to graphics.
224. 6:59[music]
225. 7:00>> Another important point to account for
226. 7:01when testing is device thermals.
227. 7:03>> [music]
228. 7:04>> Most devices that don't have fans, such
229. 7:06as phones, tablets, slim laptops, are
230. 7:08able to run at their highest performance
231. 7:10levels for anywhere from 30 seconds to
232. 7:1230 minutes, but eventually they struggle
233. 7:15to dissipate all that heat and they slow
234. 7:16[music]
235. 7:18down.
236. 7:19Sometimes by as much as 40%.
237. 7:21>> A good example is the Galaxy S8, where
238. 7:23our frame rate started around 16 and
239. 7:25drops to about 14 after 5 [music]
240. 7:27minutes of play.
241. 7:28It might not seem like much, but it's a
242. 7:30drop of about 12%. Actively cooled
243. 7:32devices like our Windows handheld can
244. 7:34maintain their performance levels more
245. 7:35[music] consistently. Here's some quick
246. 7:37footage on how this game looks and runs
247. 7:38on a variety of devices.
248. 8:01>> Hans said it earlier, but it's the most
249. 8:03important point in this video. Testing
250. 8:05must be an ongoing process throughout
251. 8:07development, where you change something,
252. 8:10test, encounter a problem, troubleshoot
253. 8:12using the microprofiler, and repeat.
254. 8:15Looking at the microprofiler dump for
255. 8:17each device, the big takeaway for this
256. 8:19experience is that it's super heavy,
257. 8:21especially the lighting and shadows.
258. 8:23>> [music]
259. 8:24>> It's not a competitive game and has a
260. 8:25fairly slow pace, so we can get it into
261. 8:28a playable state on the iPhone 13 mini
262. 8:30if we use the automatic quality setting,
263. 8:32but a lot of the ambience is lost. See
264. 8:34here how it's just brighter and less
265. 8:36atmospheric as quality drops.
266. 8:38>> [music]
267. 8:38>> If you were testing this game on a
268. 8:40low-end device during development, you'd
269. 8:42have to make some choices around what
270. 8:43devices you wanted to support and at
271. 8:45what level of fidelity.
272. 8:47>> Besides performance, it's important to
273. 8:49test how your game feels across multiple
274. 8:50form factors. If the screen size makes
275. 8:52it impossible to read anything, or if
276. 8:54your controls are stuck behind your UI,
277. 8:56or if your game's just really
278. 8:57uncomfortable to play on mobile, none of
279. 8:59your performance metrics really matter.
280. 9:01As Andrew mentioned, the Studio's device
281. 9:03emulator is a great tool to quickly
282. 9:05check your screen size ratios of how
283. 9:07your game might look, but emulation can
284. 9:09tell you something's visible, and only
285. 9:11physical testing can tell you if it's
286. 9:12playable.
287. 9:13>> It's easy enough to optimize for a mouse
288. 9:15and keyboard when testing in Studio, so
289. 9:17in this section we'll focus on things to
290. 9:18look out for with your mobile devices.
291. 9:20Think about the following. The average
292. 9:22session time for some of the popular
293. 9:24games we looked at is between 10 and 15
294. 9:26minutes, and this holds true across
295. 9:28different genres and different player
296. 9:29sizes.
297. 9:31Can you comfortably play your game for
298. 9:32more than 10 minutes at a time on a
299. 9:34mobile device? Consider your button
300. 9:35placement, where your UI elements are,
301. 9:38how often you might need to reach across
302. 9:39the screen. The standard mobile button
303. 9:41UI often feels very natural to players.
304. 9:44Does your experience introduce any UI
305. 9:46elements or menus [music] that deviate
306. 9:48from that standard?
307. 9:49>> What Andrew is getting at is that user
308. 9:51experience is your priority. Besides
309. 9:53inputs and menus, check if you can
310. 9:55easily read your UI on different
311. 9:56devices. The in-studio device emulator
312. 9:58can help here, but nothing helps you
313. 10:00understand the look and feel better than
314. 10:01the real thing.
315. 10:03There may also be some rare cases of
316. 10:04your UI or text elements loading in at
317. 10:06different times on slower devices, which
318. 10:08could be difficult to catch on your
319. 10:10development computer. You'll also want
320. 10:11to test in real-world environments, so
321. 10:13try testing on slow Wi-Fi or cellular
322. 10:15data to get a sense of how the majority
323. 10:17of players will be interacting with your
324. 10:18experience.
325. 10:19Make sure to verify that elements, even
326. 10:21basic ones like movements, input
327. 10:23controls, interaction with the world,
328. 10:24look and feel the way they should. Check
329. 10:26that any calls to the server or updates
330. 10:28to data stores or network calls work.
331. 10:31And if they don't work or delayed, think
332. 10:34about ways they gracefully fail that
333. 10:35won't impact the player's experience.
334. 10:37Since a majority of users are on
335. 10:39lower-end devices, you'll also want to
336. 10:41ensure that your rejoining logic is
337. 10:42fairly painless for that user.
338. 10:45Any other gems of advice, Andrew?
339. 10:47>> We should also call out some quirks we
340. 10:48found during testing on these devices
341. 10:50that felt important to flag.
342. 10:52Keep in mind that these and other unique
343. 10:55situations can occur when you try to
344. 10:56test on a wide range of devices.
345. 10:58>> Oh, definitely. When I was testing on
346. 11:00the iPhone, I noticed the notches and
347. 11:02dynamic islands could obscure parts of
348. 11:03the screen. Both the iPads and iPhones
349. 11:05also had rounded corners, which might
350. 11:06affect the elements that are close to
351. 11:08the edges. The iPhone and the iPad have
352. 11:10a lot of system menus that are
353. 11:11accessible by swiping from various sides
354. 11:13and corners.
355. 11:14>> [music]
356. 11:14>> You should refrain from implementing
357. 11:15buttons or interfaces that might
358. 11:17accidentally trigger these system menus.
359. 11:18>> Those are great points, Andrew. There
360. 11:20are similar system menus on the Samsung
361. 11:22as well. And what was interesting to
362. 11:24note for this device is that the display
363. 11:26extends all the way through the phone
364. 11:28bezel.
365. 11:29Any buttons or text near the left and
366. 11:31right sides of the screen might be
367. 11:32unreadable, distorted, or cut off. So, a
368. 11:35good rule of thumb is to ensure that you
369. 11:37include a safety margin away from the
370. 11:39screen edges for any important elements.
371. 11:41For the ROG Ally, I found that the right
372. 11:43stick was sometimes treated as a mouse
373. 11:46cursor when I had it set to auto, and
374. 11:48that I would have to manually move into
375. 11:49game pad control mode.
376. 11:51>> [bell]
377. 11:52>> It might be useful to have a mouse
378. 11:53option on these handheld devices from
379. 11:55time to time, but players would
380. 11:56typically use the game pad for any
381. 11:58extended sessions.
382. 12:00Roblox also has a default FPS cap of 60,
383. 12:03despite the device's 120 Hz display, and
384. 12:06sometimes defaulted to windowed mode
385. 12:08rather than full screen,
386. 12:09>> [music]
387. 12:09>> which can artificially improve
388. 12:11performance due to the lower resolution.
389. 12:13Even though you should get to know the
390. 12:14quirks of your testing devices, it's
391. 12:16important to understand how that device
392. 12:18fits into the wider Roblox ecosystem.
393. 12:20You should use that information to
394. 12:21inform your testing strategies. [music]
395. 12:23Let's run through our big takeaways.
396. 12:24Prioritize getting a low-end, that's 2
397. 12:27to 4 GB of RAM, test device. It's the
398. 12:30bread and butter of the Roblox user
399. 12:31base.
400. 12:33Leverage the microprofiler on your
401. 12:34devices. Check out Andrew's many great
402. 12:36videos on performance and microprofiler
403. 12:38to really up your game on optimization.
404. 12:40And keep in mind how performance differs
405. 12:42between graphics quality, and how
406. 12:44performance can degrade over a long play
407. 12:46session. And finally, remember that
408. 12:48testing isn't something you do last.
409. 12:50It's an iterative process you should be
410. 12:51thinking about throughout development.
411. 12:53It'll just save you time and headaches
412. 12:54to catch these things early. We know
413. 12:56that's a lot to keep track of. Thank you
414. 12:58so much for staying with us until the
415. 12:59end. For more information on performance
416. 13:02testing, the microprofiler, and more,
417. 13:04check out all of the links in the
418. 13:06description, as well as the
419. 13:07documentation at create.roblox.com.
420. 13:11Happy testing. If you have any testing
421. 13:12tips or tricks, feel free to drop them
422. 13:14in the comments. Thank you all so much
423. 13:16for watching. See you next time. Bye.