A First Look At Raytraced Audio — Transcript

Full transcript

1. 0:00Have you ever been confused by audio in
2. 0:02games where you couldn't tell which
3. 0:03direction a sound came
4. 0:07from? I've spent seven years creating a
5. 0:09game engine and I figured out a way to
6. 0:11use ray tracing to get realistic 3D
7. 0:13audio. And it can also be used to
8. 0:16visualize sound for deaf people. This
9. 0:18process has three steps and the first is
10. 0:21about tricking your brain. Your brain
11. 0:23relies on many factors like the shape of
12. 0:25your ears to work out which direction a
13. 0:27sound came from. But in games, you can
14. 0:29only hear sound from your left and right
15. 0:31headphones. To solve this, games use
16. 0:34binaural audio, which simulates how
17. 0:36sound waves interact with your head and
18. 0:38ears. This tricks your brain into
19. 0:40thinking a sound came from a specific
20. 0:42direction. Most games use this already,
21. 0:44but they still have issues with their
22. 0:46audio. We can solve these problems using
23. 0:48ray tracing. Let's say you're working in
24. 0:51an office and someone's talking in the
25. 0:53room next to you. When you hear them
26. 0:55talk, their voice will be deeper than
27. 0:56usual. But you don't think, "That's
28. 0:58strange. Jaime's voice sounds really
29. 1:00deep today." Your brain subconsciously
30. 1:02understands that there's something
31. 1:04between you and them, and that their
32. 1:05voice will sound a bit deeper than
33. 1:07usual. Then, if their voice sounds
34. 1:09normal, you'll know that they've opened
35. 1:10the door. Just by listening, you've
36. 1:13noticed things about the environment
37. 1:14around you, and this is crucial for 3D
38. 1:16games. To muffle sounds, most games use
39. 1:19a room-based audio system. So, if you're
40. 1:22standing in this room, sounds in the
41. 1:24other rooms will be muffled. But it gets
42. 1:26complicated when rooms have strange
43. 1:28shapes. If you're standing here, how
44. 1:30muffled should a sound over here be? And
45. 1:33it gets more complicated because some
46. 1:34rooms might have thicker walls, doors
47. 1:36that open and close, or they might even
48. 1:38be destructible. This is where ray
49. 1:41tracing comes in. Instead of setting up
50. 1:43rooms for our level, we'll fire rays
51. 1:45outwards from your player. These rays
52. 1:48travel faster than your eyes can see, so
53. 1:50I've slowed them down for this
54. 1:51demonstration. They also move in three
55. 1:53dimensions, but these animations use
56. 1:55two, so they're easier to look at. Also,
57. 1:58in real life, sound travels towards your
58. 2:00ears, but these rays are doing the
59. 2:02opposite. This isn't a perfect
60. 2:04simulation of real life. These rays are
61. 2:06trying to discover sounds and learn
62. 2:08about the environment around you. We
63. 2:10don't have any sounds right now, so I'll
64. 2:12play some music in this room. Now we can
65. 2:14see a new kind of ray that travels
66. 2:16directly towards the music. The more
67. 2:18green rays there are, the clearer the
68. 2:20sound. Listen to how the music changes
69. 2:23when the room closes
70. 2:25[Music]
71. 2:30up. We can change the entire shape of
72. 2:33this building and the rays will
73. 2:34automatically figure out how muffled the
74. 2:36music should be. But ray tracing can do
75. 2:38so much more than this. Here we're
76. 2:40inside a large room and will fire rays
77. 2:42outwards like before. Each time the rays
78. 2:45bounce, they'll check for line of sight
79. 2:46back to your player. These blue rays
80. 2:49represent echo in real life. I used to
81. 2:51think echo was just based on the size of
82. 2:53the room. But one day, I helped my
83. 2:55parents empty out their garage. As we
84. 2:57cleared out more stuff, the echo got
85. 2:59stronger. That's because there were no
86. 3:01more cars and other stuff lying around
87. 3:03the garage that sound could get lost
88. 3:05behind. Now that it's empty, nearly all
89. 3:07the sound is reflecting off the walls
90. 3:09and back to my ears. The room is the
91. 3:12same size as before, but the echo is
92. 3:14much stronger. To have accurate echo in
93. 3:16every scenario, we need to blend between
94. 3:18a few states. The first state is a small
95. 3:21room where you hear the echo instantly.
96. 3:24This could be a bathroom where the walls
97. 3:25are very close to you. The second state
98. 3:28is a large room where the walls are far
99. 3:30away. It takes a while for these blue
100. 3:32echo rays to return to you, so the echo
101. 3:34sounds delayed. We'll blend between
102. 3:36these two states based on the length of
103. 3:38the
104. 3:39Blu-rays. The next step is related to
105. 3:41the garage from before. If all of the
106. 3:43Blu-rays return to you, then the echo
107. 3:45should be loud. But as less blue rays
108. 3:48return to you, the echo should be
109. 3:50quieter. This works great, but it only
110. 3:52applies to indoor sounds. We need to
111. 3:54support outdoor environments as well. If
112. 3:57our building has some broken walls, some
113. 3:59of the rays will leave the building. If
114. 4:01they reach the edge of the level, we'll
115. 4:03say that they escaped outdoors.
116. 4:05As an optimization, these rays will keep
117. 4:07bouncing around so they can gather more
118. 4:09data. Now we can blend between an indoor
119. 4:11and an outdoor state based on how many
120. 4:13rays escape. Now that we can tell if a
121. 4:16player is inside or outside, we can add
122. 4:18another cool feature. When you're at
123. 4:20home in a thunderstorm, you'll hear
124. 4:21muffled rain all around you. But if you
125. 4:24open a window, it will sound like the
126. 4:25rain is coming from that window
127. 4:27specifically. We can simulate this using
128. 4:30the escaping rays. The average direction
129. 4:33of all escaping rays is the direction we
130. 4:35should hear the rain from. But what if
131. 4:37we're in another room and we don't have
132. 4:39clear line of sight with the window.
133. 4:41Let's follow the path of one ray. Since
134. 4:43it eventually escapes out the window, we
135. 4:45can use it to figure out the direction
136. 4:47of the rain. We can do this using the
137. 4:49blue echo rays. The last blue ray is the
138. 4:52direction that we should hear the rain
139. 4:54from. I'll color this ray yellow and
140. 4:56repeat the process with every ray. The
141. 4:59average of all these rays is the
142. 5:00direction we should hear the rain from.
143. 5:03As your player moves around, you can see
144. 5:04the rays point towards the
145. 5:06window. The final feature is related to
146. 5:09wall thickness. Here we have two rooms,
147. 5:12and the one on the left has thicker
148. 5:13walls than the right. Sounds in the thin
149. 5:15room should be less muffled than the
150. 5:17thick one, but currently our rays stop
151. 5:19when they hit a wall. So, let's use a
152. 5:21new kind of ray that travels all the way
153. 5:23to the sound. The longer these rays
154. 5:25spend inside a wall, the more energy
155. 5:27they'll lose. Let's graph this energy on
156. 5:29the right. Each column represents one
157. 5:32ray, and its height is how much energy
158. 5:34it has. When the speaker moves inside
159. 5:36the thicker room, we can see the energy
160. 5:38drop. To work out how muffled a sound
161. 5:41should be, we use the green rays from
162. 5:42before, as well as the energy left in
163. 5:44these orange rays. When used together,
164. 5:47these four ray tracing features produce
165. 5:49realistic audio in complex environments.
166. 5:52But we can also use ray tracing to
167. 5:54visualize sound for deaf people.
168. 5:56Somewhere here there's an enemy firing
169. 5:58their gun. Since deaf players can't hear
170. 6:00them, we need another way to locate
171. 6:02them. Before we were casting rays
172. 6:04outwards from your player, but instead
173. 6:07we'll cast rays outwards from the enemy.
174. 6:09When these rays hit a surface, a small
175. 6:11dot will appear there. As the enemy
176. 6:13moves around, the dots will update in
177. 6:15real time. If the enemy stops making
178. 6:18noise, the dots will disappear. Some
179. 6:20sounds like explosions can be heard from
180. 6:22far away, so the dots should be larger.
181. 6:25But when an enemy is sneaking around,
182. 6:27the dots should be smaller. The color of
183. 6:29the dots can also convey the type of
184. 6:31sound. Gunfire could have red dots and
185. 6:34footsteps might have green dots. If a
186. 6:37player is color blind, we can also
187. 6:39change the shape of the dots. Deaf
188. 6:41players can now see 3D sound visualized
189. 6:44around them. I'm still working on this
190. 6:46feature, and I'd love to hear your
191. 6:47feedback. So, do you need an expensive
192. 6:50graphics card to run this? The answer is
193. 6:52no. This system works on everything from
194. 6:55old laptops to the latest spaceships.
195. 6:57That's because when you play a game,
196. 6:59your input is handled by your CPU, which
197. 7:01tells your graphics card what to render,
198. 7:03and then your monitor displays it.
199. 7:05Whereas, ray traced audio runs on
200. 7:07background threads on your CPU and then
201. 7:09sends the output to your headphones.
202. 7:11This separation allows your game to
203. 7:13continue to run smoothly. The ray
204. 7:15tracing is also performed against a
205. 7:17voxal grid which is much faster to ray
206. 7:19trace against than triangulated models.
207. 7:22The size of the voxels can also be
208. 7:24increased to make it run even faster.
209. 7:26This ray traced audio system will be
210. 7:28available as a paid plugin for major
211. 7:30game engines soon, but I need your help
212. 7:32testing it. If you're a game developer,
213. 7:35let me know if you'd like to help out.
214. 7:37The code for the rest of my engine and
215. 7:38the animations in this video can be
216. 7:40accessed in the video description. I
217. 7:43also optimized another kind of rate
218. 7:44tracing up to 11,000 frames per second
219. 7:47which you can watch in this video on
220. 7:48screen.