YouTube2Text

I discovered a sound that leaks your exact location (and it's everywhere) — Transcript

by colsto · 5,635 words · 840 segments · language en · Watch on YouTube

Full transcript

  1. 0:00I have found awkward structures in the
  2. 0:01middle of a rainforest, things 600 m
  3. 0:04deep in the ocean, and some stuff I
  4. 0:06can't even share on YouTube. But, what
  5. 0:08I'm going to share today is, no
  6. 0:09exaggeration, the single most powerful
  7. 0:12geolocation clue I've come across. And
  8. 0:15to illustrate this, I'm going to
  9. 0:16geolocate this.
  10. 0:19It's a video, but nothing really changes
  11. 0:21visually. If you want to pause and try
  12. 0:22it yourself, it's linked in the
  13. 0:24description. The only information you
  14. 0:26have is [music] that it was filmed
  15. 0:27September 8th somewhere in the
  16. 0:29continental US.
  17. 0:31That's all you get. But, with that
  18. 0:33information,
  19. 0:34>> [music]
  20. 0:34>> it is possible.
  21. 0:35And I'm going to show exactly how.
  22. 0:41So, I did a video geolocating with
  23. 0:43thunder sounds. The benefit is that they
  24. 0:44travel really far, and they penetrate
  25. 0:46structures extremely well. The downside
  26. 0:48is that we were only able to get about 5
  27. 0:50km precision.
  28. 0:51>> [music]
  29. 0:51>> That could tell you basically what city
  30. 0:53it was filmed in, and maybe what side of
  31. 0:55town. What we're using today has those
  32. 0:57same benefits,
  33. 0:58but here's the precision.
  34. 1:06So, I'm going to play you two sections
  35. 1:08of audio. One has the information
  36. 1:09required to find this location, the
  37. 1:11other does not. See if you can tell
  38. 1:13which one does.
  39. 1:25You probably couldn't tell because it's
  40. 1:27completely inaudible. But, if I open
  41. 1:29this in a spectrogram, here we can see
  42. 1:31all the frequencies. So, if you're not
  43. 1:33familiar with a spectrogram, the X axis
  44. 1:35is time, and the Y axis is frequency.
  45. 1:38Now, even with this, you still really
  46. 1:40can't see anything. It's only when I
  47. 1:41zoom into the right frequency, and then
  48. 1:44increase the resolution, that you start
  49. 1:46to see it.
  50. 1:47I'll do one more.
  51. 1:50This is a Doppler curve from an
  52. 1:52aircraft, and it's what we're going to
  53. 1:54use to locate this video.
  54. 1:57Now, fair warning, this is insanely
  55. 1:58powerful, but it also requires a lot of
  56. 2:01complex stuff. This took me months to
  57. 2:03build, over 5,000 lines of code. I was
  58. 2:05working with 50 GB parquet files, 3D
  59. 2:08physics modeling, country wide DEMs, and
  60. 2:10so much more.
  61. 2:12That's all to say, this is going to be a
  62. 2:13long video, but I'm going to explain
  63. 2:15every step in a way that anyone can
  64. 2:17understand it, even if you know nothing
  65. 2:19about any of this stuff. So, with that
  66. 2:20out of the way, let's look at the rest
  67. 2:21of this audio. We have one Doppler curve
  68. 2:24here. If I keep going to the right,
  69. 2:26you can see another one. It's a lot
  70. 2:28louder, and you can also see the
  71. 2:29harmonics on top.
  72. 2:31I'll zoom out a bit.
  73. 2:33I think we have one more here.
  74. 2:35So, these two are what we'll be using.
  75. 2:37But, let's take a step back. Why do we
  76. 2:38even care about Doppler curves? What are
  77. 2:40we going to be using these for? So,
  78. 2:42there's something called ADSB data, and
  79. 2:44it's data on pretty much every plane in
  80. 2:46the sky. It's position, speed, altitude,
  81. 2:49and a bunch of other things. So, my
  82. 2:51first thought was, could I narrow this
  83. 2:52down to only places that heard a plane,
  84. 2:55and then 920 seconds later, heard
  85. 2:57another one?
  86. 2:58To do that, we need all this ADSB data
  87. 3:01first.
  88. 3:02They have an API you can pay for.
  89. 3:04There's also a Python package just to
  90. 3:06scrape the data, but it's massive. So, I
  91. 3:08ran this for a few hours to get all the
  92. 3:10flights in the continental US on
  93. 3:12September 8th. Now, here's what that
  94. 3:14data looks like. By default, it
  95. 3:16downloads the entire flight path. That's
  96. 3:17why you see things going to Hawaii over
  97. 3:19here or other countries. But, anything
  98. 3:21that flew over the US on that day is in
  99. 3:24this data set, including the two planes
  100. 3:27in our recording. So, now we just need
  101. 3:29to find them. So, like I said, I want to
  102. 3:31narrow down to the locations that heard
  103. 3:32a plane, and then 920 seconds later,
  104. 3:35heard another one. Now, these Doppler
  105. 3:36curves are made by a plane, of course,
  106. 3:39but specifically, a propeller plane. I
  107. 3:41boosted some frequencies so you can hear
  108. 3:42it better. Let me play you a bit of the
  109. 3:43louder one, and you can hear pretty
  110. 3:45intuitively that it's a propeller plane.
  111. 3:55One of the fields in the ADSB data is
  112. 3:57type code. Now, this doesn't say
  113. 3:59propeller versus jet plane, but I did
  114. 4:01some research, found this lookup table,
  115. 4:03and this does. Now, there's some missing
  116. 4:05data in that lookup table, so instead of
  117. 4:06filtering to the propeller planes, I
  118. 4:09just filtered out all the known jet
  119. 4:10types. But, after doing that, the map
  120. 4:13looks like this.
  121. 4:16But now, time for this time offset
  122. 4:18filter. So, I'll do this on a small area
  123. 4:20to illustrate first. This pink line is
  124. 4:22the ADSB data for one flight. You can
  125. 4:25see the individual flight points here,
  126. 4:27where we have coordinates, altitude,
  127. 4:29timestamp, things like that. What we do
  128. 4:31is lay a 200-m grid of points, and then
  129. 4:34for each point, note the time that it
  130. 4:36would have heard this flight. Now, it's
  131. 4:37time for our first new terminology, CPA.
  132. 4:41Not that one.
  133. 4:43Closest point of arrival. So, if we're
  134. 4:45an observer, say, here, CPA is where the
  135. 4:47plane is closest to that observer. The
  136. 4:50reason that's important is when we say
  137. 4:52heard a plane 920 seconds later, hearing
  138. 4:55a plane is not really a single point in
  139. 4:57time. If you're standing at this point,
  140. 4:59you hear the plane here, and here, and
  141. 5:02here. This whole Doppler curve is almost
  142. 5:042 minutes. So, what we need to use is
  143. 5:05the CPA time, which is roughly the
  144. 5:07middle of this. It's when the plane was
  145. 5:09closest and loudest. So, for this
  146. 5:11flight, lay a 200-m grid of points. I
  147. 5:13did a pretty conservative 10-km radius.
  148. 5:16Like, beyond that, you probably wouldn't
  149. 5:17hear a plane even in like ideal
  150. 5:19conditions. So, that's just make sure we
  151. 5:20don't miss anything. And then for each
  152. 5:22one of these points, we note the CPA
  153. 5:24time for this flight. Two important
  154. 5:27things here. First is that CPA time
  155. 5:29actually means the time when the plane
  156. 5:31was closest. For this example, that
  157. 5:33would be probably this point, so
  158. 5:3522:46:36.
  159. 5:38That's actually not what we need,
  160. 5:39though. Now, for the second terminology,
  161. 5:41what we need is the delayed CPA time.
  162. 5:44It's the time that the CPA would have
  163. 5:46been heard by an observer. And that's
  164. 5:49cuz the sound has to travel from here
  165. 5:51all the way 3-plus kilometers to a
  166. 5:53listener. That was the first important
  167. 5:55thing. The second is that planes
  168. 5:58are in the sky. Meaning, the distance
  169. 6:00that we use in this calc needs to be the
  170. 6:02slant distance, not just the horizontal
  171. 6:04distance. Luckily, the ADSB data gives
  172. 6:07the plane's altitude. However, altitude
  173. 6:10is not the height above the ground.
  174. 6:12Altitude is the height above sea level.
  175. 6:14So, to get the slant distance, we
  176. 6:16actually need to know the elevation for
  177. 6:18each grid point. And to do that, we use
  178. 6:20a DEM. That'll tell us at any point the
  179. 6:23elevation.
  180. 6:25So, now with that, we can actually say
  181. 6:27for our point here, for this flight, the
  182. 6:29CPA time of 22:46:34,
  183. 6:32but the delayed CPA time of 22:46:46.
  184. 6:37So, we do this for every point here, and
  185. 6:40that's one flight done. Then we just do
  186. 6:42that for all flights in the area, and
  187. 6:43we're left with this grid of points. Cuz
  188. 6:45basically, every point heard at least
  189. 6:47one flight, especially since I gave the
  190. 6:4910-km radius. But, for each of these,
  191. 6:52here, let me zoom in on one. For each of
  192. 6:53these, what I have is a list of flight
  193. 6:55IDs and the associated delayed CPA
  194. 6:57times. Basically, if you were standing
  195. 6:59here on this day, at each of these
  196. 7:01times, you could look up and could hear
  197. 7:02a plane. So, back to our goal. What we
  198. 7:05want is somewhere that heard a plane,
  199. 7:07and then 920 seconds later, heard
  200. 7:09another one. So, for each of these
  201. 7:10points, we then just check for that
  202. 7:12offset. Here's what that actually looks
  203. 7:13like. So, here's an example point and
  204. 7:15each of those delayed CPA times. And
  205. 7:17then we have a 920-second offset window,
  206. 7:19and we just look here, nothing 920
  207. 7:22seconds later.
  208. 7:26This one again, not quite.
  209. 7:27And then this one over here.
  210. 7:29There we go. Now, when we say 920
  211. 7:32seconds apart, it matters where we're
  212. 7:33measuring from. Like, are you calling
  213. 7:35the center here, or here, or here? I
  214. 7:38mean, that's like a plus or minus
  215. 7:395-second variance. So, that's why if you
  216. 7:41look closely on this, there's a bit of
  217. 7:43wiggle room that we're allowing for in
  218. 7:45that 920-second offset. But, anyway,
  219. 7:47that's the methodology. So, how many of
  220. 7:49our points survived that filter?
  221. 7:52It's about 35,000, and that's for this
  222. 7:54small test area, which is less than
  223. 7:560.05%
  224. 7:58of the continental US land area that we
  225. 8:00need to search. Like, just a side
  226. 8:01tangent, I don't think people understand
  227. 8:03how much land area there is in the US.
  228. 8:05If you divided the continental US into a
  229. 8:07bunch of football fields and laid them
  230. 8:09end to end, it would wrap all the way
  231. 8:11around the earth and to the moon
  232. 8:14and back.
  233. 8:16Almost 200 times.
  234. 8:18That's what we're searching
  235. 8:19for this.
  236. 8:21So we need to do a lot better and we
  237. 8:23can, but we need to know some things
  238. 8:25about
  239. 8:28All jokes aside, this is actually so
  240. 8:30cool. Doppler curves will give us a ton
  241. 8:32of additional information and actually
  242. 8:34don't need to teach you anything because
  243. 8:35you already know this intuitively. Let
  244. 8:37me play you two clips.
  245. 8:46You can already tell that clip two was
  246. 8:48going faster. The reason is because the
  247. 8:50drop in frequency was much larger and
  248. 8:53that drop in frequency can actually tell
  249. 8:54us the exact speed that an object was
  250. 8:56going. So assuming linear constant
  251. 8:59velocity, there are only two things that
  252. 9:00determine what this Doppler curve looks
  253. 9:02like. The velocity determines how big of
  254. 9:05a frequency drop and our CPA distance,
  255. 9:07remember that one, determines how
  256. 9:09quickly that drop occurs.
  257. 9:11So for the Doppler curve in our audio,
  258. 9:13we can fit a curve to that and know how
  259. 9:15fast the aircraft is going and how far
  260. 9:18away it was from the listener.
  261. 9:20So now instead of filtering to points
  262. 9:22that heard an aircraft and then 920
  263. 9:24seconds later heard another one, we can
  264. 9:26filter to points that heard an aircraft
  265. 9:28going a certain speed at a certain CPA
  266. 9:30distance and then 920 seconds later
  267. 9:33heard another plane going a certain
  268. 9:35speed at a certain CPA distance. That's
  269. 9:38a lot more constraining. So let's
  270. 9:40measure one of the Doppler curves. You
  271. 9:41can see that the velocity makes the
  272. 9:43frequency drop larger and then the CPA
  273. 9:45slant distance just changes how quickly
  274. 9:47that drop occurs. What you'll notice on
  275. 9:49this one is that it doesn't fit a
  276. 9:51perfect Doppler curve though, like this
  277. 9:53right hand side does. Left hand side,
  278. 9:56not so much. That'll be important later.
  279. 9:58But what we do in the meantime is we
  280. 9:59just take a pretty generous range for
  281. 10:01velocity so that we don't accidentally
  282. 10:03filter anything out. So here's kind of a
  283. 10:05decent fit on the right side at 100 m/s.
  284. 10:08I can't confidently say this plane was
  285. 10:10going exactly 100, but we can be pretty
  286. 10:12sure it wasn't going 70. Similarly for
  287. 10:15the time offset, there's still a couple
  288. 10:16seconds of like potential measurement
  289. 10:18error here. Like okay, best match
  290. 10:20151.15. I'll move to 151.13, slide that
  291. 10:24fundamental frequency up a little bit,
  292. 10:26and like yeah, that you know, that could
  293. 10:27work. That's just to say we still can't
  294. 10:29pin down the offset time precisely to
  295. 10:31the second.
  296. 10:32But we do this for event one and event
  297. 10:34two, and we get the measurement ranges
  298. 10:36for each.
  299. 10:37So the first thing is the speed filters.
  300. 10:39Here are all of the prop flights, but
  301. 10:41here are all the ones matching the event
  302. 10:43one velocity range. And then in pink,
  303. 10:46here are the same ones before event two.
  304. 10:48So now what we're looking for is a point
  305. 10:50that heard a green line, and then 920
  306. 10:53seconds later heard a pink line. But
  307. 10:55remember the other thing we know about
  308. 10:56this too, in addition to the velocity,
  309. 10:58is the distance away.
  310. 11:00So on this green line for event one, we
  311. 11:02don't need to lay that entire grid from
  312. 11:04zero to 10 km away. We know the pretty
  313. 11:07precise slant distance that the observer
  314. 11:09was from each of these.
  315. 11:11So here are those points.
  316. 11:13They're about 73,000.
  317. 11:15But if I zoom into any of these
  318. 11:16clusters, what do you notice?
  319. 11:20Airport.
  320. 11:22Another airport.
  321. 11:24Most of those 73,000 points are just
  322. 11:26near airports. And it makes sense cuz
  323. 11:29that's where the planes are. More air
  324. 11:30traffic just means higher chance that
  325. 11:32you happen to have two flights that meet
  326. 11:34those criteria. But the signal in our
  327. 11:36video is not only what we hear, it's
  328. 11:39also what we don't hear.
  329. 11:41There's no Doppler curve between these
  330. 11:43two. That's almost 15 minutes of not
  331. 11:45hearing a plane. You're not going to get
  332. 11:47that right outside of a major airport.
  333. 11:49And that's part of why this is so
  334. 11:50useful, cuz if you're near an airport,
  335. 11:53you hear a lot of flights, and you get a
  336. 11:54lot of signal. If you're not, then the
  337. 11:56silence itself is also a signal. So it
  338. 11:58really works anywhere on that spectrum.
  339. 12:01So what that means for our search, I'll
  340. 12:03take all of these 73,000 points, and
  341. 12:06I'll apply a silence filter. Feel free
  342. 12:08to pause here if you're interested in
  343. 12:10the specific thresholds and logic I
  344. 12:12used, but overall I just disqualified
  345. 12:13any point that would have heard a flight
  346. 12:15between our two events. I was pretty
  347. 12:17conservative with the thresholds. So, to
  348. 12:19recap, we took our velocity filtered
  349. 12:21flight segments for event one and event
  350. 12:23two, laid candidate points based on the
  351. 12:26slant distance range from the Doppler
  352. 12:28curve measurements. Then, for all those
  353. 12:30candidate points, did the time offset
  354. 12:32filter where that point would hear a
  355. 12:33flight of the right speed at the right
  356. 12:36distance away, and then 920 seconds
  357. 12:38later hear another flight of the right
  358. 12:40speed at the right distance away. That
  359. 12:42gave us this roughly 70,000 points. Now,
  360. 12:45adding on to all that logic, the silence
  361. 12:47filter where that point doesn't hear
  362. 12:49another flight between the two events.
  363. 12:51That drops the 70,000
  364. 12:53to
  365. 12:5515,000.
  366. 12:57Even with all those constraints, there's
  367. 12:59just so much land area and so many
  368. 13:02flights that 15,000 points still meet
  369. 13:05all those criteria. And usually at this
  370. 13:07point in a search, I would just start
  371. 13:08looking at some of the points, seeing
  372. 13:10why they don't work, and then encoding
  373. 13:12that logic as another filter.
  374. 13:14The problem here is we just have this
  375. 13:16picture. There's nothing else we can get
  376. 13:19from this. And we used pretty much
  377. 13:20everything we could from the
  378. 13:21spectrogram. The velocity, the slant
  379. 13:23distance, the time offsets, even the
  380. 13:26absence of the planes between the two.
  381. 13:28There's just not much more we can do.
  382. 13:31So, thanks for watching.
  383. 13:33I'm just kidding. Of course we're going
  384. 13:34to find it. Remember when I said this.
  385. 13:37So, assuming linear constant velocity,
  386. 13:39there are only two things that determine
  387. 13:41what this Doppler curve looks like. And
  388. 13:43this. What you'll notice on this one is
  389. 13:45that it doesn't fit a perfect Doppler
  390. 13:47curve, though.
  391. 13:49The reason they don't fit an ideal
  392. 13:51Doppler curve is because planes don't
  393. 13:53fly in steady, perfectly linear motion.
  394. 13:56They change heading, they speed up, they
  395. 13:58slow down, and all of that info is
  396. 14:00encoded in this Doppler curve. But, to
  397. 14:03get it out, we're going to need to go a
  398. 14:05layer even deeper.
  399. 14:09First, we looked at whether a flight
  400. 14:11occurred at this offset. Then, we looked
  401. 14:13at whether a flight of a certain speed
  402. 14:16occurred at this offset at a certain
  403. 14:17distance. Now what we're going to be
  404. 14:19asking is how exactly did that flight
  405. 14:22pass over? Now you can't go from
  406. 14:24spectrogram to knowing the exact flight
  407. 14:26geometry. Multiple things can cause
  408. 14:28variations in this. It could be a change
  409. 14:30in speed, heading, altitude, any
  410. 14:32combination of those. So instead of
  411. 14:34trying to reconstruct what flight
  412. 14:36geometry would have produced this, what
  413. 14:39we can do instead is model what the
  414. 14:40exact Doppler curve would have been for
  415. 14:43the actual flight geometry of each
  416. 14:45flight. We then do this for each
  417. 14:47candidate point on each flight path, and
  418. 14:49then we find the one that matches.
  419. 14:52So, how do we model a Doppler curve?
  420. 14:57The Doppler curve is a function of the
  421. 14:59radial velocity of the plane with
  422. 15:01respect to the observer. That sounds
  423. 15:03complicated, all it means is how fast
  424. 15:05was the plane moving directly toward or
  425. 15:08away from the observer. So if this is
  426. 15:10us, and we're right in front of the
  427. 15:11plane, plane velocity is 90, radial
  428. 15:13velocity is 90. If we're directly to the
  429. 15:15side, plane velocity is 90, radial is
  430. 15:18basically zero. It's not going toward or
  431. 15:21away from us. So at this point, there's
  432. 15:23no Doppler effect. And anywhere in
  433. 15:24between, we can see that red line
  434. 15:26representing the radial component of the
  435. 15:28plane's velocity. You can see here how
  436. 15:30the Doppler curve would change depending
  437. 15:33on the actual flight geometry. So how
  438. 15:35we'd actually create this Doppler curve.
  439. 15:37What we want to know is what frequency
  440. 15:39did the observer hear over time? So
  441. 15:42we'll start at T equals zero or CPA. In
  442. 15:44this specific example, from the flight
  443. 15:46data, that's 7:12:30. We don't know the
  444. 15:49actual time from the recording, this is
  445. 15:50just an example to illustrate. So for
  446. 15:52that time, we go look at the flight data
  447. 15:54at 7:12:30, and we get the plane's
  448. 15:56radial velocity. In this case, since
  449. 15:59it's at CPA, the radial component is
  450. 16:00zero. But we then take that radial
  451. 16:02component and run it through the Doppler
  452. 16:04equation, and it outputs a frequency.
  453. 16:06The Doppler equation looks complicated,
  454. 16:08it's not really. C is just a constant
  455. 16:10for the speed of sound. V sub R, that's
  456. 16:12the radial component that we just
  457. 16:14calculated. The only other term in here
  458. 16:16is F sub zero, that's the fundamental
  459. 16:18frequency. And that just comes from here
  460. 16:19where we measure the curve. Just
  461. 16:21remember how we slide this up and down.
  462. 16:23So that's all the components. It outputs
  463. 16:25130 hertz and then we plot that for 7 12
  464. 16:2830. Then we just do the same thing again
  465. 16:30for 7 12 35. Check where the plane is,
  466. 16:33get its radial velocity, 62.7, run that
  467. 16:36through the Doppler equation, and get
  468. 16:38109.9, and then plot that there. If you
  469. 16:40do that for all these points, you end up
  470. 16:42with the Doppler curve for this
  471. 16:44observer.
  472. 16:45However,
  473. 16:46that's actually wrong and it would make
  474. 16:48our search fail entirely. I'll give you
  475. 16:50a second to pause and think about why.
  476. 16:52The reason is we want to model the
  477. 16:53frequency heard by the observer at each
  478. 16:56of these times. So the frequency heard
  479. 16:58by the observer at 7 12 30 is not based
  480. 17:01on the plane's radial velocity at 7 12
  481. 17:0330.
  482. 17:05It's actually based on the plane's
  483. 17:06velocity when it produced the sound that
  484. 17:09we then hear at 7 12 30. So the correct
  485. 17:11flow is for t equals zero, find where
  486. 17:13the plane was when it produced that
  487. 17:15sound, take that radial velocity, run it
  488. 17:17through the Doppler equation, and then
  489. 17:19plot that frequency. And so here's what
  490. 17:21that looks like.
  491. 17:23Now if we just do that to the end, we're
  492. 17:24left with two curves. One is the wrong
  493. 17:26way, one is the right way. And you might
  494. 17:28just think, "Okay, sound takes time to
  495. 17:29travel, so just delay the curve. What's
  496. 17:31the difference?" It's not just the same
  497. 17:34curve delayed in time. See if I match up
  498. 17:36like this point, very different curve.
  499. 17:39Same thing over here, match up that
  500. 17:41point. So that first methodology didn't
  501. 17:43just make the right curve, but a few
  502. 17:45seconds late. It made an entirely wrong
  503. 17:47curve. Here's an actual example. So this
  504. 17:49is our spectrogram. This is what that
  505. 17:51wrong curve would have plotted versus
  506. 17:53this is the right methodology.
  507. 17:54So that's how we model this Doppler
  508. 17:56curve for a given observer. We then just
  509. 17:58repeat this process for all the
  510. 18:00candidate points and all of the flights.
  511. 18:03But what we're left with from that is
  512. 18:05the exact Doppler curve that each point
  513. 18:08would have heard based on real flight
  514. 18:10geometry. So then we have all these
  515. 18:12curves. The question is, how do we score
  516. 18:14them? So we take the Doppler curve, we
  517. 18:16plot it on a spectrogram, and then based
  518. 18:18on how close it looks, I raise up a
  519. 18:20scorecard, kind of like gymnastics. I'm
  520. 18:22just kidding. We do plot it on a
  521. 18:23spectrogram, but we just sum the
  522. 18:24brightness values that each curve
  523. 18:26covers. There's two things we have to
  524. 18:28account for, though. One is that the
  525. 18:30fundamental frequency that we used to
  526. 18:31model this curve was based on a
  527. 18:33measurement here. So, that could
  528. 18:35introduce some source of error. The
  529. 18:37second is, similarly, the CPA time is as
  530. 18:40well.
  531. 18:41So, the first thing, we have this
  532. 18:42modeled Doppler curve. How do we know
  533. 18:44where to put it on the spectrogram for
  534. 18:46scoring? Well, for the modeled curve, we
  535. 18:48have a delayed CPA point. That's just
  536. 18:50from flight geometry. But, remember, we
  537. 18:52don't know the time of filming. So, we
  538. 18:54could know the delayed CPA time for this
  539. 18:56flight, but on our spectrogram, we have
  540. 18:58no idea where that is. But, we do have
  541. 19:00the CPA time in terms of just the
  542. 19:01timestamp of the audio. So, that lets us
  543. 19:04say, "Put the delayed CPA time right at
  544. 19:06this timestamp." But, back to what I was
  545. 19:08saying, that is a measurement, and it's
  546. 19:10a source of error. So, the way we
  547. 19:11account for this is when we place the
  548. 19:13curve on the spectrogram, we do a sweep
  549. 19:16of a few seconds on both of those
  550. 19:18variables that we measured. So, the
  551. 19:20slight time offset, and then same thing
  552. 19:21for the fundamental frequency. And then
  553. 19:23we just take the highest scoring
  554. 19:25combination of those. That's how we
  555. 19:27score event one here. Event two is a
  556. 19:29little bit different. We still do the
  557. 19:30fundamental frequency sweep, cuz that is
  558. 19:32still a measurement. However, event
  559. 19:34two's temporal placement relative to
  560. 19:36event one is entirely determined by the
  561. 19:39flight geometry. Meaning, if we correct
  562. 19:41for that error, when we place event one,
  563. 19:43and event two is still placed at a bad
  564. 19:46offset time, that genuinely just means
  565. 19:48that this is not a good candidate. And
  566. 19:50we want to capture that, too. So, that's
  567. 19:52the combined locked scoring methodology,
  568. 19:55where the offset between these two,
  569. 19:57because it's determined only by flight
  570. 19:59geometry, is entirely locked. We correct
  571. 20:01for measurement error on event one, and
  572. 20:03then event two falls wherever it does.
  573. 20:06That's how we eliminate any potential
  574. 20:08measurement error, but still get really
  575. 20:09precise temporal offset scoring. The one
  576. 20:12downside of this is that 200-m grid
  577. 20:15resolution that we've been using is just
  578. 20:17a little bit too coarse for this. Like,
  579. 20:19the actual correct candidate could be up
  580. 20:21to 141 m off, and like kind of score
  581. 20:24pretty bad. That's good because it means
  582. 20:26it's really precise, but it also just
  583. 20:28means we have to take our 200-m grid
  584. 20:30points and expand them into 100-m
  585. 20:33resolution, and just score more points.
  586. 20:35All of this is done for one candidate
  587. 20:38point. So, we need to do this for the
  588. 20:3915,000 times whatever to make them 100-m
  589. 20:42resolution, so probably like 30, 40,000.
  590. 20:45So, super computationally expensive, and
  591. 20:48that's why you see all the steps we did
  592. 20:49prior to this to narrow down are still
  593. 20:51extremely important. Like you just could
  594. 20:53not do this on the full US for the full
  595. 20:56day. So, that's the entire methodology.
  596. 20:58I ran this for all the points.
  597. 21:01Let's take a look at the results.
  598. 21:03So, each candidate point has an event
  599. 21:05one score and event two score. So, I
  600. 21:07placed them on the scatter plot here.
  601. 21:08Higher scores are better, so we're
  602. 21:10looking for things in the top right
  603. 21:11quadrant. So, there's some high-scoring
  604. 21:13ones here, but there's not really one
  605. 21:14that stands out a ton. But, for each
  606. 21:17combination of flight IDs, there's kind
  607. 21:19of a cluster of points, and really
  608. 21:20what's in each cluster, what we care
  609. 21:22about is the highest-scoring point. So,
  610. 21:24I can do basically that on the scatter
  611. 21:25plot, group them by flight pair, and
  612. 21:27then just take the highest-scoring point
  613. 21:29for each flight pair. And now you can
  614. 21:31see there is one that scores a lot
  615. 21:33higher. And I'll show you the actual
  616. 21:35spectrograms and modeled curves for a
  617. 21:37couple of these top ones. So, this
  618. 21:39highest-scoring point with 20.48,
  619. 21:41that's this one here. For event one, I
  620. 21:44mean, you can't even see the Doppler
  621. 21:45curve because it's perfectly covered.
  622. 21:47And then for event two, very similar.
  623. 21:50But, let's also look at the second
  624. 21:51best-scoring one so you kind of see the
  625. 21:53difference between a 20 versus a 15
  626. 21:55score. So, for event one, it's pretty
  627. 21:57close.
  628. 21:58It misses this top bit right here, as
  629. 22:00well as being a little bit off in the
  630. 22:02center. But, I mean, as far as event
  631. 22:03one, this is a 9.5. This is a 10.6. What
  632. 22:07really brings down the score is its
  633. 22:08event two. And you can see its event two
  634. 22:10is pretty badly off. But, this is the
  635. 22:13second best-scoring point. It's kind of
  636. 22:15similar for three.
  637. 22:16Worse on event one, little bit better on
  638. 22:18two. Like none of these are even close.
  639. 22:20And then first place is spot on. So,
  640. 22:23let's Let's look at this point in QGIS.
  641. 22:25So, here are the scored points. Now that
  642. 22:26I know the flight IDs though, what I can
  643. 22:28do is run much finer resolution with
  644. 22:31just that flight. So, like here's the
  645. 22:32heat map when you just score event one.
  646. 22:35And then for event two, you can see it's
  647. 22:36flight path and it's heat map. And then
  648. 22:38combined we have this.
  649. 22:41But all of this has just been dots and
  650. 22:42curves and scores. We still haven't
  651. 22:45found this location.
  652. 22:47I want to do one more thing before we
  653. 22:48hop in a street view because this is
  654. 22:49probably the most insane part of the
  655. 22:52entire project. Do you remember this
  656. 22:54Doppler curve? The one that you couldn't
  657. 22:55hear even when you were listening for
  658. 22:57it. What if you just had this one-minute
  659. 23:00clip, but you knew the time that it was
  660. 23:01recorded? Could you still find this?
  661. 23:04So, say I told you this was 8:37 p.m.
  662. 23:07UTC. We could do the same spectrogram
  663. 23:09measurement, get the velocity bounds for
  664. 23:11the flight, but then also filter down to
  665. 23:14only the segments at that time. So,
  666. 23:16that's the view I have here.
  667. 23:18So, here're the scores for that. Since
  668. 23:19it's only one flight, there's no scatter
  669. 23:21plot, it's just a histogram. But
  670. 23:23right-hand side is better. And here's
  671. 23:25the view if you only take the highest
  672. 23:26scoring point per flight. So, we have a
  673. 23:28couple high scoring points here. And
  674. 23:30here's the spectrograms.
  675. 23:32I did one thing slightly different on
  676. 23:33this, which is because this Doppler
  677. 23:36curve is so faint, I just kind of traced
  678. 23:38it with a line and then scored each
  679. 23:40candidate's curve based on how close it
  680. 23:41was to that line rather than based on
  681. 23:44the actual brightness. So, the top
  682. 23:45scoring point spot on. Second place is
  683. 23:47close, but I mean you can definitely see
  684. 23:49where it's off. And beyond that it gets
  685. 23:51pretty bad. But what you might notice
  686. 23:53about this coordinate, if we go back to
  687. 23:55our original two event scoring, the
  688. 23:56highest scoring point 40.87
  689. 23:5940.87
  690. 24:01-73.5 -73.5
  691. 24:05The same location scoring first by a
  692. 24:07significant margin across two different
  693. 24:09methodologies with entirely different
  694. 24:11flights. And if I were to generalize
  695. 24:13this, you can kind of pick two things
  696. 24:15here that are bad. So, in our first
  697. 24:16example, we didn't know what time the
  698. 24:18video was filmed or where in the US, but
  699. 24:20we had two flights. In the second
  700. 24:22example, we only had one flight. We
  701. 24:23still did the full US search, but we had
  702. 24:25a much tighter temporal constraint. So,
  703. 24:27you kind of just need one of these
  704. 24:28things to be good. The thing is though,
  705. 24:30I've never had a search where all I knew
  706. 24:33was that it was just somewhere in the
  707. 24:34US. You typically know a city or a state
  708. 24:37or even okay, I'm looking north at some
  709. 24:39body of water. Like I chose this picture
  710. 24:41just to demonstrate the extremes of like
  711. 24:43even knowing absolutely nothing, you can
  712. 24:45still find the location. If you knew the
  713. 24:47city, a city is like 0.01% of the land
  714. 24:50area. So, when you constrain the spatial
  715. 24:52bounds that much, it allows you to open
  716. 24:54up the temporal bounds and you don't
  717. 24:56need to know the minute or the hour or
  718. 24:57even the day of filming. I just want you
  719. 24:59to grasp how powerful this is. We got
  720. 25:02the location of that recording within a
  721. 25:04couple hundred feet from one minute of
  722. 25:06audio where you cannot hear the plane.
  723. 25:09And I mean, these flyovers happen all
  724. 25:11the time and those frequencies go
  725. 25:13through walls. Like any video you take
  726. 25:16could have that in there. But anyway, we
  727. 25:18still have not found this exact photo.
  728. 25:20So, let's go back here in QGIS. I'll
  729. 25:22zoom into kind of the center of this. I
  730. 25:24mean, it's honestly in water, so it'll
  731. 25:25be like right on this bank. Grab this
  732. 25:27coordinate and let's take a look in
  733. 25:28Google Maps. And let's see what we got
  734. 25:30for street view.
  735. 25:33All right, let's zoom in here. So, what
  736. 25:34we're looking at is a super pixelated
  737. 25:35tree line. And we kind of want this
  738. 25:37V-shaped tree followed by a point and
  739. 25:40then this big rounded part. So, from
  740. 25:42this location, nothing really that way.
  741. 25:45I'm not seeing any V-shaped trees here.
  742. 25:50Here's kind of one, but we need it to go
  743. 25:51up into the right.
  744. 25:54There's one more here where we have this
  745. 25:56V-shape with a point and then this
  746. 25:58rounded part. Here they are side by
  747. 26:00side.
  748. 26:01And if I zoom out and look at the top of
  749. 26:03this pole, here is the exact security
  750. 26:06camera
  751. 26:07that took this footage.
  752. 26:09So, how do you protect yourself? Well,
  753. 26:10the first thing people might think is a
  754. 26:12high pass filter. There are two reasons
  755. 26:14that this doesn't work. First, the
  756. 26:16frequencies of these Doppler curves are
  757. 26:18pretty much the same frequencies of the
  758. 26:19human voice. You'd be removing a good
  759. 26:21chunk of what you actually want to
  760. 26:23record.
  761. 26:24Second, depending on how you high pass,
  762. 26:26it might not even work anyway. So, say
  763. 26:28this is my recording. I have the Doppler
  764. 26:30curve here. I'll apply the high pass and
  765. 26:32cut off the Doppler curve.
  766. 26:33I'll export this as an MP3, open that
  767. 26:36MP3,
  768. 26:37and the Doppler curve is completely
  769. 26:38gone.
  770. 26:39Now, let me just boost all the
  771. 26:41frequencies, and the curve is back.
  772. 26:45The next thing you might think, "Well, I
  773. 26:46would be talking during my video.
  774. 26:47There'd be other loud things going on,
  775. 26:49so the mic would pick up on that, not
  776. 26:51background plane sounds."
  777. 26:53What if I told you for the past 15
  778. 26:55seconds, I've had a JBL speaker across
  779. 26:57the room playing a Doppler curve?
  780. 27:03So, high pass doesn't work. You can't
  781. 27:04drown it out. One thing that works is
  782. 27:06just making cuts. That way you can't use
  783. 27:08the time offset between these.
  784. 27:11But, remember when we found the location
  785. 27:12with just 1 minute of audio? If it's a
  786. 27:14large search area, you might need to
  787. 27:16know the pretty precise time of
  788. 27:17recording. And you might think, "When do
  789. 27:19you ever get to know the exact time of a
  790. 27:21recording?"
  791. 27:22How about any phone call you've ever
  792. 27:24had? But, phone calls compress audio.
  793. 27:27They apply noise reduction. Surely a
  794. 27:29Doppler curve won't survive that.
  795. 27:32>> But, now that I'm listening to another
  796. 27:34phone speaker under a bunch of squish
  797. 27:36balls,
  798. 27:37I'm in a different room with that same
  799. 27:38JBL speaker playing Doppler curve.
  800. 27:41Just to verify there's no leakage, I'm
  801. 27:43going to go on mute.
  802. 27:48So, with the Doppler curve is going
  803. 27:49through the phone and only through the
  804. 27:51phone, what you should have seen there
  805. 27:52was a gap in Doppler curve. And then,
  806. 27:55when I started talking again,
  807. 27:56it came back.
  808. 27:59>> So, high pass doesn't work. You can't
  809. 28:00drown it out, and it can survive phone
  810. 28:02compression. So, what can you actually
  811. 28:04do?
  812. 28:05Like I said, you could make cuts in your
  813. 28:07videos or apply some additional noise
  814. 28:09cancellation. But, look, I'm not trying
  815. 28:10to fearmonger with this video. The odds
  816. 28:12that someone can and does actually do
  817. 28:14this is pretty slim. In reality, there's
  818. 28:16probably a hundred easier ways to find
  819. 28:19you than this. The one real use case I
  820. 28:21could see, I mean, the audio we used is
  821. 28:23just from some random security camera.
  822. 28:25But, it could have just as well been
  823. 28:27footage from a
  824. 28:28investigation or a human
  825. 28:30video online. Something with literally
  826. 28:33zero visual clues could be located using
  827. 28:36this.
  828. 28:37And maybe that makes you feel
  829. 28:38optimistic, maybe a bit uneasy, or maybe
  830. 28:42like me you just find all this
  831. 28:43fascinating.
  832. 28:44That when you hear a plane fly over,
  833. 28:46you're probably in the only place on
  834. 28:48Earth hearing that exact curve.
  835. 28:51And all the information needed to
  836. 28:52recreate that
  837. 28:54exists.
  838. 28:55And 30 minutes ago
  839. 28:57all of this
  840. 29:00was just background noise.

About this transcript

This page contains the full transcript of I discovered a sound that leaks your exact location (and it's everywhere) by colsto, generated from the public captions YouTube serves with the video. The transcript has 5,635 words across 840 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.