YT YouTube2Text

YouTube2Text → Transcripts → How Gaussian Processes Measure the Unknown

How Gaussian Processes Measure the Unknown — Transcript

by The Synthetic Mind · 2,586 words · 419 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:00In the early 1950s, a mining engineer in
0:03South Africa named Danny Krige was
0:05dealing with a problem that sounds
0:07simple until you try to solve it. The
0:10gold fields around Johannesburg
0:12stretched for kilometers, but the only
0:14way to know how much gold was in the
0:17rock at a particular spot was to drill a
0:19borehole and test what came up. Each
0:22borehole was expensive, so the mines
0:25only had measurements from a small
0:27number of locations spread across a huge
0:29area. Somehow, they still had to
0:31estimate what was happening everywhere
0:33in between. The obvious approach would
0:36be to draw a smooth curve through the
0:38measurements and use that curve to fill
0:41in the gaps. The problem is that the
0:43curve looks just as certain halfway
0:46between two boreholes as it does
0:48directly on top of one. In one place,
0:51the value was actually measured, while
0:53in the other, it was inferred from data
0:55some distance away. Yet, the curve gives
0:58no indication that one prediction should
1:00be trusted more than the other. For a
1:03mine deciding where to dig, that
1:05distinction matters enormously. What
1:08Krige needed was an estimate that also
1:10kept track of its own uncertainty. Near
1:13a borehole, where there was real data,
1:16that uncertainty should be small. As you
1:18moved farther away into areas that
1:21hadn't been measured, it should grow.
1:24The method that developed from this work
1:26is still used in mining and geology
1:28today under the name kriging. In
1:31statistics and machine learning, the
1:33same underlying mathematics appears in
1:36Gaussian processes, which are used
1:38whenever measurements are expensive and
1:41knowing how uncertain a prediction is
1:43matters almost as much as the prediction
1:45itself. The idea is unusual. Rather than
1:49choosing one curve and declaring it to
1:51be the answer, a Gaussian process
1:54considers many different curves that
1:56could explain the data and assigns
1:58different probabilities to them. The
2:01difficulty is that there are infinitely
2:03many possible curves, and each curve
2:06contains infinitely many values.
2:08Somehow, we need a way to describe
2:11uncertainty over all of them without
2:13ever having to list them individually.
2:16To make that possible, start with the
2:18thing we're trying to estimate. There's
2:20some unknown function that takes an
2:22input, such as a position along the
2:24ground, and gives us an output, such as
2:27the amount of gold in the rock at that
2:28position. We can observe the function at
2:31a few locations by drilling boreholes,
2:33but everywhere else its value remains
2:36unknown. A Gaussian process gets around
2:39this by noticing that we never actually
2:42need to deal with all the infinitely
2:44many values of a function at once. At
2:47any given moment, we only care about a
2:49finite set of locations, such as the
2:52boreholes we've already drilled, and the
2:54new positions where we want predictions.
2:57Once we restrict ourselves to a finite
2:59list, the problem becomes one of
3:01describing several uncertain numbers
3:03together, and we already have a tool for
3:05doing that, the multivariate Gaussian.
3:09Imagine just two positions on the ground
3:11separated by 100 m. The amount of gold
3:14at each position is uncertain, so each
3:17value has its own range of
3:18possibilities, but the two aren't
3:20independent. If the rock is unusually
3:23rich in gold at one position, a nearby
3:25position is more likely to be rich, as
3:27well. When we plot the possible values
3:30at the two locations against each other,
3:32the likely combinations form a tilted,
3:34stretched cloud. That tilt tells us how
3:38strongly the two values are related. If
3:40we then measure the gold content at the
3:43first position, the cloud immediately
3:45tells us which values at the second
3:47position have become more or less
3:49plausible. This relationship between
3:51nearby points is what allows one
3:54measurement to tell us something about
3:55places we haven't measured. The next
3:57problem is deciding exactly how strong
4:00that relationship should be for any two
4:02positions.
4:04That rule is called the kernel or
4:06covariance function. You give it two
4:08inputs and it tells you how strongly the
4:11function values at those two inputs
4:13should be related. One of the most
4:15common choices is the squared
4:18exponential kernel where points close
4:20together are strongly correlated and
4:23that correlation gradually fades as the
4:25distance between them increases. If two
4:28inputs are at exactly the same position,
4:31their covariance is as large as it can
4:33be. Move them farther apart and the
4:35covariance falls smoothly until
4:38eventually the two points are far enough
4:40apart that knowing one tells us almost
4:43nothing about the other. The kernel
4:45usually has a few settings that control
4:48how this happens. One is the signal
4:50variance, which determines how far the
4:53function typically wanders away from its
4:55average value. Another is the length
4:58scale, which controls how quickly the
5:00relationship between nearby points fades
5:03with distance. With a length scale of
5:06100 m, for example, points 100 m apart
5:10are still fairly strongly related while
5:12points 300 m apart may have almost no
5:16connection at all. With that rule in
5:18place, we can say precisely
5:20what a Gaussian process is. It's a
5:23probability distribution over functions
5:25with one defining property. Choose any
5:28finite set of input locations and the
5:31function values at those locations
5:33follow a multivariate Gaussian. The
5:36center of that Gaussian comes from a
5:37mean function, often chosen to be zero,
5:41while the covariance between every pair
5:43of values comes from the kernel.
5:46We can even see what this looks like
5:47before collecting any data. Take 200
5:50positions, calculate the covariance
5:53between every pair using the kernel, and
5:55then draw one sample from the resulting
5:57200 dimensional Gaussian. If we connect
6:01those sampled values, we get one
6:03possible curve. Draw again, and we get
6:06another. A short length scale produces
6:09curves that change quickly, while a long
6:11length scale produces smoother curves
6:14that drift more gradually. Different
6:16kernels change the character of the
6:18curves entirely. A periodic kernel
6:21produces repeating patterns, while a
6:23Matérn kernel allows rougher functions
6:25that can look more like real terrain.
6:28Choosing the kernel is therefore how we
6:30describe the kinds of functions we think
6:32are plausible before seeing any
6:35measurements. That collection of
6:37possibilities is the prior. Once
6:39measurements arrive, the probabilities
6:42of those possible functions begin to
6:44change.
6:45Suppose we drill one borehole at
6:47position zero. To keep the numbers
6:50simple, we'll measure gold content
6:52relative to the average for the area, so
6:54zero means average, and we'll use a
6:57signal variance of one with a length
6:59scale of 100 m. The borehole returns a
7:03value of two, meaning the rock there is
7:05well above average. Now, consider a
7:07point 100 m away.
7:09Before drilling the borehole, our belief
7:12about the values at these two positions
7:14formed that tilted Gaussian cloud with a
7:16correlation of about .6. Once we learn
7:20that the value at the first position is
7:22two, we're no longer interested in the
7:25whole cloud. We only care about the
7:27slice where the first value equals two.
7:30A slice through a multivariate Gaussian
7:33is still Gaussian, so the value at 100 m
7:36now has its own ordinary bell curve. Its
7:39mean is the correlation multiplied by
7:42the observed value, giving 6 * 2 or
7:45about 1.2. Its variance is 1 minus the
7:49square of that correlation, which comes
7:51to roughly .6. So, the prediction is
7:53centered around 1.2 with an uncertainty
7:56of roughly .8. The same calculation can
7:59be repeated anywhere along the ground.
8:01Right at the borehole, the correlation
8:03is 1, so the prediction matches the
8:06measured value exactly and the
8:08uncertainty falls to zero. Far away, the
8:11correlation approaches zero, so the
8:13prediction drifts back toward the prior
8:15average, and the uncertainty returns to
8:17what it was before the measurement. If
8:19we calculate that at every position, the
8:22mean curve rises toward the borehole
8:25value and gradually falls back toward
8:27zero on either side. The uncertainty
8:30band does the opposite. It narrows as we
8:33approach the measurement, closes
8:35completely at the borehole, and opens
8:37again as we move away. Now, add a second
8:40borehole at 200 m with a reading of
8:42minus one. The point at 100 m is now
8:45influenced by both measurements, one
8:48high and one low, so its predicted value
8:51moves somewhere between them. Its
8:53uncertainty also shrinks because we're
8:54no longer estimating it from only one
8:57side.
8:58This works for any number of boreholes
9:00because Gaussians have a particularly
9:02useful property. If a collection of
9:04values follows a multivariate Gaussian
9:07and we observe some of those values, the
9:09ones we haven't observed still follow
9:11another multivariate Gaussian. Its new
9:14mean and covariance can be calculated
9:17directly, so there's no need to search
9:19through possible curves or approximate
9:22the answer numerically. The prediction
9:24for the mean comes from three pieces.
9:27First, we calculate how strongly the new
9:29point is related to each borehole, which
9:32tells us how relevant each measurement
9:34should be. Then, we account for how
9:36strongly the boreholes are related to
9:38one another. So, nearby measurements
9:41that mostly repeat the same information
9:43don't get counted independently.
9:45Finally, we combine those relationships
9:48with the actual measurements. The result
9:50is a weighted combination of the
9:52observed values, where the weights
9:55depend both on how relevant each
9:57borehole is and how redundant it is with
9:59the others. In geostatistics, those are
10:02the kriging weights. The uncertainty is
10:05calculated in a similar way, starting
10:07from how uncertain we were before
10:10collecting any data and subtracting the
10:11uncertainty that the measurements have
10:13removed. One useful consequence is that
10:16the measured gold values themselves
10:19don't determine this uncertainty. Once
10:21the kernel is fixed, it depends on where
10:23the boreholes are located rather than
10:25what they found. That means we can ask
10:28how much a new borehole would reduce our
10:30uncertainty before drilling it. We don't
10:33know what value the borehole will return
10:35yet, but we already know how much
10:37information its location is expected to
10:39provide. Real measurements aren't
10:41perfect though, so we usually don't want
10:44to treat every observation as exact. If
10:46a borehole measurement contains noise,
10:49that uncertainty can be added directly
10:51into the covariance matrix. Once we do
10:53that, the fitted mean no longer has to
10:56pass exactly through every measurement,
10:59and the uncertainty no longer falls all
11:01the way to zero at the observed points
11:03because even a measurement taken there
11:05may contain some error. The settings of
11:08the kernel, including the length scale
11:10and signal variance, can also be learned
11:13from the data. For any particular
11:15choice, we can calculate how plausible
11:18the measurements would have been under
11:19the resulting Gaussian process.
11:22That quantity is called the marginal
11:24likelihood. If the length scale is much
11:27too long, the model expects functions
11:29that change very slowly, so a set of
11:32measurements that rises and falls
11:34quickly becomes unlikely. If the length
11:37scale is much too short, the model can
11:39explain almost any pattern, but it
11:42spreads probability across so many
11:44possible functions that the exact
11:47measurements we happen to observe are
11:49still not especially likely. Somewhere
11:51between those extremes is a value that
11:53explains the data without making the
11:56model unnecessarily flexible. Maximizing
11:59the marginal likelihood finds that
12:01balance automatically. Once those pieces
12:04are in place, the Gaussian process gives
12:06us both a prediction and an uncertainty
12:08around it. And that uncertainty can be
12:10used to decide where another measurement
12:13would be most valuable.
12:15That's the idea behind Bayesian
12:16optimization. Imagine training a large
12:19neural network and trying to choose a
12:21setting such as the learning rate, where
12:24each training run takes an entire day.
12:27Trying thousands of possibilities isn't
12:30realistic. So, every new experiment
12:32needs to teach us as much as possible.
12:35We can treat the network's final
12:37performance as an unknown function of
12:40the learning rate and fit a Gaussian
12:42process to the trials we've already run.
12:45At any new setting, the model gives us
12:48two things: an expected performance and
12:51an uncertainty around that expectation.
12:54Choosing the next experiment becomes a
12:56balance between exploiting what already
12:59looks good and exploring what we still
13:01know very little about. A learning rate
13:04with a high predicted score is
13:05attractive because it may already be
13:08close to the best setting, but a
13:10different region with a lower prediction
13:13and much greater uncertainty can also be
13:15worth testing because there may be
13:17something better hiding there. One
13:19common way to balance those two
13:21possibilities is the upper confidence
13:24bound. At each point, we take the
13:26predicted mean and add some multiple of
13:29the uncertainty. A point can therefore
13:31look promising because its expected
13:34value is high, because we're very
13:36uncertain about it, or because of some
13:38combination of the two. The next
13:41experiment is run wherever that total is
13:43largest. Early on, when very little has
13:47been measured, the uncertainty is large
13:49across much of the space, and the
13:51experiments tend to spread out. As more
13:54results arrive, uncertainty shrinks
13:57around the regions we've already tested,
13:59and attention gradually shifts toward
14:01the areas that appear most promising. In
14:04problems where every experiment is
14:06expensive, this can find a strong
14:08setting with far fewer trials than
14:10testing a dense grid of possibilities.
14:13These problems all share an important
14:15feature. Collecting each new measurement
14:18is expensive, so the number of
14:20observations tends to remain fairly
14:22small. That's also where Gaussian
14:24processes are most practical because
14:27their computational cost starts rising
14:29quickly as the amount of data grows.
14:32The main computational problem comes
14:34from the covariance matrix, which has
14:37one row and one column for every
14:39observation. Working with that matrix
14:42requires an amount of computation that
14:44grows roughly with the cube of the
14:46number of data points. So, increasing
14:49the data set by a factor of 10 can
14:51increase the main computational cost by
14:54about a factor of 1,000. That makes the
14:57exact method very attractive for
15:00hundreds or thousands of observations,
15:02but much harder to use when the data set
15:04reaches hundreds of thousands or
15:06millions. Larger applications usually
15:09rely on approximations, often by
15:11representing the full data set using a
15:14smaller collection of strategically
15:16chosen locations called inducing points.
15:19There's also a more fundamental
15:20limitation. Every prediction depends on
15:23the assumptions built into the kernel.
15:25If we choose a kernel that expects the
15:28function to vary smoothly, but the real
15:30world contains a sharp discontinuity,
15:33the model may smooth straight across it.
15:36A geological fault, for example, could
15:39cause an ore deposit to end abruptly,
15:42while a smooth kernel might continue the
15:44prediction across the boundary and still
15:47report fairly narrow uncertainty. So,
15:50the error bars aren't automatically
15:52trustworthy just because a Gaussian
15:54process produced them. They're
15:56meaningful only to the extent that the
15:58assumptions behind the model are
16:01reasonable. The number of input
16:03variables creates another difficulty.
16:05Along a single line or across a
16:08two-dimensional map, measurements can
16:10cover the space fairly well. As the
16:12number of dimensions grows, however, the
16:15amount of space grows with it, and
16:17observations become increasingly sparse.
16:20Points that once looked clearly close or
16:22far apart can also become harder to
16:25distinguish based on distance alone.
16:28Underneath all of the covariance
16:30matrices, kernels, and probability
16:33distributions, though, the central idea
16:35is simple. Instead of committing to one
16:38fitted curve, a Gaussian process keeps
16:40track of many functions that could
16:43plausibly explain the data. Near
16:45measurements, those possibilities are
16:47forced to agree, so the prediction
16:49becomes precise.
16:51Far from the data, they spread apart,
16:53and the uncertainty grows with them.
16:55Thank you for watching. If you enjoyed
16:57this video, please consider subscribing.

About this transcript

This page contains the full transcript of How Gaussian Processes Measure the Unknown by The Synthetic Mind, generated from the public captions YouTube serves with the video. The transcript has 2,586 words across 419 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.