YT YouTube2Text

YouTube2Text → Transcripts → Bird's Eye View_ Hardware - Part 1

Bird's Eye View_ Hardware - Part 1 — Transcript

by NPTEL-NOC IITM · 3,628 words · 536 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01[Music]
0:16So let's start in our third lecture.
0:19Think of it as the bird's eye view of
0:20all of the components involved in
0:23virtual reality systems and they are
0:25hardware, software and the human. And
0:27today we will talk about the hardware.
0:30And um you can also think of this
0:32lectures this one and the next one as
0:35the beginning of the today is the
0:36beginning of the bird's eye view series
0:39uh of lectures that overviews all of the
0:42three components of virtual reality
0:44systems in general which is system uh s
0:47which is a hardware software and human
0:49perception. And the last time we already
0:52covered human senses and sensors a
0:55little bit and then uh today we'll cover
0:58the rest of hardware and uh next lecture
1:02on Tuesday we will talk about uh
1:04software and the human perception
1:07and in our book we already done with
1:09chapter one and we now right now we are
1:12uh in chapter 2 and we'll finish it next
1:15Tuesday. Okay. So very good. Oh, we have
1:18a lot of materials today. So, we're done
1:20with administration. Let's go into let's
1:23dive into our hardware bird's eye view.
1:27Let's start with the review from last uh
1:29lecture though. As we discussed in
1:31class, how many senses
1:34do humans have and what's the correct
1:37answer given that we will be studying
1:40perception in this class and we know a
1:42little more than just you know general
1:44uh public about the things. will be
1:46studying more and many of you maybe
1:48already studied psychology. So what's
1:50the correct answer? The correct answer
1:51is it's complicated. Right? So it's more
1:54it's more than five it we will not talk
1:58about five or six senses. It's more than
1:59six. It's much much more complicated
2:01than that and we will learn more as we
2:03go along. So the cor the correct answer
2:06here is it's complicated. Realistic
2:09graphics is it essential for immersion
2:11in VR or not?
2:17Correct. False. It's it's not uh
2:20essential. At least in general for
2:23majority of people, it will not be
2:25essential because there is cartoon VR
2:27worlds, right? They're not realistic and
2:29we still feel immersed. So, it's good.
2:33And uh how many degrees of freedom does
2:35the human eye have? If you remember, we
2:37discussed it in the end of the last
2:39lecture. So, we have the human eye. If
2:42you want to track it, how many ways
2:44could the eye move? So that we need to
2:46track it. And um
2:50six is incorrect because six is just in
2:53how many ways our head can move. XYZ
2:55translations and rotations around XYZ X
2:59Y and Z. Which one is did I not rotate
3:02like this? So eyes have more than six
3:06because eyes can also move inside of
3:08their sockets as we discussed last time.
3:11And there is also accommodation. So we
3:13we have multif focal kind of ways of
3:15adjusting our eyes. Okay. So that's
3:19good. Nine is correct. Okay. So now I
3:22want to give an overview of your
3:23systems. So all of the components that
3:26are involved in it. So it is hardware,
3:30software and an organism. Right? So in
3:32our case it's humans. So we'll be
3:33studying VR for humans. So here's this
3:36kind of rough picture of all of the
3:38components that we'll have to learn
3:39about in order to understand virtual
3:41reality. So you can see here the CPU,
3:44the controllers and the mouse, maybe
3:48controllers such as mouse and keyboard
3:50or VR controllers, headphones and an
3:53HMD, right? There's also human senses
3:56and the brain. So in this class we will
3:59talk about hardware uh only during this
4:02lecture. So not much uh we will not be
4:04talking too much about hardware but we
4:06will talk much more about software what
4:08goes into all of these algorithms
4:10tracking and rendering methods and we
4:12will talk about uh perception. So how do
4:15we perceive all of this stimulus that
4:17goes from the HMD or from other displays
4:20onto our senses and how we process it
4:23and of course it all will come uh to
4:26comfort issues. So all of this uh
4:29interactions between the systems are
4:31responsible for VR comfort issues. So I
4:35want to paint this abstract picture of
4:38things first. So here on the right of
4:40this diagram is the human organism and
4:43its brain right and the sense or sensor
4:46that moves in the world and provides the
4:48neural information to the brain while
4:50the brain is controlling where the sense
4:53is going in the world. Right? So the
4:55configuration of this of the sense so
4:58some of this control sometimes bypasses
5:00the brain altogether. It's like a reflex
5:02like vestibular ocular reflex but most
5:05of this uh is in a loop from the sense
5:08to the brain and back right. So
5:11generally it's the brain that's doing
5:12the control. So then the sense is
5:14capturing the information through the
5:16transduction
5:18uh from the outside world right. of
5:20transduction of the natural stimuli into
5:23the neural signal. So this is the real
5:26world situation, right? So this is
5:28without doing virtual reality yet. So
5:31the question is what can I put here in
5:33the physical world to fool the brain? So
5:36I look at this sense and look at the
5:39stimulus that comes in and I wanted to
5:42replace it with something artificial or
5:45something engineered. Right? So here on
5:49the bottom is the VR case then. So
5:51instead of the real world I will place
5:53in front of the sense organ something
5:55that I will call a display. And by
5:58display I mean something in a much more
6:01general sense here. So it could be touch
6:04display, it could be vision, could be
6:06audio. H and I want this display to send
6:10the stimulus to the sense. So I have to
6:13figure out how and what to play on this
6:16display.
6:17Uh so so that so that uh whatever is
6:21rendered is transduced by the sense into
6:25the neural uh signal to the brain and
6:29all of this will be done by virtual
6:30world generator that produces this
6:32picture on the display and renders on
6:34the display. Right?
6:36So we will call it a VR system if the
6:39brain is fooled into believing there is
6:42no interference no know this engineered
6:45device. So hopefully for the brain uh
6:48this is how it looks like right so as if
6:51it was the real world. So it's only
6:53whoever the developers the creators uh
6:56of this experience who know about this
6:58artificial stimulation um and this
7:01engineered uh system here and they
7:04succeeded
7:05into building a VR system if they
7:07successfully fooled the brain into
7:09thinking there is known or nearly known.
7:12So this is what it means to build a good
7:15VR experience, right? According to our
7:17definition of virtual reality that we
7:18discussed last time. So now we can
7:22notice the connection between SR, AR, VR
7:27and MR right here. Right? So although we
7:30call it a virtual world generator,
7:33but maybe the world that it's rendering
7:35is
7:37the same place that you're in, right? Or
7:39maybe it's some place like the same room
7:42that you're in but with the delay of 5
7:44seconds. Or it could be recorded in some
7:47way. It could be delayed and augmented
7:49like kind of like AR then. Um it could
7:52be a completely virtual world created by
7:54Unity or Unreal Engine. Could be a
7:57cartoon world. It could be a panorama or
7:592D world. So everything nevertheless it
8:02is a virtual world. It's not a real
8:05physical world anymore. And uh I hope
8:08that now it's very clear that we talk
8:10about this SR, AR, VR, MR systems in a
8:13very general way in this class and we
8:15call them all virtual reality. So now
8:18let's continue with this hardware
8:20overview, all of the components. So
8:23let's name all of these blocks and
8:25arrows in this abstract picture of a
8:28virtual reality system. So the virtual
8:31world generator or CPU GPU through the
8:35process of rendering constructs the
8:38virtual world on the display and
8:41provides the artificial stimulus uh to a
8:45sense organ uh that transduces it into
8:48the neural signal and sends it to the
8:50brain. So the brain is hopefully fooled,
8:53right? And in response, the brain can
8:56control the sense organ to get closer or
8:59further away from the source of
9:00stimulus, right? To look at things
9:02closer or hear things better, which
9:04causes the configuration change of the
9:07display because the display is usually
9:09attached to that sense organ rigidly in
9:12some way. And we this should trigger a
9:16an event in a tracking system to cause
9:19CPU or GPU to update whatever is
9:22rendered on the display appropriately.
9:24So that's called tracking, right? And so
9:27and this is important so that it appears
9:30that the virtual world behave
9:33in the same way as the real world would
9:36would behave when we move in it.
9:40Okay. So I have a question. according to
9:43this big picture and I showed a visual
9:46display here. But what if it was an
9:48audio display for ears? So what would be
9:51the artificial stimulus produced by
9:54audio displays?
9:57So what do they send? Yeah, air pressure
9:59waves. So that's the energy the real
10:01world energy um that they have to
10:04simulate or render to the ears. So
10:07correct air pressure waves. Very good.
10:11So let's start with audio displays and
10:13start thinking about audio displays in a
10:15very general way too. So audio display
10:18is most commonly called a speaker
10:22and CPU rendering the rendering process
10:26produces the appropriate signal uh that
10:29is converted into air pressure waves
10:31which is sound and then the ear uh
10:35transduces th those uh air pressure
10:37waves into the neural signal to the
10:39brain. So let's see. So now uh
10:45right so now let's compare two um very
10:49familiar you're familiar with those uh
10:52audio systems that are quite common
10:55nowadays. So these two systems are
10:58different in how far the displays are
11:00from the ears and hence how the tracking
11:03is done right. So one is going to be
11:06placing a display far away from ears and
11:09another one will be very close one. So
11:12let's see which one is far away, which
11:13one is close one. So in the first
11:15setting on the left here, you're sitting
11:17in a movie theater, right? Or a home
11:19movie, home uh theater. And you can turn
11:22your ears wherever you like, but you
11:25generally do not change your seat,
11:26right? So once you're sitting, you're
11:28there. And there are the speakers then
11:30located in the movie theater. Um and
11:34their labels here, front, right, front,
11:35left, front, left, whatever subwoofers.
11:38So this is called surround sound 7.2.
11:40into right a system. So let's compare
11:43that to wearing headphones.
11:46And with headphones then uh when you
11:49turn your head the sound follows you
11:51right.
11:53So when you are in the theater when you
11:56turn your head the speakers don't follow
11:59you right? So the the uh location of the
12:02speakers is not changing with respect to
12:04your ear. So did you ever think about
12:06this difference? So something
12:08fundamentally different is going on
12:11between these two and we largely don't
12:14even notice it. So somehow we don't care
12:18much if we have no sound tracking. So
12:21sound tracking is not essential for
12:23human comfort. Also the distance to the
12:26audio in the displays is different right
12:28in these two settings. So in 7.2 to
12:30surround sound you're sitting far away
12:32from the speakers while in the
12:34headphones the display is very close to
12:36the to the ear to the sensory organ.
12:40So uh let's talk about very similar two
12:43setups for human eyes. So for displays
12:46for your eyes. So let's make a perfect
12:48correspondence between what we just did
12:50for audio and visual display. So in one
12:52setting the uh the display should be far
12:54away from the eyes and in another one it
12:58should be close by in a VR system. So
13:00what do we get then? Well we get a cave
13:04and an HMD right we talked during the
13:07last lecture I mentioned caves briefly.
13:10So a person in a cave is usually in a
13:12center of the room and there is some
13:15kind of projection system that projects
13:18screens on walls around. So at least
13:21three walls but sometimes four or six or
13:24you can put all all around
13:26and uh you need some kind of polarized
13:28glasses for stereo and tracking right so
13:31that it's updated. So this is what we
13:33would call a cave system and the display
13:36is very far away from your body. So here
13:39it kind of shows. So if you decide to
13:42rotate so only rotate your head without
13:45changing your position then no tracking
13:48needs to be done. So rotate rotational
13:50tracking is provided for free in caves.
13:55But um what else is free? Body tracking
13:58is free. You look down, you see your
14:00body, right? You see your hands.
14:03So now when you start walking in the
14:06cave, that's when tracking needs to be
14:07done. So the objects will need to start
14:09appear that they're closer or if you do
14:12parallax
14:13um because if you don't update what's
14:16what's uh what you see during parallax
14:19then the objects projected on the walls
14:21will appear as if they're painted on the
14:23walls like a wallpaper. So you need to
14:25implement positional tracking. So now
14:28for HMDs
14:30do you need to implement rotational
14:32positional and body tracking? Yes, yes
14:35and yes. So if you choose not to
14:37implement positional or rotation
14:39tracking, then whatever you see inside
14:41of your HMD is going to look like it's
14:44glued to your head, right? So if you are
14:47s sitting in a home movie theater for
14:50example and there is no tracking and you
14:52look at it, it will feel like it's glued
14:54to your head. The whole movie theater
14:56will start turning with your head and
14:58that's as if it's weightless. So it's
15:01usually not comfortable. Okay. So really
15:04um more position more tracking needs to
15:06be implemented the closer the displays
15:08to the eyes it seems like. Okay. So just
15:10to review what is the artificial
15:12stimulus produced then by visual
15:14displays which one of these
15:17uh energy sources I guess or optical
15:20distortion doesn't even have to do it's
15:22not it's it's just a wrong answer.
15:26Optical distortion is is not an source
15:30of energy. Right? So light waves is the
15:32artificial stimulus produced by v visual
15:35displays. That's correct.
15:37Okay.
15:40Which one of the following is a visual
15:41display as defined in class then? So as
15:44we just tal we just defined what are
15:47visual displays? Uh they need to send
15:49some kind of light waves to the eyes and
15:52fool you into believing you're in some
15:54world. So which ones of these could
15:57produce light, right? Essentially. So
15:59which one of them could be a visual
16:01display? Not headphones. They produce
16:04air pressure waves. Not magnifying lens.
16:06They um they interact with the light but
16:10they don't produce light. Right. Camera.
16:12It captures light but it doesn't produce
16:14light. So only phone screens is the
16:17correct answer. Very good.
16:20Okay. Give three examples of visual
16:23displays. So we already said that a
16:26phone screen would be one. A projection
16:29system like a cave would be another one.
16:31A laptop. Perfect. Yeah. So, phone,
16:34phone screen, HMD,
16:36um TV. Yeah. Desktop, television. Yeah.
16:40Monitor, smartwatch. Yeah. Anything that
16:43produces light
16:46could potentially produce enough uh
16:48visual stimulation to to be a display
16:52for VR system. Very good. Okay. Santan
16:56is actually a virtual reality system,
16:58right? Yeah, it could be. Yeah, of
16:59course. For for um for audio for ear
17:04is the sole goal of VR is to fool the
17:06brain. Yes,
17:09I think an organism is a very important
17:11part of virtual reality. I mean
17:13otherwise what this is presented for,
17:16who is going to be the judge of this
17:19experiences, right? So there is an
17:21organism there and usually they have
17:23brains to analyze this kind of stimulus.
17:28Okay. So let's see let's move on maybe
17:33um and we answer more questions as we go
17:36along. Okay. So to summarize
17:40tracking is necessary
17:43for visual uh senses and in HMDs but not
17:48for audio senses and headphones. Right?
17:50So why is that the case? Why do we make
17:52a big deal for tracking for our visual
17:54senses and not for audio senses? This is
17:56because as we learn later, vision sense
17:59is dominant and the brain seems to make
18:02much much more of a big deal analyzing
18:05visual stimuli than any other stimuli
18:07coming from any other senses. Therefore,
18:09the level of fidelity that is needed uh
18:13for your visual senses to analyze or for
18:15the brain to analyze must must be of
18:17higher quality than for audio for human
18:21comfort.
18:23So I want to make a little note on
18:27naming. So conventional naming is not
18:29maybe consistent. So let's think about
18:31it. Headphones, right? So this is
18:33headphones. But what about the bone
18:35conduction hearing aid? Is that a
18:38headphone as well? Right. So then there
18:40should be kind of a maybe uh we should
18:44call earphones the headphones and then
18:46bone conducting herinade is something
18:48else. But then for eyes would it be
18:51iPhone which is not a good name either
18:53because of iPhones and just because it's
18:55phone which is related to sound. But
18:58then for eyes it's used HMD is the uh
19:02the term that is used conventionally. So
19:04headmounted display but headphones are
19:07also headmounted displays.
19:11So to be consistent with what we just
19:14learned probably the best names would be
19:16audio display and visual display.
19:19However
19:21uh we'll use conventional names. Of
19:22course the conventional names is
19:24headphones and HMD. So headphones for
19:26ears and headmounted display for eyes
19:30the visual display. So, um, another
19:35interesting note is bulkiness, right?
19:37So, while with HMDs,
19:41uh, when you're wearing it, it's a break
19:43cover in your face, right? So, you look
19:45ridiculous from the third person
19:48perspective because you're wearing a
19:50blindfold. However, with headphones,
19:53although they're bulky, it's quite okay
19:55to wear them, right? So, you cannot hear
19:57anything when you're wearing headphones,
20:00but that's okay. However, if you don't
20:02see anything when while you're wearing a
20:05blindfold, that's not okay. And in
20:08public, we probably these will never be
20:10accepted, while headphones are perfectly
20:13fine to wear in public. So, something to
20:15think about, but like it just it's just
20:18weird and um and bulky and inconvenient
20:23and uncomfortable to wear a a blindfold
20:26and a brick. So, what's going to happen
20:28with these devices? I don't know.
20:29Probably something more like hollow lens
20:31or glasses light devices will be the
20:35future because this is not really
20:36acceptable.
20:38Now, lot let's talk about power
20:41considerations. So let's look at the
20:42amount of power we need to render
20:45screens at various distances. So for
20:48visual displays in the case for example
20:52the screen is far away from you right?
20:53So, you need to light up the entire a
20:55large area uh entire walls around you,
20:59right? But if you're wearing an HMD,
21:01then the screen is very very close. In
21:04fact, so close that you need to put some
21:06kind of optics lenses so that your eyes
21:10can even focus on the screen that close.
21:13So, that's one consideration,
21:16right? We we can't see things too close.
21:19So, we need optics to see close visual
21:21displays, but you need to then use less
21:24power to light them up, right? Small
21:26small screens. However, if you're
21:29sitting in a movie theater, then uh the
21:31screen is very large and you have to
21:33power a much larger area. So, the same
21:37goes for speakers
21:39or for uh audio displays. They need to
21:42be louder if you're in the movie
21:43theater, but not so much when they're on
21:46your head, right?
21:48Okay. So, there are other considerations
21:50for size of visual displays. So, there
21:54is um clearly an engineering advantage
21:57of having them small, right? You need
21:59less power to light them up. When
22:01they're small, displays are also
22:02portable and potentially cheaper. But to
22:05make them work, these smaller divi uh
22:08displays, you might need to invest in
22:11some kind of high quality optics, but
22:15also high quality eye and body tracking
22:19and then distortion distortion
22:20correction uh for objects potentially.
22:24So the advantages are greater
22:26portability and lower cost while
22:28disadvantages are that you need to have
22:30certain engineering components that are
22:33uh more high-tech.
22:36And these were these engineering
22:38components only appeared recently. They
22:39they became possible recently with
22:42portable screens with OLEDs with
22:45portable IMUs. But in terms of the uh
22:49signal, the strength of a signal itself,
22:52of course, the power of that signal is
22:54cheaper the closer it is to the sense.
22:59And of course, we need to also remember
23:01that there's next generation displays
23:04that are um that are going to be maybe
23:07more popular soon. And these are virtual
23:10retina displays and waveguide optics. So
23:12that's kind of like asking the question
23:14why do we even have to worry about this
23:17uh display uh if we can beam the
23:20information directly on the retina and
23:22that's what this virtual retina displays
23:23and wave guide optics are about. So they
23:26bypass this display altogether and
23:29instead they just stimulate the uh
23:32retina uh itself directly and that is
23:36whole lens 2 and possibly Facebook and
23:38Apple will have something like this
23:40soon.
23:42Okay.
23:46[Music]

About this transcript

This page contains the full transcript of Bird's Eye View_ Hardware - Part 1 by NPTEL-NOC IITM, generated from the public captions YouTube serves with the video. The transcript has 3,628 words across 536 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.