YT YouTube2Text

YouTube2Text → Transcripts → Bird's Eye View_ Hardware - Part 2

Bird's Eye View_ Hardware - Part 2 — Transcript

by NPTEL-NOC IITM · 3,468 words · 523 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01[Music]
0:17Oh, absolutely. Yeah. Other security and
0:19privacy guidelines, but there's concerns
0:22for sure. Um, nobody can see what you're
0:25seeing, right? Or whether you're filming
0:28anything. I'm not aware about any
0:30guidelines yet. I I think we're not
0:32there yet. But once they become really
0:33mainstream, if people start wearing them
0:35everywhere, which didn't happen, I'm
0:37sure we will have to have a conversation
0:39about that as well.
0:42Okay.
0:46Uh we really do not want you to feel
0:48disoriented. So you really need to have
0:50perfect calibration.
0:53So please take as much time as needed is
0:56it's crucial that you calibrate your
0:58headset as much as pos as well as
1:00possible and we give you guidelines on
1:02discord so please go there we'll give
1:04you some other tricks that you can do
1:08uh
1:11can hackers oh yes I don't know exactly
1:15like I don't have an answer for you like
1:17I don't have an example um but I
1:20definitely we will go in this class
1:22through a lot of illusions that you can
1:24uh see in VR which are really spooky
1:27sometimes like you know perception of
1:30time or how much you don't even notice
1:32which is right there in front of your
1:34eyes. So we'll we'll talk about it as we
1:36go but I but I don't know like any
1:39anything this dramatic I I don't know an
1:41example but potentially yes very good.
1:49Um uh we are not I I'm not aware of any
1:54of this but maybe I'm wrong. So this is
1:56not my research area and I haven't heard
1:58of real time VR scene generation
2:04but anyway so let's
2:07let's move on. So spatial audio is not
2:11important for immersion. It is that's a
2:13good question. We'll have an entire
2:17series of lectures on audio and I think
2:21professor manan will talk about audio
2:23and you'll learn a lot. I think it's
2:25it's um important but not essential.
2:30Okay. So let's move on. Okay. Now let's
2:33talk about frames for a second for a
2:35little bit. We talked about it last time
2:36but let's talk a little more. So from
2:39last lecture do you remember if we talk
2:42about screens and frame rate refresh
2:45rate of the frames
2:48at 8 frames per second if there's some
2:52motion on the screen will it appear
2:54continuous or not
2:59and the correct answer is false. The
3:02reason is that usually 12 is the
3:05threshold. After 12, you will not
3:07perceive individual frames. Before 12,
3:10you will even perceive the frames
3:12themselves, not even some objects
3:14moving. So, so this is definitely false.
3:17But now I have a a little more
3:18complicated question. So, this requires
3:20a little bit of thinking. If you have a
3:22screen that is really a good advanced
3:24screen, it has 100 frames per second
3:26refresh rate. So, frames change at 100
3:29hertz frequency. And there's some object
3:32on the screen.
3:35So will it always appear continuous? If
3:39you have such great refresh rate,
3:42will any object appear moving
3:45continuously on your screen? So imagine
3:47you're in a movie theater and there is a
3:50rocket going really fast across the
3:52screen and you but you have 100 frames
3:54per second refresh rate. Will you notice
3:56discontinuity?
3:59And the correct answer is not true. So
4:01the the correct answer is false.
4:05Some motions if they're too fast,
4:07they'll appear discontinuous. And that's
4:09why Hollywood movie directors and
4:11Bollywood movie directors, all of the
4:13movie directors out there, they will
4:15avoid any kind of fast objects across
4:18streams uh screens. So they uh
4:21consciously avoid those kind of motions
4:23because even at 100 frames per second if
4:27the object moves too fast you'll just
4:29you know jump too far between the frames
4:33and you'll you'll notice discontinuity.
4:35So that's something that we need to
4:37consider. And now I want to explain even
4:41in more detail. So you can conduct this
4:44experiment on your on your own. take an
4:47LED
4:48uh attach it to the square function
4:51generator so you have it pulsing right
4:54so you can have it turn on and off and
4:57here in this animation the LED is
4:59pulsing at two hertz or three hertz
5:01maybe but imagine that you make it do
5:051000 hertz pulsing so what would you see
5:08if you look at it you wouldn't notice
5:11flicker anymore right so if you look at
5:14an LED like this but pulsing
5:16at higher frequency you would not notice
5:19the flicker while at this low frequency
5:22you notice the flicker and th these kind
5:25of LEDs pulsing LEDs that's the
5:27principle of all of the screens that we
5:29use today all of the modern screens such
5:32as phones LEDs phones and and so on so
5:37then think of this LED and what if you
5:40start moving it what if you start moving
5:43it and while if you move it and you
5:46still follow it with your eyes and it's
5:49uh pulsing at 1000 hertz, you will still
5:52know not notice the flicker. But what if
5:55you don't look at the LED and you start
5:57moving it to on the side from side to
6:00side like this
6:02and you look straight. So you don't look
6:04at the LED and you move this LED very
6:07very fast.
6:09Do you think you would notice flicker if
6:11the motions are very very fast?
6:16And the answer is I didn't put the
6:17slider question but the answer is yes.
6:19You would notice if this motion is
6:21really fast
6:24then the LED even at 1000 hertz would
6:28leave a zip line on your like a zipper
6:31on your retina.
6:34And we've done this experiment in our
6:35research lab. So everybody got to try it
6:37and you would perceive this zipper on
6:40our retina. we would perceive
6:41discontinuity
6:43and this means that some flicker may
6:47never be eliminated in VR even if we
6:50increase frames per second to a,000
6:53hertz and the reason is that our eyes
6:56move really really fast therefore we
6:58will interact with this display at
6:59really really high speeds as this
7:01simulated example so we may never
7:03eliminate flicker some amount of flicker
7:05may always be there or what we will call
7:08there's flicker or retinal image slip
7:10So that those two so some kind of
7:12discontinuity or some kind of uh
7:15problems
7:17may always remain due to the fact that
7:19we will interact with screens at much
7:22higher speeds than um with other
7:25displays.
7:29What continuous motion is? Uh so if an
7:32object moves in real world do you
7:34perceive discontinuities?
7:37uh we actually do sometimes too if they
7:39f if they move really really fast like
7:41for example propellers of the airplanes.
7:43So we still notice some kind of
7:45artifacts due to fast motions but at
7:48most of the speeds that are around us in
7:50nor during normal day uh everyday life
7:53we don't notice discontinuities but on
7:56VR screens even at those speeds we may
7:59notice due to the frame per second uh
8:02due to the refresh rate. Okay, so now
8:04let's look at all of the other hardware
8:06components
8:08in VR systems. So, um let's make sure
8:11that all of the modern VR systems at
8:13least are covered like caves, HMDs, and
8:15these next generation um retinal
8:19displays. So, there is rendering
8:21hardware, that's number one. For visual
8:25uh displays, we have um tubes, plasma,
8:28LCDs, OLEDs, projectors, vers uh virtual
8:31retinal displays, light fields, right?
8:33So we kind of talked about all of them.
8:35Maybe I missed some. So please write
8:38what I missed. Your favorite display
8:40that I missed. For audio, we have
8:42speaker, surround sound, headphones,
8:44bone conduction.
8:47For the touch display. So that's very
8:49interesting because this is called
8:50haptic devices. And I will not talk much
8:53about them, but maybe professor Manuvan
8:56will cover more about haptics because
8:57he's expert on that. So there's but
9:01there's generally two categories for
9:04haptic feedback and they're called
9:06cutane cutaneous versus kinesthesia. So
9:10cutaneous feedback is some kind of
9:12vibration like in your ramble pass in a
9:14game devices. But for kinesthesia it's
9:17when the motion is actually getting
9:18blocked. So you have like this resisting
9:20force. So there may be a robot that is
9:23pushing you against your hand. So it's
9:26much more kind of realistic than
9:28feedback.
9:29Um and um but but yeah, so th those are
9:35the two it can kind of simulate touching
9:37a wall and you know grabbing objects and
9:40stuff like that
9:42but there's also a lot of controllers
9:43like Oculus Touch, HTC Vive, Kinet
9:47Haptive Suits. So all of those are touch
9:50displays. So I will not talk about smell
9:54taste and vestibular devices vestibular
9:57uh displays because they're mostly R&D
10:01but we've tried the smell and
10:03vestibular. I I didn't try vestibular
10:06but I know that I I know people who
10:08tried and it's very painful.
10:10Okay. Then uh optics. So as the displays
10:15get closer to our visual senses we need
10:17some kind of optics. So there's lens and
10:19waveguides kind of optics.
10:22Then we have tracking tracking hardware.
10:24I'll talk about it later in more
10:26details. So we have IMUs, cameras,
10:28lighouses.
10:30Um and uh uh we will not talk about the
10:35hardware too much in this course but we
10:37will talk about uh um tracking software.
10:40So how to write those methods. So you
10:42could actually write your own tracking
10:45uh systems
10:47also input devices. So this is something
10:50interesting right? Right. So what are
10:51the input devices? Game controllers,
10:53keyboards, mouse, data gloves. So when
10:56you're wearing a headset, which is a
10:57blindfold, the last thing that you want
11:00to is to try to type something on a
11:03keyboard, right? So that's really
11:05difficult to have input devices in VR.
11:08Of course, there you can input something
11:10with controllers, but this is a big
11:12limitation essent especially with our
11:14cardboards, right? How are we going to
11:16input any kind of information in our VR
11:19systems? So that's something that we
11:22have surprise for you here. But this is
11:25a big limitation and maybe one of the
11:27big reasons why cardboard uh the whole
11:31project was discontinued.
11:34So uh and finally we should talk about
11:36processing units. So what does the
11:38processing? It's CPU and GPU. Right now
11:42here is an Oculus Quest tearown. So
11:44let's look at that.
11:47Uh so let's look what's inside. Uh
11:50nowadays this is a video uh of the tear
11:53down of Oculus Quest. Not two but just
11:55Oculus Quest. And um the main components
12:00are the same as the original Quests or
12:03very early versions of Oculus Rifts or
12:06whatever um the the early prototypes. So
12:10the main components are or the main
12:12categories of the hardware are cameras,
12:16IMUs, displays and lenses and and they
12:20also have the controllers but we in in
12:23our cardboard will not have controllers
12:25but those are the main categories. Let
12:27me play it again because you will see
12:29that they went through the main
12:30categories as well.
12:34So let's see.
12:36Of course there's a lot of custom
12:38electronics now. So they're not cameras.
12:40Here they are. So cameras for tracking
12:44for positional tracking. I'm used for
12:47orientational tracking
12:50controllers. They have these LEDs here
12:54so that cameras can see them. They
12:57infrared LEDs.
13:00But of course if you if they're out of
13:02sight of cameras, then you cannot see
13:04it. So there's an IMU for rotational
13:06tracking of the controller because with
13:09cameras you can only do positional you
13:11really need IMU for do for doing
13:13orientational tracking. So all those are
13:15the main components in Oculus Quest. But
13:18here is
13:21uh a tear down of a much earlier version
13:24Oculus DK2. So development kit 2. So the
13:29same main components in the left corner
13:32here you see a smartphone screen right
13:35so that serves this as a display this is
13:37a circuit board that has IMUs so again
13:41for tracking uh rotational tracking then
13:44there's lenses because the screen is too
13:46close to the eyes so you need to u
13:49increase the field of view and make your
13:51eyes being able to focus on it the so
13:55here is just a mask to to uh fit it all
13:59in the lenses and the screen. And here
14:02is the there are OLEDs there that are
14:06hidden behind the infrared plastic so
14:09that this camera can see it for
14:11positional tracking. And this is an
14:13external camera. Okay. So the external
14:16camera was added um or is used for
14:20positional tracking.
14:22Okay. So something interesting that I
14:25wanted to point out right here. So
14:27notice that your cardboards as well as
14:30some early development kits of Oculus,
14:33they didn't have this camera. So let me
14:35cross it. They didn't have this camera.
14:36They didn't have this LEDs on the cover.
14:39They only had IMUs and lenses and the
14:42phone. And that's essentially what your
14:44cardboards have, which means that they
14:47only provide three degrees of freedom
14:50tracking. So Google Cardboard, Oculus
14:53Go, Samsung Gear, and early Oculus
14:55prototypes.
14:57had three degrees of freedom. So I have
14:59a question for you. What does it mean?
15:02So what's the difference? How did it
15:03feel? So what's the difference between
15:05six degrees of freedom that it was only
15:09enabled due to the use of cameras versus
15:123° of freedom that was enabled using
15:15IMUs? So that's the question for you.
15:18What does it feel like? So that's that's
15:20the question. Let me rephrase it. What
15:23does it feel like when you bent to look
15:25under a virtual table while wearing 3°
15:30of freedom rotation only HMD? So, you're
15:33in a virtual environment. There's a
15:35table. You want to look under it. How
15:37does it feel?
15:40And I have the answer in the next slide,
15:42but let's see.
15:45Any other guesses?
15:48We cannot go underneath the table.
15:50Translation will not happen.
15:53Okay. Yeah, the environment moves with
15:55your movement. Very good. So, here's the
15:57answer. I have this cool animation. So,
16:00this is what three degrees of freedom VR
16:02headsets are.
16:04When you rotate your head, the world
16:06will behave appropriately. It will not
16:09rotate with you. But if you want to
16:10translate your head in any way, the
16:13world will just not allow you to do
16:14that. So it will be as if they glued the
16:18whole world is glued positionally to
16:21you. So you cannot get to any place. The
16:25world will not respond. While here in
16:27six degrees of freedom tracking
16:31what you expect will happen in virtual
16:33world just like it would happen in the
16:36real world. Right? So you would get
16:38closer without positional tracking. You
16:41cannot get closer in position to
16:43anything. you're stuck. Okay, so I hope
16:46this is clear.
16:49Another part of hardware that is present
16:51in any HMD is optics and we talked about
16:54it. So in most mo in most modern HMDs,
16:59the optics are just magnifying lenses
17:02just like what you have in your
17:03cardboard.
17:05So what do they do? Magnifying lenses
17:09increase the size of the smartphone and
17:11therefore they wrap the image around
17:13your head and increasing the field of
17:16view as well. But then interesting thing
17:18happens that the image then appears or
17:22feels like it's infinitely far away
17:24optically. Probably it doesn't make
17:26sense now but um we'll learn about it in
17:29more details later but you don't even
17:31notice. You probably didn't even notice
17:32that but we'll learn we'll train you to
17:35notice those kind of things. Okay, so
17:38the magnifying lenses then increase the
17:40field of view. But unfortunately, as a
17:42side effect, they also create
17:44distortion. That's your homework. One is
17:46to correct for that distortion. So what
17:49is field of view is the center of the
17:51observable observable world, right? It
17:53was a bit small screen in front of you,
17:55but suddenly you feel immersed. And it's
17:57not as much field of view in your
17:59cardboard. It's probably more like this.
18:01You calculated field of view is like 190
18:04something. It's still small but it's
18:06still much bigger than your phone was.
18:09But field of view is this observable
18:11amount of the world or how much retina.
18:13You can think of it as how much of your
18:15retina is being stimulated by this
18:17stimulus. And distortion is when
18:19straight lines don't appear straight
18:21anymore. Again, you calibrated your
18:23cardboards. You know that you had to
18:24correct for that.
18:27And it is in software that needs to that
18:31distortion needs to be corrected.
18:33And it's not always trivial to do so. So
18:36the reason why it's not trivial is that
18:38lenses do not come with the formula for
18:41the distortion.
18:43And therefore it's not trivial to figure
18:46out what the distortion is. And you do
18:48it by trial and error. Whatever feels
18:50more right uh you you set that
18:53parameter. It is a very tricky thing and
18:56it seems like these distortion
18:57parameters are perceptually
19:00um unique to humans. Because at Oculus,
19:05we try to make a rig, an engineering rig
19:07to exactly compute distortion
19:09parameters, but those parameters were
19:11not as good as something that we set up
19:14manually to fit how we feel. So that's
19:17really interesting. So whatever you're
19:18doing now as calibration, that's very
19:20good perceptual task to fix to to find
19:23the best distortion parameters that fit
19:26fit you.
19:29Okay. So most of the modern HMDs come
19:32with lenses nowadays. However, as I
19:34mentioned uh already some other some
19:38advanced uh optical device HMDs like
19:41Microsoft hollow lens or magic leap or
19:45maybe Apple pro that is still not
19:48released but they promise to have
19:50advanced optics and that those are not
19:52lenses anymore. And maybe I'll have when
19:55we come into when we study optics, I'll
19:57show you what exactly is that retinal
19:59display. So here is
20:03um here are some useful terms for
20:06distortion. Uh and it's called a pin
20:09cushion or barrel distortion. So here's
20:11an example. If you have this original
20:14grid, if you were to put a lens, a
20:16magnifying lens, and look at that grid,
20:18then you would see it as push as pin
20:21cushion distorted. So if you look at
20:24this image through the lens, you put
20:26lens over it, you will perceive it as
20:28this. Therefore, if you want to correct
20:31for it, you need to introduce
20:34this barrel distortion. Then looking
20:36through this uh looking at this
20:39predistorted barrel distorted grid
20:43through the lens, you will perceive it
20:46as straight. So essentially the lens
20:50will shrink the grid. And if you want to
20:53make it appear straight, you need to pre
20:56fatten it. Then it will you will see
20:59with your eyes um and it will look
21:02normal. And that's your exercise. You're
21:04doing that in your homework one. Okay,
21:08let's move on. So now let's talk about
21:11tracking hardware. We talked about
21:14displays, we talked about optics, now
21:16tracking hardware. So it is this arrow
21:20here, right? So once the sense organ
21:22changes its configuration
21:25um by interacting with the virtual world
21:28then the virtual world needs to react
21:31appropriately or move right because if
21:33there is no tracking then it's kind of
21:35like looking under that virtual table.
21:37It will not it will move away but you
21:39want to track your body in it so that it
21:41moves closer. So the virtual world needs
21:43to react to your motions or the motions
21:46of your sense organ. So this process
21:49is called or tracking is the process
21:52that um updates the artificial
21:56stimulation according to the
21:58configuration change of the sense that's
22:00being tracked and as we already saw
22:03there are a lot of uh different um
22:05tracking systems. So um and a lot that
22:09can be done with tracking. So the closer
22:11the display to the sensor organ, the
22:13more tracking needs to be done. And you
22:15can implement 3° of freedom like in
22:17cardboard, six degrees of freedom like
22:19in most HMDs that we have today. You can
22:22do pupil tracking that is done in vario
22:25headsets. And then you can do what's
22:27called fiated rendering. So a little uh
22:30more advanced rendering. You can do
22:32hands, body, and other um other friends
22:35or family or pets or robots in your
22:38world tracking so that you bring more
22:41and more people or objects into your
22:43world. But that is harder. So even hands
22:46tracking is not as advanced as we would
22:48hope it would be at this point, right?
22:51If you have controllers, that's easier
22:53to track controllers, but hand tracking
22:55is not as reliable yet.
22:58So now the mo most important component
23:02in tracking systems in modern H&M is
23:05IMU. So what is IMU? Can you tell me?
23:07Inertial measurement unit. Very good.
23:15[Music]

About this transcript

This page contains the full transcript of Bird's Eye View_ Hardware - Part 2 by NPTEL-NOC IITM, generated from the public captions YouTube serves with the video. The transcript has 3,468 words across 523 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.