YT YouTube2Text

YouTube2Text → Transcripts → Bird's Eye View_ Software and Human Perception - Part 1

Bird's Eye View_ Software and Human Perception - Part 1 — Transcript

by NPTEL-NOC IITM · 3,332 words · 490 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:01[Music]
0:16of the bird's eye view series of
0:18lectures in which we overview all of the
0:20components involved in VR systems. Last
0:23time we talked about the hardware and
0:25today we will talk about software and
0:27human perception. And on Friday we'll
0:29already start uh going in topics in
0:32depth and we'll start with 3D modeling.
0:35So today we are finishing chapter 2.
0:38We'll talk about software and human
0:39perception and next week we'll start on
0:41chapter 3. So then let's start with the
0:43review question. I have a a review quiz.
0:47So what is the number of degrees of
0:49freedom that is strat in in an Oculus
0:53Quest 2 or a similar high-end HMD?
1:05So the correct answer is six. That's
1:08true. Then um of course there are some
1:11high-end HMDs that have eyetracking as
1:14well. So that would be eight. But we are
1:17talking about Oculus Quest 2 or a
1:19similar which is the correct answer is
1:21six. Very good. So now what is the
1:23number of degrees of freedom that is
1:25tracked by your phones when you put them
1:28in the VR cardboard of your
1:31so it's quite different from high-end
1:32HMDs and yes it's just three right and
1:36you probably already noticed because
1:38many of you have tried homework two part
1:41two experiences that we provided for you
1:43something fundamental is missing there
1:45right so the three degrees of freedom
1:46for positional tracking they're just not
1:48there tracked and that means that you're
1:52constrained in some ways. You cannot
1:54experience the world the way you would
1:55experience in an Oculus Quest 2. And um
1:59we have to do something with that. But
2:02we'll leave it for later. Now from last
2:05lecture, if you remember, what does
2:07accelerometer measure?
2:09So your phones or highend HMDs, all of
2:13those displays, all of those uh have uh
2:16an IMU board which has an accelerometer.
2:19An accelerometer is essential for
2:21tracking.
2:23What does it actually provide?
2:26Oh, sorry. This question is what does it
2:27measure?
2:34So do remember the mass on the spring.
2:36What does it measure? Like when does the
2:38spring stand or contract
2:42after you apply what kind of forces? And
2:45the correct answer is linear
2:46acceleration due to
2:48forces acting on accelerometer angular
2:52linear.
2:57Okay. The correct answer is not even
2:59there. Right? Do you see that? So um the
3:02correct answer that it measures all and
3:05you have to check all of that they
3:07apply. Okay. Right. Right. So this is
3:09not the multiple choice question. This
3:10is like you have to select all that
3:12apply. So which one? Linear acceleration
3:14due to gravity. Yes, it does measure.
3:17Let's go back. Uh linear acceleration
3:19due to other forces acting
3:20accelerometer. Yes, it measures that
3:22linear acceleration due to rotations.
3:24Yes, it does. When the when the objects
3:27rotate, when our head rotates, there is
3:28a centrial force and that acceleration
3:31is also measured. Angular accelerations,
3:34no. Accelerometer does not measure
3:36angular acceleration. So, this one
3:38shouldn't be there. And it doesn't
3:40measure linear velocity due to
3:41translations. No. So these two are not
3:44measured by accelerations, but these
3:46three are all measured. And the result,
3:49the resulting measurements returned by
3:51an accelerometer is the vector sum of
3:54all of those three.
3:56So here is our abstract picture of a VR
3:59system and let's review it. So that's
4:02from last time. So we will name again
4:04all of the block scenarios in this
4:06abstract picture of virtual reality. So
4:08the virtual world generator
4:11or CPU GPU um through the process of
4:14rendering constructs the virtual world
4:16on the display and provides the so the
4:19display will provide the artificial
4:21sensory stimulation to a sense organ in
4:24this case the eye but other sensory
4:26organs as well that um then transduces
4:30it into a neural signal and sends it to
4:32the brain. So the brain is hopefully
4:34fooled and in response the brain can
4:36control the sense organ to see closer
4:39maybe things you know to move it in
4:41space in in physical space and in
4:43correspondingly in the virtual space. So
4:45how does that happen? This causes the
4:47configuration change of the display
4:49itself which should trigger the event in
4:52the tracking system that causes CPU to
4:55update or rerender the display
4:57appropriately so that the virtual world
4:59behaves the same way as the real world
5:01would behave and um the organism would
5:05enjoy the experience right and then the
5:08cycle will repeat. So now let's zoom in
5:11onto the virtual world generator and see
5:14what is the input to it and what is the
5:16output. So let's follow the arrows
5:18coming in and out.
5:21So for the vision the arrow coming in is
5:24from the head tracking hardware. So the
5:27IMUs that we talked about and cameras
5:29and in case of cardboards it will be
5:31just RMUs from on your phones and coming
5:34out is the rendering. So on the visual
5:38displays right. So this smartphone
5:40screen screen screen in this case.
5:43Uh of course there are other hardware uh
5:47displays and other tracking hardware as
5:50well but for the vision sense this is
5:52mostly what we have in our uh VR
5:55headsets VR systems. So now let's look
5:59at the bigger picture for the software.
6:02So we are in the bird's eye view of the
6:04software. So let's look at this big
6:05picture. What does the virtual world
6:07generator do? So the input to the
6:09virtual world generator comes from all
6:12of the tracking hardware, not just I'm
6:13using cameras like for vision, but also
6:16keyboard, mouse, game controllers,
6:18haptic suits, everything that we we want
6:20to put in in this VR system. And the
6:23output from the virtual world generator
6:25is to render all of the displays then.
6:28So the visual display, the haptic
6:29display, the audio display to produce
6:32the appropriate signal to come to the
6:34sense organs. then to hopefully fool the
6:37brain. So uh the job of the virtual
6:40world generator or the software
6:42component of VR systems is to maintain
6:44this fictitious world virtual reality
6:47world which means that it will have to
6:49maintain a lot of physics and geometry
6:50and maybe other laws in such a way that
6:53the world feels immersive enough right
6:56so you enjoy experiencing it. So what is
6:59the software challenge then in
7:01maintaining a proper geometry of objects
7:04in virtual world. So there are two types
7:06of objects, moving objects and
7:08stationary objects. And um so those that
7:12move with respect to the world itself
7:15such as this rotating box on the screen
7:18and more of course there will maybe
7:19people walking on streets in your
7:21virtual world, the hands and legs,
7:24leaves blowing, clouds floating. So all
7:26all of that and the virtual world
7:28generator will have to maintain proper
7:31geometry of those objects and render
7:34them properly on the screen. So that's
7:36good. Another type is stationary
7:39objects. So that's a little
7:41counterintuitive because stationary
7:44objects don't really move with respect
7:46to the virtual world but they are
7:48constantly moving with respect to your
7:50senses. Right? So the rocks here or the
7:53ground they will appear stationary in in
7:57when you experience this virtual reality
7:59world but they actually constantly
8:01moving with respect to your senses and
8:04therefore they are constantly moving on
8:06the displays themselves. They have to be
8:08so this requires a lot of math and we'll
8:11start on that on Friday. So you'll learn
8:13how to model these objects and how what
8:16what is 3D modeling that goes into this
8:18to render them appropriately. So you
8:21could write your own renderer maybe if
8:22you want to bypass Unity which is very
8:25expensive for our cardboards which is
8:27very interesting. So here is another way
8:29to look at it. So imagine that we are
8:32superimposing a virtual world and the
8:35local physical world that you're
8:38actually in. So maybe there is some
8:40furniture walls maybe robots or animals
8:43moving around in the virtual world. So
8:45there is this kind of story right that
8:47is being maintained by the software and
8:50uh fulling your brain. Let me apologize
8:52for my phone that is so loud.
8:55Okay. And um so while you are in the
9:00virtual in the real world and there's
9:02also furniture and maybe pets and other
9:05people right so somewhere in this
9:07virtual world then you would appear in
9:10in your virtual world and you have your
9:12own region there where you are able to
9:15move with respect to your real world but
9:18it can it is embedded in the virtual
9:21world right and your body can be trapped
9:23in that part of virtual world. So your
9:26eyes are maybe looking this way and you
9:28maybe have some avatars. I'm not or
9:30maybe not right in cardboards we don't
9:32have avatars but as you start moving in
9:36this virtual world think of this region
9:38here like moving your head and um and
9:42maybe body think of this region in the
9:45virtual world in the re in the real
9:46world this yellow zone which is an
9:49overlay between the local real world and
9:52the virtual world. So let's think about
9:53what happens with respect to this
9:55overlay of the two worlds. So as you
9:57start moving in the virtual world in the
10:00real physical world, you may be sitting
10:03or maybe you are standing and moving
10:05like this lady here and there is
10:08furniture and obstacles in the real
10:09world. So let's think about how should
10:12these obstacles be dealt with, right? So
10:15should furniture in the superimposed
10:17parts be identical or different? Right?
10:20What about the physics of the world? So
10:22how do you deal with mismatches?
10:25So if you reach out or bend in the in
10:28the virtual world, what if you not some
10:31furniture in the real world like in this
10:33video, right? So what should happen? So
10:35is it even preventable or not?
10:39And that's a design choices. So the
10:41thing is for cardboards, we don't have
10:43any support for this. So, you will
10:45probably knock off real furniture if you
10:47move in real world, although you're
10:48exploring your virtual world and there's
10:50nothing you can do about it because
10:52there's not enough support. In high-end
10:54HMDs, they actually explicitly define
10:56this overlay regions now so that you
10:58make sure you don't knock anything over,
11:00but you still can if you're not careful.
11:03So, uh that's what another this is part
11:07of what uh virtual world generator
11:09software needs to handle.
11:11So also interesting things will happen
11:13with respect to the obstacles in the
11:16virtual world, right? So what if you're
11:18moving your head in the real world and
11:20then all of a sudden there is a wall or
11:23furniture, right? And you decide you
11:25move your head towards it or maybe even
11:28through it. What should happen? So if
11:30you had haptic feedback, then you would
11:33feel the wall, right? So you would be
11:35hit back by it by this virtual wall. But
11:38what if you don't? So maybe you want to
11:40go straight through the wall because
11:42that's the physics of your virtual
11:44world. So these are again design choices
11:47and they should be ba based on whatever
11:49you want to do the task or things such
11:52as comfort. Um so maybe it's comfortable
11:55to go through the walls maybe it's not
11:57right. So again design choices something
11:59that goes in the software. So also
12:02virtual world generator maintains self
12:05motions. So there are two kinds of
12:07selfotions. If the user moves in real
12:10world and the motion is matched one for
12:12one or being tracked one for one in the
12:15virtual world. So just like in this
12:17video was right. So then if you turn
12:20your head in the real world the same
12:22turn will happen in the virtual world if
12:24you're being trapped properly right and
12:26the two motions match perfectly. So if
12:30your software is such that it provides
12:33zero latency and no delay
12:37then this is a very comfortable type of
12:40motion and is very convincing. So you
12:43this is great types of self motion and
12:46um the other motion though is not. So
12:48now imagine that you're instead sitting
12:51uh taking a controller and moving your
12:54avatar with the joysticks right
12:56joystick. So this moves your virtual
12:58body through the virtual space
13:00somewhere. So it would correspond to
13:02moving this yellow rectangle then by dis
13:06by some distance using a controller
13:08while you're actually sitting in the
13:09chair. So your motion is artificial in
13:12this case and does not happen in real
13:14world and you're trying to convince your
13:16brain that you're actually accelerating
13:18and reaching some velocity to get to
13:20some new place while in reality you're
13:22not. So this is where the discomfort
13:25comes from. and the VR sickness is a
13:29problem. So some games use teleportation
13:32to prevent nausea but that can lead to
13:36disorientation.
13:37So a lot of research is going on how to
13:40make locomotion comfortable in virtual
13:43reality for everyone. So I have a
13:46question for you. Can we have locomotion
13:51in VR cardboards when we don't have
13:53controllers and we have only three
13:56degrees of freedom?
14:06No. Yes. I guess those who say no, they
14:09didn't try Shadows of Truth detective
14:12game in homework too, right? Because
14:14over there they show how to implement
14:17locomotion using only
14:20well using what? So that's my next
14:23question. I don't want to say what it
14:25what it is but how would you implement
14:28locomotion in VR cardboards?
14:34So how do they do that in Shadows of
14:36Truth?
14:38I gaze, right? So you stare at some
14:41point for a certain amount of time like
14:43one or two seconds and then you have
14:46this little uh clock. Once you stare it
14:50for a long enough time, you can teleport
14:52to another place. Eye gaze. Eye gaze.
14:54Yeah. Gy accelerometer. No, no.
14:58Gyro accelerometer. They track your head
15:00motions. So they need to they they used
15:03for tracking you and the world reacting
15:06appropriately. But you cannot really
15:10they cannot detect your motion. So they
15:13can attract degrees of freedom. Okay. So
15:15eye gaze is the way to do it and it's a
15:18very annoying kind of thing. So we'll
15:20talk about it in homework 4. So remember
15:22homework 4 is where we will implement
15:24locomotion in your cardboard experiences
15:27in Unity and it will not be eye gaze. So
15:30it will be it's a secret for now. Well,
15:34it's not a secret anymore. You can look
15:35in homework 4 and see how to do it.
15:39And um so what we ideally so let's uh
15:43kind of summarize what software does and
15:46what we ideally would want to have
15:49to u easily create VR environments and
15:52that would be great software is to have
15:54a VR engine. So how do we create VR
15:58environments? Right? So at this point
16:01what Mark Zuckerberg promised
16:04was that we will have at some point VR
16:06metaverse. Right. So building VR
16:09experiences will be as easy as creating
16:12a Facebook page which will be really
16:13nice. But until then what we are
16:16leveraging in in your exercises too in
16:19this class is the existence of game
16:21engines such as Unity 3D or Unreal
16:23Engine or there is Gadot. Now there is
16:26cute. So some engines that make things
16:29easier for you. So the Unreal uh and
16:32Unity 3D they got much better for VR in
16:36recent years because they got optimized
16:38for a lot of things to get VR friendly.
16:41So what these engines then provide for
16:43free is that you don't have to implement
16:44your own textures, you know, your
16:46lighting, collision detection is
16:48implemented for you and much more. So
16:51you can use it and all of this is given
16:53for you for free. But then you need a
16:56powerful CPU to use them, right? So
16:58because they were originally built for
17:00gaming and not necessarily for VR and
17:03definitely not for VR on cardboards. So
17:06what are other options? We don't really
17:08have many other options but there's some
17:10I guess which I should mention. So we
17:13can what about panoramas right like
17:16Google street views and others. So there
17:18are so many panoramas out there. So
17:20maybe we could create a VR experience.
17:23So unfortunately the li there are
17:25limited options
17:28uh for for them. So Unity will allow you
17:31to use panoramas and place them as
17:33spheres in your Unity experience but
17:34then you're using Unity again, right? Um
17:37and you would need powerful PCs then for
17:39viewing panoramas. We that's what we
17:41want to avoid. So uh what if you just
17:44wrote your own 3D image viewer in VR? So
17:48that would be amazing because then you
17:50could just watch them on watch these
17:52panoramas or experience these panoramas
17:54in on the phone. But then you have to
17:57implement a lot of things like UIs and
17:59locomotion by yourself if you don't use
18:01Unity anymore. But I think this should
18:04be relatively simple. I I I I hope to
18:07see one of the VR projects using
18:10panoramas somehow in a very smart way.
18:13So what if now you stick a panoramic
18:17camera on a robot? So you want to create
18:19a very experience that way, right? You
18:23experience the world through the ro
18:24through the eyes of something that's
18:26moving like a robot or a drone. So
18:28that's a cool idea. There's a lot of
18:30research going on that there's startups
18:32doing that. But that's not trivial at
18:34all because then you need to do a lot of
18:37things from scratch um like video
18:40processing, right? So you would need to
18:42use some kind of video processing
18:44software to maintain this panoramic
18:46representation while you're also you
18:49issuing controls of where to go and how
18:51to move around. So this is a very
18:53interesting research topic. But there's
18:55also a problem with uh discomfort due to
18:58vection that is generated by this moving
19:00panoramas. So u our research is actually
19:02working on this. So it's not trivial at
19:04all and that's why we are stuck with
19:07Unity 3D in this course.
19:11Okay, so we are done with the software
19:13overview and now let's talk about the
19:15last part of VR system which is the
19:17organism or a human in our case. So when
19:20we talk about the role of a human in VR
19:23systems, we need to talk about
19:25physiology, neuroscience and perceptual
19:28psychology. So what is perceptual
19:30psychology? It's a discipline that
19:32studies how the senses work and how they
19:35get interpreted by the brain. So that's
19:38why it's so extremely relevant and
19:40important for VR systems and we need to
19:42study it because human is a part of the
19:45VR system just like you saw in
19:47calibration homework right so it's human
19:50perception that ultimately determines
19:53whether or not what you see in VR is
19:55good that's why we really need to study
19:58this
20:00so here what Murphus describes in the
20:03movie the matrix is a perfect V system.
20:07So, can we build such a VR system with
20:10say Oculus Quest 2? Probably not, right?
20:13I mean, definitely not. Definitely not
20:15with VR cardboards either. But
20:18nevertheless, let's see what Murphus has
20:20to say because he describes how it
20:22should be done to be perfect. So,
20:25it is all around us, even now in this
20:29very room. You can see it when you look
20:31out your window or when you turn on your
20:34television.
20:35You can feel it when you go to work,
20:39when you go to church,
20:41when you pay your taxes.
20:45It is the world that has been pulled
20:47over your eyes to blind you from the
20:49truth.
20:52What truth?
20:54[Music]
20:57That you are a slave, Neo. Like everyone
21:01else, you were born into bondage. Born
21:03into a prison that you cannot smell or
21:06taste or touch. A prison for your mind.
21:12Okay.
21:16[Music]

About this transcript

This page contains the full transcript of Bird's Eye View_ Software and Human Perception - Part 1 by NPTEL-NOC IITM, generated from the public captions YouTube serves with the video. The transcript has 3,332 words across 490 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.