YT YouTube2Text

YouTube2Text → Transcripts → INTRO TO AR / VR / MR / XR : TECHNOLOGIES , APPLICATIONS & ISSUES | MODULE - 1

INTRO TO AR / VR / MR / XR : TECHNOLOGIES , APPLICATIONS & ISSUES | MODULE - 1 — Transcript

by LEARN FROM EXPERT · 80,197 words · 11,202 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:06hello there uh I am Mich nebling uh I'm
0:10your instructor I am a professor an
0:14assistant professor to be precise at the
0:16University of Michigan where I work in
0:18the school of information um I thought I
0:21should introduce myself and tell you
0:23maybe a few things about myself before
0:24we go really into this XR Muk
0:27specialization um so I I've been working
0:30in the space for about 5 years now I
0:32would say uh I've been doing research in
0:35this space specifically in the area of
0:37human computer interaction and I've been
0:39teaching uh in the residential program
0:41at the University of Michigan for three
0:44years now um two arvr courses one
0:47focused really on design and one focused
0:49on development this is in many ways an
0:51update of the teach out uh that I've
0:54have done a few years ago with a few of
0:55my colleagues it was the augmented
0:58reality virtual reality and mixed
0:59realties teach out from the University
1:01of Michigan maybe you've seen me there H
1:03it was great to meet some of the
1:05Learners in that uh moo and uh many of
1:09you have actually said we wanted to
1:11learn more about design and development
1:13so we've heard you and we've added a
1:15significant content in both of these
1:18areas right so I care about XR you guess
1:23that um it is really important for me
1:26that we all understand these
1:27Technologies they are making it into our
1:30everyday lives um they are there they're
1:32not going to go away uh we still have an
1:35influence on how to shape those
1:37Technologies so that we have a better
1:40control right how these Technologies
1:43make it into our everyday lives in any
1:45case it's time for me to pay back so and
1:48pay it forward so I hope what you learn
1:51here today uh is that is something that
1:53I had to learn uh myself not just today
1:56but over the next few weeks it's not
1:57going to be that fast almost trick do
2:00there right and um I hope that you share
2:03that knowledge and that you invite other
2:05people to also learn about these things
2:07not everybody cares about development
2:09maybe some of us care about design
2:10hopefully even more and then a lot of us
2:13should really care about where this
2:15space and these Technologies are going
2:17and we should have some critical
2:19discussions about this so thanks for uh
2:23well letting me meet you and joining me
2:25in this uh effort of spreading the
2:27knowledge around XR and um yeah I really
2:32look forward to seeing uh you in the
2:34course hope we get to interact with each
2:35other maybe perhaps even meet in person
2:37one day that would be exciting um hello
2:41and welcome to the XR
2:56Muk welcome to this first course uh as
3:00part of the XR Muk this course is
3:02focused on providing an introduction
3:05really uh covering all the fundamental
3:07knowledge that we need to have to
3:09understand
3:10XR uh so this is really what this course
3:12is about uh understanding this
3:15terminology learning about the
3:17Technologies learning about the
3:18applications seeing the potential um
3:21maybe being a little bit enthusiastic
3:23about it and then learning about you
3:25know strategy and Trends and also then
3:28really talking about the issues
3:30Let's uh jump in and try to get a little
3:32bit of an overview of what we actually
3:34learn about in this uh first course so
3:36here's an overview of the course in the
3:39first module we're going to look at the
3:41terminology and also learn about the
3:42applications of XR then we're going to
3:45go deeper actually in each of these
3:47Technologies so we learn more about
3:49virtual reality and also augmented
3:51reality so after providing this
3:53fundamental understanding of some of the
3:55VR and AR Concepts and Technologies
3:57we're going to do a little bit of a an
3:59Outlook into where things are going with
4:01XR um so we we're talking about some
4:04trends that we can identify so after
4:07we've looked into some of the trends
4:08we're also going to talk about a variety
4:10of issues they're actually quite a bit
4:11so we're going to switch gears and uh
4:15from enthusiastic to critical and we're
4:17going to think about some of these
4:19Technologies more critically in the last
4:21module we're going to talk about
4:22strategy we're thinking more about like
4:25how to learn more about XR uh how to
4:29grow knowledge in the team how to
4:31actually grow the team uh what what
4:33should we think about when we acquire uh
4:35new equipment should we always go for
4:37the latest and greatest or broad or deep
4:40like do you buy a million times the same
4:42device no you don't so these are
4:44essentially the lectures like some of
4:46the main content I will convey through
4:48videos and through some panels uh
4:51bringing in quite a few people even in
4:53this first course but then also in the
4:54other
4:55courses uh in terms of how we assess
4:57learning well you'll you'll read the
4:59details later but there will be quizzes
5:02after every module there's a quiz so
5:04this is a well a mechanism for us to
5:07make sure that we are engaging with the
5:09content critically what you will also
5:11see is that there is actually an on us
5:13track so there's additional content
5:15especially some activities and exercises
5:18that I think uh will require more of
5:20your time but also allow you to engage
5:22much more critically with the content
5:24and so actually go deeper and develop
5:26your knowledge further and so I think
5:28the Honest track is really a great
5:30opportunity so and as part of this
5:32honors track we actually have then a
5:34number of activities now I'm going to
5:35give you a little bit of an overview of
5:37what you can expect to learn uh
5:39throughout this course with a little bit
5:40more detail so really you learn about
5:43terminology and applications you learn
5:45about Concepts and Technologies and you
5:48learn about the trends and issues I
5:50think of these as three fundamental
5:52pillars of knowledge uh this is really
5:54something where I want you to be
5:56familiar with each of these areas and
5:59and know a little bit about so that you
6:01can actually demonstrate knowledge
6:04around XR and also shape uh where XR is
6:08going so I said we learn about terms
6:11right so here I list out what virtual
6:12reality is augmented reality mixed
6:15reality and extended reality we're going
6:17to learn about this obviously in a
6:19lecture uh and one is going to be a
6:22little bit more traditional and then one
6:23is actually using XR Technologies so
6:26here you're seeing me sketching in a
6:29tool called tiltbrush I'm actually in VR
6:32and I'm sketching out uh the reality
6:34virtuality Continuum so after learning
6:36about these key terms and really
6:39understanding some of the fundamental
6:40concepts we're going to look at the XR
6:42technology landscape we're going to look
6:44at four things really in this landscape
6:46the devices that are out there the
6:48platforms the applications and the tools
6:52you're going to learn about some of the
6:54applications like you're going to see me
6:56play Beat saer and obviously there was a
6:58learning goal behind this what we learn
7:00here is more about VR so on the AR side
7:04of things one thing that you may have
7:05heard about is Pokémon go it actually
7:07popularized AR a few years ago so these
7:11are some applications and they were more
7:14in the space of gaming frankly but this
7:17is really where a lot of movement is
7:18right now in the XR space but we're also
7:21making it slowly at least into
7:22productivity you learn about some of the
7:25tools um but you learn much more about
7:28tools in the course is focused on uh
7:31design and development so in this one
7:33I'll talk a little bit about immersive
7:35authoring and I give a little bit of
7:37that overview um here I'm just editing a
7:403DC on an iPad but then I'm actually
7:41jumping into AR and I'm continuing the
7:44authoring right in the studio uh using
7:47the iPad here what I'm using here is
7:50actually Apple reality composer that's a
7:53tool that's available to you as well if
7:55you have an iPad so this was actually
7:57just a summary of what happens in in the
8:00first week so you you you'll see we will
8:02learn actually quite a bit and we learn
8:04through examples we then actually also
8:07you know develop a little bit more
8:08knowledge in on the concept sides of
8:10things and then go going deeper into
8:12Technologies so I have like uh two
8:15modules uh they're divided into AR and
8:18VR and then each into Concepts and
8:19Technologies and we are going to ask
8:21some of the basic questions around AR
8:24and VR uh the key Concepts you're going
8:27to learn about the Technologies with
8:29lots of examples and also you will see
8:31demos throughout so as we learn about
8:34some of these Concepts and obviously the
8:37Technologies in AR and VR I want us to
8:39develop a mental model that we can use
8:42to answer some of these questions so
8:44when you engage in a new project you're
8:46going to start some kind of new XR
8:48project or initiative should it be
8:51around AR or VR should it be both should
8:54it be neither of them um what type of AR
8:58we are should you be doing um this is
9:01more for projects or what are the
9:03display and tracking Technologies and
9:05these are some of the decisions you can
9:07make at that stage because you will have
9:08that knowledge uh which types of
9:10platforms and devices are out there and
9:12which ones fit the best uh the
9:15requirements that we have and the needs
9:16of our users and then which tools and um
9:21actually design tools and development
9:22tools should we be using throughout the
9:24process so these are the questions that
9:26you ask yourself at the beginning of the
9:28project and that I want you to be able
9:29to answer at the end of the course we're
9:32going to learn about this XR technology
9:34tree and then we're going to learn about
9:36how to you know turn this technology
9:38tree around uh because what's important
9:40is to know how to ask the right
9:42questions we're going to talk about
9:44Trends in XR I said we're going to use a
9:46human computer interaction lens so we
9:48think about design and we're thinking
9:50about these three angles from people or
9:52users from tasks and technology and then
9:55we're going to talk about issues
9:57covid-19 really has been a major major
10:01disaster in many ways and it actually
10:03had quite an impact on the XR uh
10:06landscape at least from a needs or
10:08desires perspective a lot more interest
10:10in XR we're going to talk about three
10:12other classes of issues uh around social
10:16and ethical concerns around
10:17accessibility and equity and around
10:20privacy and security finally as I said
10:22we're going to look into the strategy
10:24right we're going to look at the
10:25knowledge equipment team and users and
10:30how to grow each of these really how to
10:32have this mindset of growth and what are
10:35some strategies uh to for example
10:38increase uh the knowledge on your team
10:40or make your team stronger in the XR
10:42space or actually well educating more
10:45people and being an evangelist of these
10:48Technologies so that eventually there
10:49will be more users so basically at the
10:52end of this course you will have the
10:54fundamental knowledge uh and also have
10:59well at least some language and um
11:02insights into strategy and can
11:05contribute to discussions around shaping
11:06XR really knowing about the trends the
11:09issues and as I say uh strategy so this
11:13is an overview of the first course I
11:16really packed quite a bit in it and I
11:18really wanted to look at XR from all
11:19these different angles and perspectives
11:22this course is really uh something that
11:26I'd like everybody to take independent
11:27of whether you are interested in Des
11:29design or development you just want to
11:31have a rough understanding of these
11:33Technologies you want to be able to
11:34participate in discussions you want to
11:36contribute meaningfully uh to how some
11:39of these Technologies are shaped and as
11:42you can probably tell I'm really
11:43passionate about these things and I want
11:45everybody to know about it so that we
11:47can have actually critical discussions
11:49around some of these issues and the more
11:51people know about this the better we are
11:53prepared for this future where XR I
11:56think will be part of our everyday life
12:12what is XR a question that I'm often ask
12:16so this is an intro to XR and I want you
12:18to learn about the main terms and
12:20Technologies in this
12:22segment so the terms that you are
12:24probably most familiar with at this
12:25stage is virtual reality and augmented
12:28reality we often say arvr but there are
12:31actually very different Technologies
12:33associated with these terms so when you
12:36say virtual reality I always think of
12:39this idea of replacing the reality
12:41around the user we do this by providing
12:43a computer generated virtual
12:46environment so there is no more real
12:48world view it is only the virtual view
12:52augmented reality on the other hand
12:54enhances reality so the user actually
12:57sees both the real world and the virtual
12:59world they are like merged together so
13:02the real and virtual we call this a
13:04composite view now what is interesting
13:07is that the r in VR is actually not the
13:11real world it is a synthetic a computer
13:13generated world and the r in AR actually
13:16refers to our physical real world around
13:19us so let's focus on virtual reality for
13:21a moment virtual reality is defined in
13:25terms of three main characteristics the
13:27first being that virtual reality
13:30emphasizes autonomy and agency so the
13:33user at all times stays in control we
13:37achiev this through head tracking and
13:39using the body as input so if I show you
13:42an example using a VR headset when I put
13:46this on my head the device is being
13:49tracked in 3D
13:51space and I can use my body as input now
13:56the way this often works is through
13:59virtual reality
14:01controllers but what is increasingly
14:03popular possible as well is to just use
14:07your hands as
14:09input in addition to your hands you can
14:13also use speech as input so this is
14:17actually leading us directly to the
14:19second main characteristic that virtual
14:20reality provides natural interaction
14:23using gestures with a controller hand or
14:27fingers and speech as input the third
14:31characteristic of virtual reality is
14:33that virtual reality gives us a sense of
14:35presence the feeling of being there so
14:39as we enter the virtual
14:42world we are being immersed this
14:46immersion is happening by stimulating
14:49multiple of our senses the visual any
14:53kind of audio as well as with more
14:56advanced VR Technologies you will have
15:00uh forms of haptic feedback and some
15:04other ideas including even all Factory
15:07displays which is you smelling whatever
15:10the virtual environment is around you
15:13great now let's focus on augmented
15:15reality augmented reality is also
15:17defined in terms of three main
15:19characteristics the first being that
15:21augmented reality combines real and
15:23virtual objects to produce a composite
15:26view so it's not just visual objects it
15:29can also be audio information
15:32overlaid let me give you an example if I
15:35put on this augmented reality
15:41headset I can still see the real world
15:44around me and any kind of virtual
15:47content is overlaid over that real
15:50world so the second main characteristic
15:54of augmented reality is that AR is
15:56interactive in real time it supports
16:00both explicit and implicit interaction
16:03by explicit interaction I mean the
16:05similar styles of interactions that we
16:07have described for virtual reality using
16:09gestures and speech as forms of explicit
16:13interaction augmented reality also has a
16:16lot of implicit camera based interaction
16:19as I move the device around the device
16:22has to make sure that the content the
16:24virtual content is displayed in a way so
16:27that it looks like it is part of the
16:29real world this brings us to the third
16:33characteristic of augmented reality AR
16:36is actually registered in
16:383D we need to do this in order to align
16:41the real and virtual
16:43objects spatially in the physical world
16:47they appear as if they were part of the
16:49same world so these are the three main
16:53characteristics of
16:55augmented
16:57reality another term that you have
16:59probably come across is mixed reality
17:01mixed reality is a term that often
17:03causes confusion in fact talking to
17:06experts researchers and designers we
17:08have identified at least six working
17:11definitions the reality virtuity
17:13Continuum as defined by mgram is one of
17:16the popular
17:17definitions mix reality as a synonym for
17:20augmented reality mix reality as a
17:23combination of AR and VR mixed reality
17:26as a stronger version of AR mixed
17:29reality as a type of
17:31collaboration and mixed reality as the
17:33idea of aligning physical virtual
17:36environments these are all valid
17:38definitions and I want to zoom in on one
17:41that is popular at least in the Le
17:43research Community now mixed reality in
17:46terms of the reality virtu reality
17:48Continuum is defined as follows it is
17:51the Spectrum from the Real Environment
17:55without any augmentation to the virtual
17:58environment which is completely
18:00synthesized computer generated virtual
18:03content now augmented reality is
18:05something we start talking about as we
18:07bring in Virtual content but mostly
18:09still see the physical world the Real
18:12Environment there is a term augmented
18:14virtuality which is not as popular but
18:17refers to the idea that we see mostly
18:20virtual content and bring in physical
18:23content so in some sense it is the
18:25reverse of augmented reality as we move
18:28closer towards the fully virtual
18:32environment mixed reality is a term that
18:35refers to a large part of that Spectrum
18:37it actually includes augmented reality
18:40as well as augmented virtuality in the
18:43original definition by mgrm it does
18:45however not include virtual reality so
18:48mixed reality is really this blending of
18:52the physical and the virtual world with
18:54varying degrees of
18:57augmentation now this this is the
18:59definition of mixed reality and what
19:01often helps me is to think about mixed
19:03reality in terms of
19:05displays so I'll show you a number of
19:07examples of displays here as they map to
19:10the
19:12Spectrum so closer towards the Real
19:15Environment where we have little
19:17augmentation we find the displays uh
19:20called tangible interactive displays or
19:23spatially projective
19:25displays now somewhere in this
19:27transition between a a to AV we can find
19:32tablets and smartphones that are
19:34producing augmented reality content the
19:37way they do this is by mostly showing
19:39the camera feed and then so that it
19:42creates a see-through display and then
19:44actually putting virtual content on top
19:46of it so we right in the middle of the
19:48spectrum the closer we move towards the
19:51virtual environment we can think of
19:53displays that are caves room size wall
19:58size
19:59displays around the user or a
20:02head-mounted display like the one I
20:04showed
20:05earlier one of the typical examples of a
20:08virtual reality
20:10headset so in this segment I've covered
20:13a few of the key terms VR or virtual
20:16reality which refers to the computer
20:18generated virtual world AR or augmented
20:22reality which refers to the idea of
20:23enhancing the physical world with
20:25virtual
20:27content I've also talked about Mr or
20:30mixed reality which according to
20:32definitions by mgram include augmented
20:35reality and augmented virtuality but
20:37usually not
20:40VR now XR is a term that is often used
20:43where X is just a wild card for
20:46augmented virtual or mixed reality
21:03so I wanted to teach you about virtual
21:06reality and how it relates to other
21:07Concepts while I am actually in virtual
21:10reality uh right now okay so if we were
21:13not in virtual reality we would call
21:15this the reality the real world okay and
21:18so I'm going to put on the other side of
21:20the spectrum if you will VR okay and so
21:24I want to tell you how we actually get
21:26from The Real World world to the virtual
21:30world the virtual reality that we often
21:32talk about in this course and I do this
21:34by introducing a few Concepts okay first
21:37we often going to talk about AR in this
21:40moo
21:42specialization and um AR is augmented
21:45reality is the idea of adding virtual
21:48content on top of the real world now you
21:50still mostly see the real world but you
21:52do add virtual content if we added a lot
21:55a lot of virtual content we would
21:56actually make it over to the point it's
21:58all virtual and this is like where I am
22:00right now everything in here is virtual
22:02and I can resize it rescale it I can
22:04walk through it can do all these things
22:07in virtual reality many of which we can
22:09also do obviously in the physical world
22:11it is still some form of reality so if
22:14you add all this we end up over at the
22:17virtuality side now there's one concept
22:19that not a lot of people are familiar
22:21with but it's called AV it's augmented
22:23virtuality and augmented virtuality is
22:25really this idea it's the inverse of AR
22:27it is mostly viral and we augment the
22:29viral with physical content so for
22:31example what you will probably notice is
22:33that obviously I do have physical hands
22:36but I don't see them in VR what I see
22:39instead is the controllers now these
22:40controllers are very good approximations
22:42of my hand movements and where they are
22:44in 3D space so you might call this a
22:46form of augmented virtuality where you
22:48actually bring the real world
22:50coordinates and to some sense also the
22:53pose and the motion of my hands into the
22:55virtual world and so again there are
22:59also versions of this where you actually
23:00do see virtual representations of your
23:02hands and now we actually do have hand
23:03tracking and uh can actually get very
23:06good at actually entering virtual
23:08reality to the extent that we may one
23:10day not be able to tell the difference
23:12between the virtual and the real world
23:14anymore now two more Concepts that are
23:16relevant for us first of all we usually
23:19call this entire space here mixed
23:22reality so Mr and it's really like a
23:24fancy term that has been used a lot in
23:27promotional videos as this kind of like
23:29new thing but to be honest mixed reality
23:32has been defined in 1994 in the re
23:34research literature at least uh by Paul
23:36mriam from the University of Toronto
23:39that was really based on a literature
23:40review of all the things we have been
23:41doing for more than uh a decade at the
23:44time right in 1994 so AR VR and Mr are
23:48really old Concepts now we call this Muk
23:53specialization I'm going to bring this
23:55back so I can actually draw this nicely
23:57we call it the X R moo okay and now how
24:01does XR really fit here into the picture
24:03well it actually refers to everything
24:06that is augmenting the real world
24:10including full virtual reality and the x
24:14is really just a wild card for AR VR or
24:17Mr and so that's how I'd like you to
24:20think about it uh some people refer to
24:22it as extended reality again it is
24:25actually not introducing any new kinds
24:27of interactions or any new ideas it is
24:29really just a concept to refer to this
24:31entire design space and that's how I'd
24:33like us to use the term where X is
24:35mostly the wild card again for AR VR or
24:40Mr pretty cool isn't it I think this is
24:43a really powerful tool for teaching and
24:46learning uh could imagine that in the
24:48future we actually do more and more
24:50lectures just in VR perhaps with more
24:53interesting content conceptually this
24:55was definitely interesting content but
24:57you could imagine that I actually start
24:59sketching out entire virtual spaces to
25:02describe architecture ideas or explain
25:05phys physical phenomena and this is a
25:07lot of the stuff that we'll talk about
25:09in the smuk specialization and I really
25:12look forward to actually welcoming you
25:14once this is going to launch and it's
25:16going to be cool
25:30in this video we're going to talk about
25:32the XR technology landscape so the
25:35landscape is actually quite big I'm just
25:37like going to reach out here to one of
25:39my examples that I'll show you it is the
25:42Hollens 2 now the Hollens 2 is one of
25:44the latest uh AR devices on the market
25:49so definitely a lot of changes around
25:52the device and comfort is good and the
25:56the visor has changed and some other
25:58things have changed but like how do we
26:01actually now assess these improvements
26:03and are they actually improvements and
26:06yes they are but I want to give you some
26:09tools so that you can better classify XR
26:12technology and because the landscape is
26:15so big we really need some of these
26:17tools so I'm going to actually Jump
26:19Right In and I'm going to start with a
26:22little bit of a complaint here and my
26:24complaint is that there is actually a
26:26lot of hype around
26:28AR and VR Technologies and we're not
26:30always very honest so there's often new
26:33terms for essentially the same
26:34technology we see over promising
26:37promotional videos all the time there is
26:40a proliferation of new devices with
26:42actually just minor differences uh in
26:45display resolution and field of view
26:47which are actually some of the key
26:49factors so the Holland 2 is a good
26:50example here in terms of resolution and
26:53field of view the improvements are
26:55actually quite minor and I have to say I
26:57understand it's technically really
26:59really challenging to improve and
27:01increase the field of view for example
27:04but um yeah let's just be honest about
27:07it many technical aspects are rapidly
27:10improving I'll grant you that so Hand
27:13finger and eye tracking are actually now
27:15becoming a standard and actually quite
27:17good already but we still have to make a
27:21lot more progress there are still strong
27:24Technical and design limitations for
27:27example Environmental understanding are
27:29still rather
27:31basic and by environmental understanding
27:34I mean if you were a computer and you
27:37were looking at me well would you be
27:39able to tell that I'm a human what kind
27:42of gestures I'm doing and where I am in
27:44the environment so the idea of this
27:47segment is to introduce you to four
27:49classes of XO technology yeah really
27:51help you think through that space the
27:53first class we're going to talk about is
27:55devices now devices well we can
27:59distinguish between Standalone or
28:01builtin kind of XR devices so devices
28:04that have built-in XR Technologies or a
28:06standalone they're like separate as an
28:08additional device you have to for
28:10example put on your
28:11head and we can think of them as
28:14tethered or adapter based devices so for
28:17example some devices wrap around your
28:19smartphone like the cardboard wraps
28:21around your smartphone or their Rift and
28:24the Vive they have to be connected to a
28:27computer so I'm going to show you two
28:29examples here on the built-in Standalone
28:32well we have the hollow lens 2 it is
28:36when it's charged it is a standalone
28:38device it's a fully self-contained
28:40computer yeah just goes on your head and
28:42off you go do a little bit of AR pretty
28:44cool um in the that's an AR device and
28:49then on the VR side of things I have the
28:53quest here now you're wondering why is
28:55the tethered well because it can and
28:57that gives it some improvements but it
28:59doesn't have to be tethered okay so this
29:02is also a standalone device um that just
29:05goes on your head and gives you VR now I
29:08said it can be tethered so this is where
29:10this classification starts to blend and
29:13blur a little bit because the quest
29:14actually with the Oculus link which is
29:16essentially just a USB cable that goes
29:19into the device and we can connect it to
29:21a computer and that actually allows us
29:25to well turn this Quest test into a rift
29:29s kind of capable device giving it more
29:33computing power and ability to render
29:35heavier Graphics if you will my other
29:37example is the Google cardboard here so
29:40the cardboard is actually an adapter and
29:45uh we call this also tethered it needs
29:48to be connected to something so the more
29:51traditional understanding of teed would
29:53be you connecting a device to your
29:56computer and I can do this with a quest
29:59and it actually can be changed it's a
30:01it's one of these hybrid devices between
30:04stand
30:06alone and tethered so I connected back
30:10all right this was just the first part
30:11of our little classification we're going
30:13to look at platforms next so when it
30:16comes to platforms we can distinguish
30:18between specific platforms so for
30:21example the Oculus the Vive or the magic
30:25leap and they connect to the the
30:28corresponding devices that I list here
30:30there are differences between just Ulus
30:32and Vive and so these are technical
30:35differences and so you could build
30:37specifically for these platforms or you
30:40can take a crossplatform approach so you
30:42can build on some technology like steam
30:44VR Windows mixed reality or web XR which
30:48allows you to bridge multiple platforms
30:51I want to introduce you to a third and a
30:53fourth class my third class is
30:54applications so some people refer to XR
30:57Technologies as applications and then I
30:59think we need to distinguish between XR
31:01apps like beatsaber or Snapchat and apps
31:06with are just mobile apps for example
31:08with specific XR views and modes so they
31:10allow you to go into VR or into AR so
31:13here I have two examples Ikea and Amazon
31:15shopping that actually now have built-in
31:17AR views so just at the Press of a
31:20button you can actually blend in the
31:23environment and explore things in AR so
31:26you can test whether the the furniture
31:28fits into your home where you can
31:30preview a product you're about to buy
31:32from Amazon my fourth class is tools and
31:35here I actually think a lot in terms of
31:37software tools but it's different from
31:40applications so there are tools for
31:42design I'll introduce you to some of
31:43those including tiltbrush quill and
31:46arrow so tiltbrush and quill are VR
31:48examples and arrow Adobe arrow and then
31:51Apple reality composer these are all
31:53design oriented AR examples and when it
31:57comes to development well we are going
31:59to distinguish a few key development
32:02tools here we are going to cover A-frame
32:05unity and unreal so this gives you a
32:08first important overview of four classes
32:10of Technologies uh we started with
32:12devices we looked into platforms and
32:15then we talked about applications and we
32:17talked about tools so now we have some
32:19basic language to look at the XR
32:22landscape and this will allow us to
32:24learn a little bit more as I go forward
32:27what you see and what you learn and what
32:29you notice uh when you watch more and
32:31more content of my courses here I'm
32:34someone who actually brings in all these
32:36devices tools platforms and applications
32:39because I think that's the best way for
32:41us to learn so I'm going to show you
32:43some of these in the following slides
32:45all right let's look at VR devices uh
32:48very quickly so here we just learned
32:50adapter so smartphone cardboard I just
32:53showed you and we talked about tether so
32:56the rift here the Oculus Rift with
33:00various sensors H could be plugged into
33:02a computer you can see all kinds of
33:03cables in fact you need three USB ports
33:06and one HDMI or display port depending
33:09on uh the configuration and then the
33:11quest is really a standalone VR device
33:14um that can however be tethered but all
33:16it takes is one USB cable and Standalone
33:21really for that to work we really had to
33:23make a lot of improvements you can see
33:25how the sensors these little black
33:27sticks in the middle picture how they
33:29have disappeared from the right picture
33:31and that means we had to make some
33:33technological advances and so this is
33:36the difference between outside in and
33:37Inside Out tracking in fact we were able
33:40to integrate the sensors into the
33:42headset looking at AR devices here you
33:45see the adapter smartphone so this is
33:48Holo kit so let me show you this very
33:50quickly so this is the Holo kit okay so
33:55it opens like the cardboard this is the
33:58AR version of the
34:01cardboard VR cardboard opaque AR
34:05see-through in fact I can actually now
34:08still see you and wave at you and I know
34:11all that because it is an augmented
34:14reality display my example in the middle
34:16here is a built-in AR technology device
34:21like the iPad that I show here and it
34:24actually has AR kit inside and I'll show
34:27you lots of examples with smartphone
34:30based and uh tablet or iPad based AR
34:33later and my last example here is
34:36Standalone an example of a standalone AR
34:38device and that is the Hollin actually
34:41the first version here but I showed you
34:43the Hollin to earlier so if we were to
34:46place these devices on the reality
34:48virtuality Continuum from The Real World
34:52to the completely computer generated
34:54virtual environment where would you
34:57place those devices so the Google
34:59cardboard I put towards clearly virtual
35:02environment and I put it almost to the
35:04right however I still think that there
35:06are more immersive VR devices like the
35:09Oculus quest for different reasons for
35:11example it supports 6 degrees of freedom
35:13and has two controllers and I can do a
35:16lot more in this device when it comes to
35:18AR I would put the hollow kit there
35:21somewhere between the real environment
35:23and augmented reality it does provide a
35:27relative
35:28simple form of AR so essentially just
35:31overlaying virtual content but uh the
35:35tracking is relatively limited so the
35:37holens I would put here I have an
35:39example of the Hollens one here I would
35:40put it uh clearly it falls into the AR
35:44Spectrum so in some sense it produces
35:46better augmented reality than the Holo
35:49kit obviously it sits on youread it has
35:52support for advanced interactions
35:54including voice based interaction but
35:56also the Holland 2 now also has eye
36:00tracking in Buu and hand tracking was
36:02already possible to some extent with
36:04Holland one so clearly more immersive
36:08more realistic augmented reality now
36:11where would you put the smartphone based
36:14AR well I'll put it here between AR and
36:17AV because technically what it does is
36:21it uses the camera to bring in the
36:24camera feed and then we can augment that
36:26view and so in some sense it really sits
36:30at the middle between real and virtal
36:33environment because you're looking at
36:35the mosty virtual content a camera feed
36:38that is augmented and you're seeing a
36:40lot of the real world around it as
36:42you're holding that device in front of
36:44you all right so now we have placed
36:46these Technologies on the reality
36:48virtuality Continuum so these were
36:50obviously just a few examples of AR and
36:52VR devices um there's a lot more to be
36:55found on the VR side so the rift the
36:58rift s the Vive and all these would
37:02somehow go here and I wouldn't actually
37:04draw too many differences between where
37:06they fall they just fall into this
37:08quadrant obviously on the AR side we
37:10also have more devices out there the
37:12magic leap is still kind of out there
37:15and it would fall into this spectrum
37:17there have been many other kinds of AR
37:19devices and VR devices that I don't
37:22include here I did include some of the
37:24popular examples and I realized that
37:27over the the next few years we will
37:28actually find new examples of arvr
37:30devices and it will be up to you to
37:33place them on the Spectrum so let's look
37:35at VR platforms and devices I talked
37:38about the aulus we have the rift the go
37:40the quest as examples here they have
37:43applications they can be accessed
37:45through the Oculus store there's
37:47actually a difference between each of
37:48these so they all have their own Oculus
37:50stores and we are only now beginning to
37:53see cross store application access so
37:56this is a little annoying but something
37:57to keep in mind for HTC Vive it's a
38:00similar story we have Vive and Vive Pro
38:04as examples here and applications are
38:07distributed through viveport that is the
38:09App
38:10Store on the Microsoft side of things we
38:13have Windows mixed reality as primarily
38:16VR hollowland also um falls into uh this
38:20Windows mixed reality Spectrum but uh
38:24let's just consider it separately and uh
38:27a virtual reality app based on some kind
38:29of Microsoft platform would go through
38:33the Windows store so as a developer it
38:36really makes a difference whether you're
38:37developing for aulus for HTC or for
38:40Microsoft and there are other examples
38:42as well so what are some ways to
38:45actually do crossplatform
38:47development here I show steam VR and
38:50webxr so steam VR is the VR plugin to
38:54the popular steam platform the steam
38:57platform form can be used to acquire
38:59both games and also applications and
39:02with a VR integration you can actually
39:04get applications that run on each of
39:06these Platforms in fact you can run them
39:08through Steam VR and then you can use an
39:11arus an HTC Vive or a Windows mixed
39:15reality VR device from Microsoft and you
39:19should be able to see it another
39:21important crossplatform approach is
39:24webxr web XR is really an evolving
39:27specification for the web that allows us
39:31to build web applications that can then
39:33actually be accessed through each of
39:35these devices so it's a very important
39:38way to reach across
39:41platforms looking at the AR side of
39:43things here we have on the smartphone
39:46side arcade and AR core they allow you
39:48to reach iPhone and iPad and then maybe
39:50in the future Apple glass and then AR
39:53core Android devices you can reach with
39:56that so AR core runs on a number of
39:58smartphones based on Android and then
40:01distribution of applications is through
40:03the Play Store on the Microsoft site
40:05we're talking about hollowland and again
40:07distribution is Through the Windows door
40:11now you notice that steam VR is gone we
40:13only have web XR as the primary way to
40:16actually distribute across all these
40:18kinds of platforms through the web
40:34welcome to the Honest track in the
40:36Honest track uh we're going to do a much
40:39more handson exploration of some of the
40:42topics um so usually what will happen is
40:45I ask you to do some kind of activity an
40:48actual task I always have a description
40:50of what the task is and uh what I think
40:54the expected results are of those tasks
40:56what are we learning from doing this why
40:58is it important why do I want you to
41:01engage in some of these activities in
41:03this course we don't actually have a
41:05peer review but in the other courses
41:08focused on design and development where
41:09you you will actually create a project
41:12uh we do have more of a peer review uh
41:14we understand and fully realize that the
41:16peer review is a major activity and I
41:20think in this course we can do without
41:21the peer review but that doesn't mean
41:23that you should not discuss the issues
41:25with your peers in fact you will
41:27actually ask you to do so but you will
41:29not be formally evaluated and assessed
41:32by your peers so that's a major
41:34difference to the other tracks uh in the
41:36other courses so in the first course
41:39you're currently here uh exploring
41:41terminology and
41:43applications and uh in the next two
41:46weeks we're going to go deeper in the
41:48concepts and Technologies of both VR and
41:50AR and then we're going to talk about
41:52the trends and issues and so this is
41:55really where the course then shifts uh
41:56more forward looking to the Future and
41:59really to the shaping uh parts of this
42:02course and so as part of this uh we'll
42:04ask you to do an app classification I
42:06introduce you to that learning about all
42:08the concepts and Technologies it's
42:10important for you to to understand and
42:12then also make some recommendations
42:15based on your current understanding of
42:16these Technologies so I'll give you a
42:17few scenarios to do this and then
42:20finally we also uncover a lot of
42:23important issues really key issues to XR
42:26and we want to have a nice and informed
42:28discussion about these issues finally at
42:31the end of this course we're going to do
42:33a little bit more exploration and talk
42:35about strategy and how to approach new
42:37EXA projects and initiatives and for
42:40that I wanted to leave a little bit more
42:42open-ended so in this Honor track in
42:44this course app classification Tech
42:46selection and key issues what does it
42:48entail so you're going to try out an XR
42:50app and then you're going to classify
42:52that app along the reality virtuality
42:54Continuum the second activity is really
42:56about text selection I'll give you a
42:58number of scenarios and you are the
43:01person telling me which Technologies I
43:03should be using and uh to really work on
43:06this U you are asked to perform what is
43:08called a qoc analysis A questions
43:11options and criteria analysis and I'll
43:14have some reading that guides you
43:16through the process finally we're going
43:18to talk about three big classes of
43:20issues so ethics and social concerns
43:23accessibility and Equity really really
43:26big issues in the Exile space and then
43:28also obviously privacy and security and
43:31you can pick which of these issues you
43:33want to focus on and I have three Key
43:36activities for you to try out and then
43:38assess from these different kinds of
43:39angles so this is going to be
43:41interesting and I hope we'll learn a lot
43:43that way as well so let's talk about the
43:45app classification activity first so in
43:47that one I'm going to show you the
43:49reality virtuality Continuum you've
43:50already started looking at this and uh
43:53the devices will obviously help you to
43:56it's a strong indicator which device you
43:58are using to try out some of the apps is
44:00a strong indicator where you may be
44:02ending up on the Continuum and again
44:05you're choosing the most specific terms
44:07rather than terms like Mr or XR in the
44:11second module so that's next week we're
44:13going to concern ourselves more with the
44:15concepts and really the Technologies
44:17implementing these Concepts so really
44:19understanding arvr Technologies and for
44:21this activity I've developed a few
44:23scenarios here is one of them it's the
44:25home decoration scenario
44:27so you could imagine using multiple
44:30different mobile devices and maybe it's
44:32collaborative with your family and uh
44:34what would be the right kinds of
44:36Technologies in terms of AR so you're
44:39going to have the technology tree that
44:41I'll introduce you to so we're going to
44:44understand a little bit more about all
44:46the different kinds of options we have
44:47on the AR side and on the VR side and uh
44:51the main goal is to really get you
44:53thinking about what kinds of questions
44:54should you be thinking about when it
44:56comes to to making recommendations for
44:59Technologies we're going to base this on
45:01the qoc paper from mle 1991 so questions
45:06options and criteria really a method a
45:08systematic method to explore the design
45:11space and I'm going to introduce you to
45:13the notation uh how do you actually make
45:16decisions how do you perform the
45:18assessment finally we're going to talk
45:19about issues and there are three big
45:22main classes of issues so social and
45:25ethical concerns accessibility and
45:27Equity really important as well as
45:29privacy and security and with these
45:32obviously come trust and safety and all
45:35kinds of concerns so I'm going to ask
45:37you to perform one of three activities
45:39you can perform more activities here's
45:40an example of a social VR meeting
45:43activity I would like you to actually go
45:44into a social platform I'll show you how
45:47I visited a conference virtually the i e
45:50virtual reality conference in 2020 where
45:53I went to and so this is like some
45:55footage from there and these are the
45:57kinds of um Impressions and this kind of
46:01experience that I'd like you to have so
46:03that you can then
46:05analyze uh these kinds of activities
46:08focusing on some of the concerns uh
46:11finally it's important to think about
46:12the design impact of some of the
46:15solutions that are out there and I
46:18already would like you to think like a
46:20designer so design thinking is something
46:22I would like uh to prepare you for and
46:24obviously the second course in this Muk
46:27specialization really focuses on that so
46:29for any of these activities it's very
46:31important to think about what happens
46:33before so how how are you prepared for
46:35that kind of activity as a user and then
46:38obviously as a designer what would you
46:40do to set the expectations of your users
46:42for this new reality that they enter so
46:44it's interesting to observe with
46:46yourself how much you feel like okay I
46:49think the app that I was using prepared
46:51me well for what was to come or not
46:54obviously all the kinds of thoughts and
46:56concerns you have why you were using it
46:58so during uh the activity and so
47:00therefore obviously uh as a designer we
47:03would want to balance realism with
47:05comfort and safety and safety is really
47:07uh the utmost uh important issue that we
47:11want to definitely address our users
47:13need to be safe and finally what happens
47:17after so as a return to reality help
47:20users transition back to the old reality
47:22the actual reality that is something
47:24that you should also consider so this
47:26honor trke is going to be hopefully fun
47:29uh and I think it would be great if you
47:31obviously did the entire XM Muk
47:34specialization and even all the honors
47:37track associated with them and looking
47:39forward to our discussions looking
47:41forward to the kinds of results you're
47:43coming up with and um I hope to see you
47:46in the honors track
48:01again welcome to the honors track this
48:03is actually the first exercise as part
48:06of the honors track every exercise comes
48:08with the task description uh in this one
48:10we're going to focus on uh trying out an
48:13XR app and actually classifying it along
48:16the reality virtuality Continuum or the
48:18mixed reality Spectrum we have readings
48:21to support you in this task and I'm
48:23going to go over some of the specifics
48:26uh to set the ex expectations so the Tas
48:28that I'm talking about is this one
48:30classify an XR app along the reality
48:33virtuity Continuum so this is something
48:37from mman Cino it's an a relatively old
48:39paper but it's an interesting concept
48:41and I'll show it to you in a second
48:43again but we also obviously talk about
48:45in the lecture so the first step in this
48:47exercise or in this task is actually to
48:49find obviously an XR app that you can
48:50try out with your equipment and so if
48:53you have questions about this obviously
48:55I write about it I give some tips of how
48:57to find XR apps I provide an overview in
49:00the technology landscape lecture so we
49:02should get we should have a relatively
49:04Good Sense by now of the kinds of XR
49:07applications you should be able to try
49:09out so you try out the app on your XR
49:12device it's important that you're using
49:13an AR or VR version of that app uh it's
49:17okay if it's 360 but I'd really like you
49:19to see it in virtual reality and analyze
49:23the user experience so in this case you
49:26don't have to think think like a
49:27designer you just have to think like an
49:28end user and uh you should then think
49:31about the only thing that I'm asking at
49:33this stage is to just classify the app
49:35as either AR or VR uses mixed reality
49:38only if you're totally unsure so this is
49:41actually not a good option and XR only
49:44if both AR and VR are supported and uh
49:47you tried out both um if if you have an
49:50app that supports both but you only have
49:52AR or VR available uh then you just say
49:56which of these modalities you've tried
49:58out and so you can classify the app as
50:01either a VR in that case you should
50:04consult the water is mix reality paper
50:06to confirm your choice and I have a
50:08little bit more information about that
50:09paper and obviously would like to invite
50:11you to read it it was actually written
50:13for a broad audience it's a Kai paper
50:16from 2019 uh from my lab and uh I
50:20thought this might be a good resource
50:21for some of us um and then optionally
50:24you can try out another app ideally at
50:26the opposite end of the spectrum so you
50:28get to see a bit more about these
50:30different extreme points in the
50:33Spectrum uh so better overview of the XR
50:36technology is what I'm hoping you would
50:37get from this uh exercise also this idea
50:41of considering the entire Spectrum so
50:43the entire Continuum not just the
50:46extremes um and so really exploring
50:48whether something could be characterized
50:50as augmented virtual reality and then
50:53possible definitions of mixed reality so
50:56I think I think that's where the paper
50:57is very useful that I ask you to read
50:59because it does identify six working
51:01definitions that we have uncovered from
51:03talking to uh experts and uh in the in
51:07both in industry and Academia and it was
51:09interesting to see how they they don't
51:11disagree but they have very different
51:12views on mixed reality as a spectrum and
51:15what in what it entails and what makes
51:17an app a mixed reality app and so it's I
51:20think it's good to have this overview
51:22that's the whole point of our paper to
51:24facilitate communication and so here
51:26show you obviously some examples I have
51:29tried out some of the apps so this is me
51:32playing a beat saer and where do you
51:34think this is I mean the fact that I'm
51:37wearing this uding headset um suggests
51:42that it is
51:44a reality app right okay and that's
51:48where you put it on the Spectrum and you
51:51can also reflect a little bit on the
51:52user
51:55experience here's this one is tricky and
51:58Pokémon go which I actually still don't
52:00really understand initially this view is
52:04neither augmented nor a virtual reality
52:07it is actually I don't even know what to
52:09call this it is a location aware 3D mode
52:14this is not augmented reality I mean it
52:16does map to the space but according to
52:19the mixed reality paper it might
52:21actually be weak augmented reality but I
52:23would normally not call this augmented
52:25reality and so so you can see obviously
52:28augmented reality is the term that is
52:30usually associated with Pokémon go but
52:33as I show now we can see a little bit
52:36more um kind of like augmented reality
52:38here it's really not great actually it
52:40doesn't work very well maybe it works
52:43better for you but in my experiments um
52:46no I'm not just looking for excuses why
52:47I didn't catch the Pokemon don't know I
52:50I can hear you say that no no uh it's
52:52just very interesting how I just didn't
52:55really understand how the game supposed
52:56to work also the fact that these objects
52:59appear in the middle of my room kind of
53:02like as if they were in another room so
53:04no occlusion no Advanced augmented
53:07reality but yes this would be more along
53:11the side of augmented
53:14reality so yes refer back to the
53:16definitions obviously virtual reality
53:18replaces reality so by now you would
53:21have guessed that beat saer was an
53:23example from the virtual reality space
53:25and augmented real
53:27uh is obviously enhancing reality but
53:29you can still see the real world it's
53:31actually a composite View and we talked
53:33about these definitions I also highlight
53:35the fact that the RS in each of these
53:37are different here we have a computer
53:39generated uh a virtual reality and here
53:42we actually have the real world the r is
53:44the real world and that's something to
53:46consider so again as I said as a little
53:49tip obviously the device that they're
53:50using helps you put things on this
53:53Continuum more easily again use the most
53:57specific term uh rather than saying it's
54:01somewhere on the Continuum or this is an
54:03XR app I use XR as a placeholder to say
54:07AR VR or Mr I wouldn't really like it if
54:11you classified something as mixed
54:12reality without a good reason if there's
54:14no more specific way to classify an app
54:18um but we do have quite a few options
54:20here so I prefer that and really I'd
54:23like you to go through the exercise and
54:25actually try out the app and then place
54:29it localize it somewhere on this
54:31Continuum it is a Continuum so this may
54:33make things a little bit ambiguous
54:36sometimes and that is also part of the
54:38exercise so to provide more support as
54:41you're going through this exercise and
54:42also as a way of confirming some of your
54:44choices I wanted you to read the waters
54:46mixed reality paper it's an invitation
54:48okay so this is a paper that my former
54:51postto Max my former PhD student Brian
54:53and myself have co-authored for Kai the
54:56kai conference the ACM Kai conference
54:592019 we also have more data available if
55:02you're more of a researcher I think it's
55:03a good paper and if you're starting out
55:05as a researcher in the XR space so in
55:08the mixed reality space this is really
55:10the paper for you H but even if you're
55:12not a researcher we wrote it for a broad
55:15audience and I hope you will find it
55:17helpful and interesting so the paper
55:20presents a literature review among other
55:21things a quite sophisticated L
55:24literature review 68 papers that we have
55:26looked at in detail and uh so there is
55:29quite a bit of stuff uh that you would
55:32learn just from reading the paper uh so
55:35we also Trace back various Notions of
55:37mixed reality and that's a little bit of
55:39the academic part so maybe that part is
55:41not that important and not that
55:42interesting to you and that's fine so
55:45you can skip ahead I wanted to really
55:47come to the discussion where we
55:49establish the dimensions so the number
55:51of environments to consider the number
55:54of users that a mixed reality
55:57application supports the level of
55:59immersion virtual reality the degree of
56:02interaction and input and output and
56:03these are things we discuss in there and
56:05so overall this provides a much more
56:06sophisticated framework and perhaps a
56:09better way to think about mixed reality
56:11experiences my postto Max really wanted
56:13to say in the end he really wanted to
56:15write in the conclusion the answer is it
56:16depends and so we kind of left it there
56:19but um it's more than it depends I mean
56:22the paper offers different ways to think
56:24about it it really establishes
56:26the aspects of reality that are often
56:28considered so motion is very prominent
56:30one audio haptics taste and flavor or
56:34even smell so L Factory displays is
56:36something that is supported in some at
56:39least virtual reality systems that I've
56:42tried out and uh the other way to think
56:44about is obviously if you start
56:46classifying an app in terms of the
56:48number of environments so is it both in
56:50the physical and in the virtual world or
56:52is it just in one of them that is a
56:54strong indicator of where we are on the
56:56Spectrum the number of users helps you
56:59further classify obviously level of
57:01immersion and level of virtuality these
57:03go hand in hand but they are different
57:05Notions as we established in the paper
57:08and so I invited to read it there and
57:09then obviously the degree of interaction
57:12and throughout my XR Muk specialization
57:14I also distinguish between implicit and
57:17explicit interaction so this comes back
57:19every now and then especially in the
57:20design and development oriented uh
57:23courses so implicit would be more camera
57:25based you looking at something
57:27accidentally so implicit can also be
57:29accidental actions uh but it's more like
57:32camera based interactions for example in
57:34AR when you're bringing markers into
57:36view although this could also be done as
57:38a more explicit form of an interaction
57:41other ways to think about explicit
57:43interactions include uh specific
57:45gestures and speech commands but these
57:46are just examples and so the degree of
57:49interaction and what is supported in an
57:51app also tells you a little bit more
57:53about the immersion uh the level of
57:55virtual and these all really go hand in
57:58hand together these factors allow you to
58:00characterize an XR app h a little bit
58:03more advanced and sophisticated way than
58:06just placing it on the Spectrum but so
58:08this paper would allow you to verify
58:11some of your choices and also be aware
58:14of different working definitions and
58:17also the fact that there can actually be
58:19some ambiguity uh where we put an app on
58:23the Spectrum really depends on the
58:25number of factors as As We Lay out in
58:26the paper and uh I'm looking for these
58:29kinds of cases where it's not clearcut
58:33and that would be an interesting
58:35starting point for some of the
58:36discussion in our forums so I'm looking
58:39forward to seeing what you're what
58:40you're trying out what kinds of apps
58:42you're trying out also if other people
58:44are trying out the same apps or other
58:46Learners report on the same apps that's
58:49fine it's especially interesting if you
58:51start to disagree on some of the
58:53classifications but maybe you can
58:55provide rational a better rational for
58:57how to think about things and in the end
58:59if you cannot decide I'll come back and
59:02help you decide and weigh in uh where I
59:05can so I think it's great that you are
59:08thinking about doing some of these
59:09exercises here in the Honor track the
59:11Honor track is supposed to be fun but
59:14also going deeper and obviously allowing
59:17you to earn a certificate in the end
59:19which I think would be a great result as
59:21a way of showing hey we've completed
59:23this exam Muk specialization or at least
59:26one of the courses so I have more so
59:29check back later there are other
59:31exercises in the honors track and really
59:34uh looking forward to seeing what you're
59:53doing so now the this is my car okay so
59:59sitting
1:00:01inside um okay to be honest I don't
1:00:03actually have my hand at the gear shift
1:00:06at the moment but that's what it
1:00:08suggests um and we can change gears here
1:00:11so that's the that's that's the car and
1:00:14obviously this is quite cool because now
1:00:16I can also look into the mirror I don't
1:00:18have a rear mirror but who needs a rear
1:00:20mirror this is a rally car and um so
1:00:24we're going to we're going to get
1:00:24started and we drive a little bit around
1:00:27and it's definitely quite
1:00:30immersive I now have cool sound in my
1:00:33headset so I'm not sure whether I'm
1:00:35capturing that sound I'm going to check
1:00:38that in a second it's definitely a
1:00:41little more immersive than what I showed
1:00:44you before and that was a good slide and
1:00:47obviously I can like look around right
1:00:49so just like when you're driving I mean
1:00:51if you were sitting next to me I could
1:00:54be looking at you the whole time a the
1:00:56whole time okay nothing happening here
1:00:58this is just
1:01:00practice and
1:01:02um I'm going to double check that we
1:01:04have the right sound um and
1:01:09then cuz now the sound is going to the
1:01:11headset and not to the TV maybe that'll
1:01:13make a difference so I'll be back in one
1:01:18second and we're going to just start the
1:01:20championship
1:01:22here and I'm so excited uh this quite
1:01:25tough
1:01:27so definitely challenging game a good
1:01:30simulation I mean I don't know what it's
1:01:32like to drive rally but I think it's
1:01:34pretty
1:01:35cool and off we go the Jer to your
1:01:38advantage you only need to take it once
1:01:40you have a little bit of a coach just
1:01:42giving you some tips we're probably not
1:01:44going to do very well just I'm nervous
1:01:46you know and I'm actually not that good
1:01:47in the
1:01:48game so let's get started um a yeah
1:01:53thanks have a good race all right so
1:01:56thumbs up and uh yeah yeah I'll put my
1:01:58headset on don't worry okay we're just
1:02:01getting ready I'm sitting in the car um
1:02:04it's a little bit weird there's this
1:02:09chair okay you
1:02:12know definitely not so well we have a
1:02:15car in front of us but that's fine there
1:02:17are still four others so oops I'm going
1:02:19to push this guy a little bit off we
1:02:21really have to cut the corner here and
1:02:23I'm being a you know attacked left and
1:02:27right but that's just life and um there
1:02:30we were jumping oops
1:02:35oh and now I need the handbrake but I am
1:02:38definitely not doing so well
1:02:42today I usually struggle but it's
1:02:44actually quite cool with VR
1:02:46um uh so is it feels quite
1:02:49realistic and this is the hard part and
1:02:52there are people who are really good at
1:02:53this but I'm not and and as you can
1:02:59tell but I'm not doing so badly so ooh
1:03:03that was not so
1:03:05good I think I'm almost the last so that
1:03:09is obviously quite
1:03:11sad but
1:03:13um maybe yeah this is very useful tip
1:03:17follow the cars
1:03:19ahead
1:03:22anyway yeah yeah I haven't been driving
1:03:25for a long time time
1:03:29now I'm not sure whether it looks like I
1:03:32know what I'm doing
1:03:33but I mean the whole point is you know
1:03:36to show you that it's fun and can be
1:03:39quite
1:03:40immersive um I'm going to after this lap
1:03:43I'm going to be done I
1:03:45promise maybe uh we're going to go oh
1:03:48come on so it's very realistic in terms
1:03:52of like the materials that I'm on also
1:03:54have a very different
1:03:56driving style so we're definitely the
1:03:57last now
1:04:00so uh this one is hard I really never
1:04:03really know how to drive on this
1:04:07um uh gravel or whatever it
1:04:11is and also sound isn't ideal at the
1:04:14moment but I
1:04:17hope okay we're just stop here and we'll
1:04:21accept the defeat and um I think the the
1:04:25360 died but well that was me driving a
1:04:29little
1:04:30bit so you know how they usually do
1:04:34interviews um post game uh we're going
1:04:38to so we're going to let me just say it
1:04:42was a uh tough race and
1:04:46uh uh fifth place is a good place right
1:04:49so I mean out of five but um nobody
1:04:51knows
1:04:52that all right
1:05:05so let's get started at the beginning
1:05:07when you decide to get a virtual reality
1:05:09headset and there's often some kind of
1:05:12tutorial or some kind of introduction to
1:05:14it and we're going to look at Oculus
1:05:16first steps just to get a sense for what
1:05:19virtual reality is at least the first
1:05:21time you see it so there you are in your
1:05:25headset just getting used to this new
1:05:28virtual reality around you looking
1:05:30around and then you see your hands and
1:05:33you actually don't see your hands you
1:05:35actually see some kind of model of hands
1:05:38some abstraction and they're moving as
1:05:40you are moving it's actually getting it
1:05:42relatively right and so you're surprised
1:05:44wow how does this actually work I'm just
1:05:46holding these controllers and the reason
1:05:49that this kind of works is because of
1:05:51the button placement on the controllers
1:05:54uh we can actually estimate where your
1:05:56fingers are and then render it
1:05:57accordingly and so we can even hide the
1:06:00controllers now in the latest headsets
1:06:02you have had hand tracking available so
1:06:05directly tracking your hands we don't
1:06:06even need the controllers anymore but in
1:06:08the aradus rift and Rift s in the
1:06:11beginning at least this is what it looks
1:06:14like so then you're getting familiar
1:06:16with this virtual reality H there are
1:06:17buttons that you can press and objects
1:06:20spawn like out of nowhere it's like
1:06:22magic so you're picking them up you're
1:06:24throwing them around and you are
1:06:26surprised by their behavior and the fact
1:06:29that they always come back I wish this
1:06:31were money that I could spend and it
1:06:33would just always come back and then
1:06:35you're picking up some other objects so
1:06:37here I'm starting to you know freak out
1:06:39because paper airplanes I mean paper
1:06:40airplanes and virtual reality that's
1:06:42really what we need at this stage and
1:06:44it's flying and I'm doing a good job so
1:06:47the term virtual reality that's however
1:06:49something that we should explore so in
1:06:52this video we're going to talk about
1:06:53some of the key characteristics we're
1:06:55going to look at it this way we're going
1:06:56to start with the five W's so basically
1:06:59really establishing what is virtual
1:07:02we're going to ask a lot of www
1:07:04questions who what when where why
1:07:09Etc then we're going to talk about key
1:07:11Concepts okay what makes it actually a
1:07:13virtual reality so this is the part that
1:07:16I want to focus on in a separate video
1:07:18I'm going to talk about the Technologies
1:07:19really this is how these key concepts
1:07:21are actually implemented how does VR
1:07:23work so when we talk about these Tech
1:07:25Technologies this will then maybe be a
1:07:27little bit technical but it doesn't hurt
1:07:29to understand some of the stuff in a
1:07:30little bit more
1:07:31detail and then throughout you'll see
1:07:34some demos me demoing things and
1:07:36actually as part of the course I want
1:07:38you to get involved as well and so
1:07:39you're trying out some of the demos
1:07:41yourself and you can report on some of
1:07:43these experiences so let's start with
1:07:45the five W's for virtuality the who what
1:07:48when where and why the why is probably
1:07:52like the best one for you why virtuality
1:07:55why wanted unless you are of course
1:07:57Enthusiast and you like never would you
1:07:59would never ask this question but a lot
1:08:01of people that I work with are like
1:08:03wondering Michael why do you have all
1:08:05these toys and so before we get to the
1:08:08why let's start with the who so the who
1:08:10I would say some of the key players in
1:08:12the space are Google oculus now Facebook
1:08:15HTC and Microsoft and there are others
1:08:17so don't be sad if I don't mention you
1:08:19but I mention you because you have some
1:08:21of these key products that kind of like
1:08:24pave the way way for virtual reality I
1:08:27want to start with cardboard cardboard
1:08:30really from Google an open source
1:08:32platform really cheap and all you have
1:08:35to do is like put your phone inside and
1:08:38then you can experience virtual reality
1:08:41for many of us this will be or was the
1:08:43first time we actually saw it and I
1:08:47often do this in the classroom and so
1:08:49this is actually a really good way to
1:08:50bring virtual reality to the masses if
1:08:53you will unfortunately this is is kind
1:08:55of like uh on the verge of dying so
1:08:59mobile the idea of using a mobile phone
1:09:01and an adapter like this is not being
1:09:04pursued actively anymore we're actually
1:09:06going really towards some of these more
1:09:09mature six degrees of freedom headsets
1:09:11so these are some of the concepts I talk
1:09:13about here as well so that we get a
1:09:15better understanding why is this
1:09:17happening why is this how is this
1:09:18different from the other thing you just
1:09:20talked about we'll get there all right
1:09:23and now I just grabbed the daydream so
1:09:25the Daydream is kind of like the
1:09:27deluxe cardboard okay made of nice
1:09:31material comes with a controller 3
1:09:33degrees of freedom we'll talk about that
1:09:35again goes on your head boom even stays
1:09:39there so can be wrapped around and it's
1:09:42relatively comfortable and actually
1:09:43feels quite good depending on the
1:09:45resolution of your phone because the
1:09:46phone goes inside you may actually have
1:09:48a pretty solid a pretty good virtual
1:09:51reality experience at least a on the
1:09:54lower end of the spectrum experience so
1:09:56cardboard and daydream and then Rift go
1:09:59and quest which are really the Oculus
1:10:01products and then HTC has a number of
1:10:04different Vive models out there and
1:10:06Microsoft is in the space with Windows
1:10:08mixed reality which is more like a
1:10:10platform and specification because
1:10:12different kind of vendors are then
1:10:13actually implementing the headsets like
1:10:15Acer or Lenovo so how did it how did it
1:10:18get to this uh there have been many
1:10:20before that and many things many efforts
1:10:23have actually died I would say I can
1:10:26identify these three waves there was one
1:10:28early on it was really exciting in the
1:10:311960s and the first head model displays
1:10:34sensorama and some other things that you
1:10:36may read about when you read about
1:10:38virtual reality history I don't go that
1:10:40far back I don't think it's that
1:10:42productive for purposes and then the
1:10:44second wave was really virtual reality
1:10:47movies and games so this is in the '90s
1:10:49H so Michael Douglas when he was still
1:10:53doing movies and that VR was a thing
1:10:56okay and it was really interesting and
1:10:58then Nintendo had the virtual py and
1:11:01that's not something I've experienced
1:11:03myself but maybe you have I never played
1:11:05with this I never got to play with it
1:11:06but I know that there was a lot of hype
1:11:09around it there were also various data
1:11:10gloves so really happy gloves um that
1:11:14people some people were at least buying
1:11:16into this and so this was the second
1:11:18major wave and I'd say the third major
1:11:20wave is the one we are kind of like
1:11:22riding on at the moment it started more
1:11:23or less with the Oculus Rift on as a
1:11:26Kickstarter project really in 2012 and
1:11:29then bought by Facebook and then made it
1:11:31more or less kind of like a mainstream
1:11:34uh platform for VR
1:11:37now so where is VR being used well it
1:11:40used to be mostly in research labs and
1:11:42Enterprise contexts but I'd say now it
1:11:45might actually be used by you you as the
1:11:47consumer or you as one of my students in
1:11:50my classes including this online course
1:11:52as
1:11:53well and then one why well why is a good
1:11:57question so I think there is something
1:11:59about virtual reality that I find very
1:12:00fascinating this idea of being ported
1:12:02into a different place is maybe that's
1:12:04maybe for some people but to me it's
1:12:06just like a really interesting
1:12:07interaction modality it's really around
1:12:09the user and it's an immersive 3D
1:12:12interface that I can explore I would say
1:12:14another reason why this is really
1:12:16happening and especially now is because
1:12:18these technological advances are really
1:12:21there's a lot of key players behind now
1:12:23and so we're pushing really forward
1:12:25quite a bit and it's become affordable
1:12:27which I think is also a really key
1:12:29factor that helps adoption so I'd say
1:12:33okay let's quickly review some of the
1:12:34key characteristics I introduced you to
1:12:36so I said VR by the book more or less is
1:12:39about autonomy or agency really giving
1:12:42control to the user and we achieve this
1:12:44through head tracking and body input
1:12:46like using your hands gesture voice
1:12:49speech or your eyes as input so this
1:12:52then leads to Natural interaction gestur
1:12:54and speech Fe are supported and then you
1:12:56can also use your hands or controller
1:12:58input or some combination of it um on
1:13:02some of the later models like the aradus
1:13:04quest you now have hand tracking
1:13:06available so you can put the controllers
1:13:08away and that's something I'll show in
1:13:10other segments but um uh it it wasn't
1:13:14like this it used to be more controller
1:13:16based and I do think that there are
1:13:17still lots of tasks where the controller
1:13:18is actually just a lot more useful and
1:13:21then presence which is really a concept
1:13:23that we are going to look at here
1:13:25but think of it as this idea or this uh
1:13:28feeling of being there and we achieve
1:13:31this through immersion which is actually
1:13:33another concept I need to explain a
1:13:35little bit more and then also
1:13:37multisensory input or stimuli so
1:13:39basically not just visual senses but
1:13:41also audio and then if you can do
1:13:44tactile haptic or maybe even Al Factory
1:13:46which is smell you could really create
1:13:49really interesting virtual reality to
1:13:51some extent realistic virtual reality
1:13:54experiences
1:13:56so in my mind however it doesn't have to
1:13:58be all that realistic yeah I think of we
1:14:02are being in a virtual environment some
1:14:04kind of virtual world it doesn't have to
1:14:06be nicely decorated it doesn't have to
1:14:08look like a fairy tale land that's fine
1:14:11um I want to be able to work on
1:14:14something or solve something in 3D and
1:14:16being immersed in it usually I think of
1:14:18steroscopic views right the rendering
1:14:20for each eye I don't work a lot in these
1:14:22caves that I talk about as well these
1:14:25room sized virtual reality displays so
1:14:27like inside the display rather than
1:14:29having it on your head it is really cool
1:14:32but I don't really work there so what I
1:14:34associate with virtuality is like
1:14:36something on my head and then I can do
1:14:39things in 3D and not just like look
1:14:41around but also walk around six degrees
1:14:43of
1:14:44freedom and then I think of an immersive
1:14:47task really some kind of spacal
1:14:49interaction and sound around me it's
1:14:52immersive in the sense that I'm really
1:14:54like into that task I'm not distracted
1:14:56by all the other things around me
1:14:58virtuality in many ways although that
1:15:00can sometimes be a little bit dangerous
1:15:01allows me to focus on the task at hand
1:15:05in virtual reality you don't actually
1:15:07this is not a conscious thing this is
1:15:09actually really it's like because it's
1:15:11wrapped around you this will happen so
1:15:14this can be good and bad but I do think
1:15:16the minimum we should aim for is some
1:15:19kind of believable experience I want to
1:15:21be good at the task that I'm doing I
1:15:23want to believe uh that that this is a
1:15:26virtual reality that I'm like engaged in
1:15:28there I don't have to be in a complete
1:15:30virtual world as I said it doesn't have
1:15:32to be photo realistic uh I think of
1:15:35virtuality really as an interaction
1:15:37modality from a human computer
1:15:39interaction perspective it's just a very
1:15:41interesting design world the interfaces
1:15:43that we can create in there and so
1:15:45that's how I think a little bit more
1:15:47about virtual reality so before we talk
1:15:50about some of the key Concepts let us
1:15:53first quickly introduce
1:15:55uh the kinds of VR displays we actually
1:15:59have so we can distinguish between
1:16:01head-mounted displays or
1:16:04hmds and room sized VR environment so
1:16:07you're basically like inside the display
1:16:10and we refer to these as caves cave
1:16:14automatic virtual
1:16:16environments and here I have an
1:16:18illustration of this so on the left you
1:16:19see a head monitor display the way I've
1:16:22tried to draw it here it looks supposed
1:16:24to look like a Google cardboard um so
1:16:28it's supposed to be this and again this
1:16:30like goes on your head and then you have
1:16:32a rendering for each eye um so that's
1:16:35what I'm trying to illustrate here in
1:16:37this concept so uh for the room siiz VR
1:16:41or the cave I mean you would actually be
1:16:44standing inside these inside of this
1:16:46room you would have uh multiple walls
1:16:49and even a floor and sometimes also a
1:16:51ceiling that is being projected on
1:16:54usually caves are projection based or
1:16:56projector based so the main difference
1:16:58between the head mon display and the
1:16:59cave is well you're putting this on your
1:17:02head and then you are moving with a
1:17:04display and in the cave you're actually
1:17:07moving inside well the display if you
1:17:09will each of these walls function as a
1:17:12display projection based and you see the
1:17:14kind of like the same
1:17:16content uh roughly here there's no
1:17:18stereoscopic view um however you usually
1:17:21wear something um that uh track you
1:17:25because we do need to render the content
1:17:27from the perspective where you currently
1:17:29are so as you're moving to this space
1:17:31here this projection would actually
1:17:34change a lot and would look very
1:17:36differently H so if people are like
1:17:39observing on the outside it looks cool
1:17:40for people H having people inside the
1:17:42cave but like on the outside the
1:17:45perspective isn't correct the
1:17:46perspective is usually for the people
1:17:48inside the cave let's look at some of
1:17:50the more common Concepts so that we get
1:17:53a better sense of what what virtual
1:17:55reality is really
1:18:03about key concepts of virtual
1:18:07reality so now that we have established
1:18:09these basic display Technologies in the
1:18:11remainder of this uh video I want to
1:18:14talk about uh these key Concepts so the
1:18:18seven key Concepts that I've chosen for
1:18:19us to focus on are autonomy and agency
1:18:23presence and immersion we will talk
1:18:25about embodiment and avatars cognition
1:18:28and empathy and we're also going to talk
1:18:29about prop perception and motion
1:18:31sickness and then haptics and pseudo
1:18:33haptics and finally we're also going to
1:18:34talk about this concept of of boundary
1:18:37and redirected walking and now we're
1:18:39going to look at each with a little bit
1:18:40of an example and then I think we have
1:18:42established some of the key uh the key
1:18:45Concepts that we should be familiar with
1:18:46before we learn about the technologies
1:18:48that really Implement some of these
1:18:51Concepts so the first concept is
1:18:53autonomy and agent
1:18:55and I wanted to give a few examples here
1:18:58so the first example is the user can
1:19:01actually choose their own perspective in
1:19:04the scene right the user is actually in
1:19:06control of the virtual camera my second
1:19:08example is the user can usually choose
1:19:10to navigate the scene in many different
1:19:12ways so most VR environments um allow
1:19:16you to teleport or provide some kind of
1:19:18menu functionality but if you're using a
1:19:20six Dees of Freedom headset you can
1:19:21actually obviously physically walk and
1:19:24then uh the the third example here for
1:19:26autonomy and agency is the user can
1:19:28choose to interact with any objects now
1:19:30that is really important right uh
1:19:31basically everything you show to them
1:19:33unless it's like colorcoded in some
1:19:35specific way they may choose to pick up
1:19:37any object and expect that it behaves
1:19:39when they drop it so physics and gravity
1:19:42are usually things that are also
1:19:43expected by users so in a nutshell the
1:19:46user can choose their own perspective on
1:19:48a scene the user can choose to navigate
1:19:50the scene in many different ways and the
1:19:53user can choose to interact with with
1:19:54any objects that they see both in the
1:19:56virtual world but also in the physical
1:19:58world let's talk about presence and
1:20:00immersion so according to Mel later
1:20:03there are like two things going on when
1:20:05you're entering a virtual world um there
1:20:08is this idea of play illusion which
1:20:11really answers this question of am I
1:20:14there and the more time you spend or the
1:20:17the the better the world is designed you
1:20:19may feel like wow you are there and
1:20:22there is this uh plausibility illusion
1:20:25which really answers the question is
1:20:26this happening and kind of like it's
1:20:29it's overcoming this question in some
1:20:31sense the suspension of disbelief kicks
1:20:33in and then these two things will play
1:20:36together the place illusion and the
1:20:38plausibility illusion will make you
1:20:40forget that you are in VR the other main
1:20:44concept often uh used in virtuality is
1:20:47immersion and so immersion we usually
1:20:49distinguish between something is less
1:20:51immersive and something is more
1:20:52immersive but what does that mean so I
1:20:53wanted to give you some examples so I'll
1:20:55start on the very left here less
1:20:57immersive my first example is 2D images
1:20:59and videos and that you probably would
1:21:01agree it's uh not that immersive um and
1:21:05you will even more agree when you see
1:21:06the other things I put up there so 360
1:21:09images and videos um on a phone so what
1:21:13I mean by that is like you have the 360
1:21:15content and you're like moving your
1:21:16phone around and obviously according to
1:21:19the dros scrub and acceler meter in the
1:21:21in the phone so the inertial measurement
1:21:24unit the you it adjust the perspective
1:21:27and it is arguably more immersive than
1:21:29looking at a 2D image and video on just
1:21:32a
1:21:32screen so now what is more immersive is
1:21:35when you actually have the 360 image and
1:21:38video uh in a headset so even if it's
1:21:41just cardboard like you being in there
1:21:44and looking around and having this whole
1:21:48this whole rendering in your face
1:21:50through steroscopic views and really
1:21:52kind of like a a high field of uh view
1:21:56anyway when you're comparing uh the
1:21:59experience of 360 photos and videos on a
1:22:01headset as opposed to a phone you would
1:22:03probably agree that the headset is more
1:22:05immersive so the question becomes how
1:22:07can we get even more immersive well and
1:22:09then I put on 3D images and videos and
1:22:13now we're talking about three degrees of
1:22:14freedom headsets going to use the Oculus
1:22:17go and then you have a 3 degrees of
1:22:18freedom controller so it can figure out
1:22:21the orientation and a three degrees of
1:22:23freedom headset so it knows knows where
1:22:25you're looking but it doesn't know
1:22:27position it just knows orientation so
1:22:31that is arguably not as immersive and
1:22:34why is 3D image and video more immersive
1:22:36than 360 well that's because it has real
1:22:38depth and that really plays a role as we
1:22:41enter even more of the immersion
1:22:43Spectrum here so we're going towards 3D
1:22:45images and videos with a six degrees of
1:22:47freedom headset so for the six degrees
1:22:50of freedom headset I just grabbed as an
1:22:53example the Ulus Quest six degrees of
1:22:55freedom headset so means you can
1:22:56actually not just like look around but
1:22:59also walk around now that doesn't make a
1:23:01lot of sense for a 360 because 360 is
1:23:03usually captured at like one static
1:23:05position where the camera is or
1:23:07sometimes a 360 camera is moved through
1:23:09the scene but you don't your movement
1:23:12usually doesn't make a difference in a
1:23:14360 photo or video it's really just the
1:23:16orientation but for 3D where we actually
1:23:19have real depth you can now actually
1:23:20start to walk around and we can track
1:23:23this with a 60° Freedom headset all
1:23:25right my last example here is a 6er
1:23:28freedom headset with haptics and other
1:23:30kinds of sensors being stimulated uh
1:23:32there is are ways to make it even more
1:23:34immersive and basically the more senses
1:23:37receive input where the more interactive
1:23:40the content is uh I mean both implicit
1:23:43like how you explore it with the headset
1:23:46but also explicit like how you touch it
1:23:47how you voice activated um that how much
1:23:51you become part of the story of whatever
1:23:53the virtual reality content is that
1:23:55really drives the immersion and so my
1:23:59example here for immersion is like this
1:24:02I'm getting up I'm like in the Oculus
1:24:04Lounge again and the first thing I
1:24:06actually did was actually just looking
1:24:08at this View and just looking
1:24:11around and feeling like being there it
1:24:16wasn't exactly a feeling of presence
1:24:19obviously I think uh the more time you
1:24:22spend in the v VR experience also
1:24:23contributes
1:24:25uh to you forgetting around about the
1:24:27real world and and on the immersion
1:24:30level this was really quite immersive it
1:24:32really is a beautiful scenery done in
1:24:34multiple layers and actually quite a lot
1:24:36of detail so let's talk about embodiment
1:24:39and
1:24:40avatars here uh I will continue with the
1:24:43example and what I'll do is I'll
1:24:45actually just show you my avatar and so
1:24:48I'm going to that mirror uh to actually
1:24:50take a look at myself and after I've
1:24:52been experimenting a little bit with the
1:24:53options that I have have I decided this
1:24:56is actually quite cool and well I mean I
1:24:58look like a cool guy in the mirror I
1:25:00could live with that so these were the
1:25:02first three concepts I wanted to
1:25:04illustrate and now we're going to look
1:25:06at the second set of Concepts so
1:25:08cognition and empathy prop perception
1:25:09motion sness and also haptics uh not so
1:25:12much pseudo haptics in this example and
1:25:14the example that I have here is we're
1:25:16going back to me driving in this ready
1:25:18car from dir Ready 2 yeah so here you
1:25:21see me in my uh car I'm in the virtual
1:25:24reality headset now this is 60 Dees of
1:25:27Freedom Inside Out tracking so quite
1:25:29immersive just to say and then obviously
1:25:31I have um the the steering wheel and the
1:25:36gearshift and that adds to the haptics
1:25:38aspects but the example here is about
1:25:42cognition and empathy and as you watch
1:25:44me
1:25:45drive um I really develop this mental
1:25:48model of this uh map that I'm driving on
1:25:51so this I mean to be honest I've now
1:25:52spent quite a few hours playing this
1:25:54game and it's really hard to get better
1:25:56at it it's very realistic
1:25:58simulation and so I can definitely say
1:26:01that I'm building empathy towards race
1:26:03car drivers I mean that's not really the
1:26:05intention of this game the intention of
1:26:06this game is probably to have fun and
1:26:08maybe get better and it's quite
1:26:10competitive but um it really allowed me
1:26:13to build this empathy towards uh really
1:26:16sitting the drivers of a ready car and
1:26:19so the other thing I mentioned is proi
1:26:21reception and associated with that I uh
1:26:24have also mentioned motion signal so
1:26:26proper reception is really like actually
1:26:29feels pretty good in this car cuz I'm
1:26:31sitting and I'm like moving around here
1:26:34checking my uh mirror I don't have a
1:26:37rear mirror and then I'm like driving
1:26:39around and I'm looking around while I'm
1:26:41driving and all that feels really kind
1:26:43of natural the speed also kind of feels
1:26:46okay and I don't actually feel motion
1:26:48sick um in this game it's actually
1:26:51pretty solid when I uh pressed the
1:26:54brakes or actually use the handbrake um
1:26:58for drifting the car responds um as I
1:27:00would expect so overall I don't actually
1:27:03experience a lot of motion sickness
1:27:04there's really a good mapping between
1:27:06what H what I do physically with my body
1:27:09and uh what happens in the virtual world
1:27:11finally I wanted to talk about haptics
1:27:13and pseudo haptics and then I wanted to
1:27:15share the equipment with you that I'm
1:27:17using uh because they really generate a
1:27:20quite uh realistic experience good
1:27:23haptic cuz I actually have a steering
1:27:25wheel I have a gear shift and a
1:27:28handbrake and uh obviously the last
1:27:30piece of equipment is actually the VR
1:27:32headset now that feels a little bit like
1:27:34a helmet so I imagine that if you're
1:27:36rally driver you have a helmet so um it
1:27:39does it does feel still quite realistic
1:27:41to have something on your head and so
1:27:43overall the whole experience is actually
1:27:45quite realistic and immersive and that's
1:27:47also why I showed it to you earlier and
1:27:50obviously it's quite fun at least for
1:27:52the person inside there and and so what
1:27:55is interesting when you look at it
1:27:57from uh from this outside perspective I
1:28:01know where the gear shift is I know
1:28:03where the handbrake is I cannot see
1:28:04through the headset so it just really
1:28:06took a few minutes to train the motor
1:28:08memory if you will so that I remember
1:28:10how to reach out and uh where the pedals
1:28:13are obviously I I can feel those so yeah
1:28:16this is like how over time uh basically
1:28:18through haptics and the whole physical
1:28:21layout and my motor memory is being
1:28:24trained and I know exactly where the
1:28:25gear shift is and where the handbrake is
1:28:28uh but like me showing this to you now
1:28:31and paying attention to making it like
1:28:32fit into the camera view but when I'm
1:28:34sitting in the seat I really just it's
1:28:37just there I just I just know and um
1:28:40this really translates nicely to this
1:28:42experience now I gave some other
1:28:45examples around pseudo haptics right
1:28:47this idea of you uh lifting an object
1:28:50but there's actually no force feedback I
1:28:52mean we can also do muzzle stimul ation
1:28:54and all these kinds of a little bit more
1:28:56crazy things um that I'll invite you to
1:28:59read about and maybe talk about in the
1:29:01forums but I don't actually have these
1:29:03examples there's no approved product
1:29:05right now uh that I know at least uh
1:29:08that you know does muscle stimulation
1:29:11but I've seen a lot of research projects
1:29:13by really good colleagues and so this is
1:29:16also an exciting uh concept I talked
1:29:18about redirective walking uh one thing
1:29:20that I definitely want to talk about is
1:29:21the boundary the virtual boundary that's
1:29:23something you usually set up in a VR
1:29:25experience so I'm going to show you this
1:29:26as the last
1:29:30example in this video I'm going to set
1:29:32up the stationing boundary um and uh
1:29:36introduce you to the concept of the VR
1:29:39boundary if you want so um as it shows
1:29:43you in the video is like the safest way
1:29:45is to just like Mark really the area
1:29:49that is really safe to use now I have a
1:29:52chair here but that one moves all the
1:29:53time time and that wouldn't be too much
1:29:57of a play area but that is uh what it
1:30:01recommends now what you wish you had
1:30:03were like obviously the whole room
1:30:07available um for you to
1:30:10play and and and that's cool but as you
1:30:14can see now the headset finds all these
1:30:16obstacles so lots of obstacles here in
1:30:19the room it finds all these obstacles
1:30:22and it shows you hey that uh dangerous
1:30:25potentially so be
1:30:27careful um so what I usually do is and
1:30:30here's a couple of tips for how to do
1:30:31this boundary so we are going to use
1:30:34mostly the safe area but then I'm going
1:30:37to do a couple of things to it to help
1:30:40me better understand the room layout the
1:30:44first thing I'm going to do is I'm
1:30:45actually going to include my desk area
1:30:49so that I stay in VR while I'm seated so
1:30:52I'm okay to accept these obstacles here
1:30:56that's fine um at least I can enjoy a VR
1:30:59while I'm seated I'll leave this area of
1:31:02the room untouched we really don't want
1:31:03to bump into the shelf and the wall and
1:31:07then I'm going to actually Mark the exit
1:31:09out of the room it's not too much
1:31:13um okay so I'm going to do that and that
1:31:17will help me illustrate something in a
1:31:19second the first thing I'm going to show
1:31:21is like when I sit down now and I'm
1:31:23going to going to sit down I'm going to
1:31:26reenter uh the menu as I'm seated so
1:31:29this is kind of cool now I can actually
1:31:31leave things that way and um we're kind
1:31:35of sitting down hand tracking and all
1:31:38that would work um I have the I'm going
1:31:41to show you so I'm kind of like within
1:31:45VR and even if I move lean forward or
1:31:48something I'm still in VR so that's
1:31:50useful if I had not included uh the
1:31:53bounce down there I would exit VR all
1:31:56the time because as I'm like walking to
1:31:59the door to the
1:32:01exit um if I exit that space now you'll
1:32:05see how I'll say you see the film set
1:32:08and this actually is really disruptive
1:32:11to a VR experience obviously so you want
1:32:13to make sure that you are like have it
1:32:16mostly set up nicely and be in an area
1:32:21where it's safe and have it anible so
1:32:24when you look down if you're like unsure
1:32:26where you are in the room this happens
1:32:28sometimes um it'll show you and you
1:32:30really want to avoid bumping into like
1:32:32areas like this and so you don't want to
1:32:35go too close to the shelf or to the wall
1:32:38and so when it becomes red is really an
1:32:40issue I'm going to show you roughly
1:32:42where I'm what I'm doing here so when
1:32:44I'm going too close to the wall it's an
1:32:46issue or the shelf and um yeah you want
1:32:49to avoid that but otherwise this is uh a
1:32:53few tips to keep in mind about the
1:32:56boundary and um yeah it helps to have a
1:33:01couple of areas
1:33:03marked uh to really understand the safe
1:33:05zones so this was my overview of key
1:33:08Concepts in VR um I chose a few of them
1:33:13there are more um I decided to do it by
1:33:16example and intuitively rather than
1:33:19giving you the cut and dry definitions
1:33:22uh some of these you can look up
1:33:24um if it help if it's better for you to
1:33:26have a texture definition of of some of
1:33:28these Concepts but like the seven that I
1:33:31shared about autonomy in agency empathy
1:33:35cognition presence immersion these are
1:33:38all some key terms uh that you should be
1:33:40familiar with and um that's what I
1:33:43wanted to
1:33:44accomplish and uh we also talked about
1:33:46haptics to do hatic so actually there
1:33:48was a lot of stuff in here and now in
1:33:50the next video we're going to look at
1:33:51how these concepts are implemented in
1:33:53technology Oles and these Technologies
1:33:55are actually enabling uh the experiences
1:33:58that I was talking about including
1:34:00obviously adding to this feeling of
1:34:02presence and
1:34:15immersion in this segment we're going to
1:34:18look at virtual reality Technologies now
1:34:21there is a lot of Technologies out there
1:34:24and uh what I want to do in this segment
1:34:27is actually look at some of the key
1:34:30ingredients so what does it take to
1:34:33create uh this immersion this feeling of
1:34:36presence and some of the concepts we
1:34:38talked about earlier so what does it
1:34:40take in terms of
1:34:42Technologies so we are going to look at
1:34:44virtual reality Technologies and a few
1:34:46of the enabling ones from 360 photos and
1:34:49videos to steroscopic displays rendering
1:34:52for H ey we also going to look at caves
1:34:55or cave which is in a recursive acronym
1:34:58for cave automatic viral environment
1:35:02we're going to learn about 3° of Freedom
1:35:04versus 6 degrees of tracking or three do
1:35:06versus six Dove we're going to learn
1:35:08about outside in and Inside Out tracking
1:35:12uh the idea of putting sensors into the
1:35:14environment and actually embedding these
1:35:16sensors into the devices that we put on
1:35:18our body we're going to learn about hand
1:35:20tracking as I think that hand tracking
1:35:23is a technology that is rapidly
1:35:26improving and now becoming standard in a
1:35:28lot of VR devices and finally so not to
1:35:32just keep it entirely visual I thought
1:35:34it is also important to incorporate
1:35:37spatial audio all right so now let's
1:35:40look at these in a little bit more
1:35:41detail so what you can see here is that
1:35:44in this 360 video I have agency I have
1:35:47control um because it was recorded in
1:35:50360 it was captured all around the
1:35:53camera
1:35:54and I can I can choose to look at
1:35:56Michael or not next we're going to look
1:36:00at steroscopic displays so the concept
1:36:03of vendoring a different view a slightly
1:36:05different view for each eye there's a
1:36:08little shift between the pupils and uh
1:36:10that little shift then creates a depth
1:36:14uh view for our eyes so it's actually
1:36:16quite cool so here we have the cardboard
1:36:19VR display source code uh this has
1:36:21really a few demos that I think are cool
1:36:24so here we see an example of a rotating
1:36:27Cube and and that is just 3D okay so
1:36:31we're just looking at it in 3D but now
1:36:34I'm going to uh start the steroscopic
1:36:36view here the steroscopic display and
1:36:40now what it does is it creates a
1:36:42rendering for each eye left eye right
1:36:45eye there's a tiny little shift between
1:36:47these eyes and this is has to do with
1:36:49the interp distance the ipd and this is
1:36:52something that can be adjusted on some
1:36:53headsets as well and so basically we're
1:36:56not seeing two cursors and two cubes
1:36:58actually when you put this on your face
1:37:01you actually then perceive depth you see
1:37:04one cursor if it's done well it gives
1:37:06you really this illusion of depth and
1:37:08that is the whole key behind steroscopic
1:37:11displays and this is really the main
1:37:15thing behind cardboard so remember
1:37:18cardboard phone goes inside and then we
1:37:21actually have a rendering for H eye with
1:37:23these barrel
1:37:24distortions uh left eye right eye okay
1:37:29and um yeah and then we perceive it like
1:37:33this like you see in the
1:37:35video rendering for each
1:37:39eye cool so that was uh that and I think
1:37:43that is really a first important thing
1:37:45yeah creating this uh immersive view if
1:37:49you will a rendering for AGI that gives
1:37:52you then a a sense of depth and we are
1:37:55starting to see things and we are
1:37:57starting to call this a virtual reality
1:38:00360 photos and videos by and in
1:38:03themselves are not virtual reality they
1:38:05are often associated with virtual
1:38:06reality but they only become a virtual
1:38:09reality when we see them through a
1:38:11steroscopic viewer like the cardboard
1:38:14with a rendering for each eye cuz then
1:38:16our head movement can be tracked just
1:38:19the rotation not normally the position
1:38:23um so we cannot actually walk inside 360
1:38:26photos and videos but we are starting to
1:38:28get control we're gaining some agency
1:38:30over some of the content at least in
1:38:32terms of
1:38:34rotation my next example is a little bit
1:38:37different it's a different display
1:38:38technology it also creates a virtual
1:38:40reality display but in this case you're
1:38:43actually stepping inside what is called
1:38:45a cave automatic virtual environment
1:38:48inside a cave and here I'm showing it
1:38:51based on the example that a colleague of
1:38:54mine at the University of Michigan gave
1:38:55to me so Ted Hall um he is in our maidan
1:39:00and he shows one of the applications he
1:39:02was working on for some of the
1:39:03researchers that allows him to explore a
1:39:06mammoth and look at it in
1:39:093D what was really impressive here is
1:39:12how the view is rendered just for him he
1:39:15wears these funny antennas and there is
1:39:18some tracking system an Opti track
1:39:21system in the room that actually tracks
1:39:23these antennas and uh allows him to
1:39:27actually walk around and if you look at
1:39:29the video one more time you will see how
1:39:31the view seems to be like somehow
1:39:34distorted and overlaid and that is the
1:39:38effect uh that is being created to uh
1:39:42give him a depth sense so these will
1:39:44shift further away the further the
1:39:46virtual content is uh away from him and
1:39:49there will be coming more
1:39:51overlapping um the the closer the Cent
1:39:54is to him and when you go to the movies
1:39:56and watch a 3D movie you can also you
1:39:58know you put on your glasses that they
1:40:00give you but then sometimes you probably
1:40:02take them off because you're tired or
1:40:04not and um or you just feel like hey
1:40:07let's look at this from a VR perspective
1:40:09so these 3D classes are actually
1:40:12counteracting the shift and they're
1:40:14putting it together in front of you so
1:40:17usually some kind of shutter display
1:40:19that quickly so it can be active or
1:40:21passive but some kind of shutter display
1:40:23display that then shows you just the
1:40:25image for the left eye and then the
1:40:27image for the right eye and then this is
1:40:30such a high frequency that you don't
1:40:31even notice and it'll create a 3D image
1:40:35for you uh that you perceive as depth so
1:40:39let's look at uh actually head tracking
1:40:41uh and also controller tracking if you
1:40:43will so we have virtual reality motion
1:40:45controllers they can be tracked in 3
1:40:47Dees of Freedom or six degrees of
1:40:49freedom so let's look at this in a
1:40:51little bit more detail so three degrees
1:40:54of freedom like the cardboard um looks
1:40:57like this okay and then we can actually
1:40:59track along the X so pitch we can track
1:41:04along y yaw but just rotation and we can
1:41:08track along along Z but just again
1:41:11rotation so roll you cannot actually
1:41:13move along these axes but you can rotate
1:41:16your head so this is me in cardboard and
1:41:20so this
1:41:21is PID
1:41:24ya and roll if you will this is just 3°
1:41:28of freedom and if you can actually also
1:41:31adjust your position so the second
1:41:34example here is the hollow kit the one
1:41:36that I had earlier right and so it's a
1:41:39device wrapper just like cardboard but a
1:41:41little different for AR so phone goes
1:41:44inside we can see through and if we
1:41:48expose the camera we can actually do
1:41:50some tracking and we can do positional
1:41:52tracking so now now in addition to head
1:41:54roll pitch and Y we can also actually
1:41:58determine whether the device moves along
1:42:01the z-axis along the x axis and along
1:42:04the Y AIS we can do this through um a
1:42:08few techniques tracking techniques um
1:42:11like slam okay which is simultaneous
1:42:14localization and mapping of the device
1:42:17in 3D space uh these are some computer
1:42:20algorithms that actually work based off
1:42:22the camera Fe
1:42:24and allow us to determine some kind of
1:42:27motion Optical flow if you will Optical
1:42:30flow would be Computing a vector as I
1:42:33move the device this way and we are
1:42:36actually producing a vector from here to
1:42:38there or if you will what happens to the
1:42:40camera feed all the pixels actually
1:42:42shift to the right okay and uh that way
1:42:46we can actually figure out that the
1:42:47device moved to the left now that is
1:42:50what I'm showing here in this
1:42:52illustration so this is 6° of Freedom so
1:42:56we have also positional changes plus and
1:43:00minus x a positional changes along the y
1:43:03axis and positional changes along the
1:43:05z-axis now we actually do have position
1:43:08and orientation and we refer to this as
1:43:11six degrees of freedom tracking again
1:43:14with this kind of tracking you can
1:43:15actually walk around in a 3D space you
1:43:17can not just look around like with 360
1:43:20video but you can actually then move
1:43:22around around and actually like this
1:43:25gives you really a lot of agency uh in
1:43:28the content and that's then when it
1:43:29becomes to when it starts to become hard
1:43:32for designers and developers that have
1:43:33to design for that
1:43:37agency so here I have an example of the
1:43:41Oculus go the Oculus go is technically a
1:43:453° of Freedom headset it comes with a
1:43:49three degrees of freedom controller that
1:43:51controller will always be on on the
1:43:53right down there and we can figure out
1:43:55its rotation but we cannot figure out
1:43:57where it actually moves in 3D space so
1:44:00it is really a little limited and but um
1:44:04you can already do quite a bit just by
1:44:06being able to look around and uh rotate
1:44:09the controller if we use this laser uh
1:44:12interaction here this laser controller
1:44:14laser pointer interaction you can reach
1:44:16objects at a distance and you actually
1:44:17do have quite a bit of control already
1:44:20over the virtual reality content
1:44:23all right so here we actually do have
1:44:25the go that's the controller I just
1:44:28showed and that's the go the device that
1:44:30goes on your head okay and it is three
1:44:33degrees of freedom okay so no travel
1:44:36just rotation and the same for the
1:44:38controller so just a little bit of a
1:44:40pointing device but whether I have it
1:44:42here or there doesn't make a difference
1:44:45really the difference is rotation and so
1:44:48this makes really for quite some
1:44:50differences in terms of how you design
1:44:52and develop and then also Al how you can
1:44:53use uh virtual reality applications
1:44:56we're going to learn more about this in
1:44:57other segments of the Muk so when it
1:45:01comes to six degrees of freedom a very
1:45:03good way to demonstrate this is through
1:45:05the Oculus Quest so you usually draw a
1:45:08boundary and then you can walk inside
1:45:11those bounds and that's then supposed to
1:45:13be kind of like a save Zone and with six
1:45:17degrees of freedom both for the
1:45:19controller and for the head I can now
1:45:22actually walk around and move the
1:45:24controller in 3D space and I'll be
1:45:27tracked all the time and um that is
1:45:30really quite cool so this was 3° of
1:45:32Freedom with a 6° of Freedom tracking
1:45:35for both The Head and the controller and
1:45:37now we're going to talk about outside in
1:45:39and Inside Out tracking so in a nutshell
1:45:42the difference between outside in
1:45:45tracking is that you have some sensors
1:45:47in the environment that are pointed
1:45:48towards you and they track you uh as you
1:45:52move around in a certain uh area that is
1:45:55covered by these sensors inside out
1:45:57tracking means you actually have those
1:45:59sensors inbuilt into the device and the
1:46:02cameras are actually inside the device
1:46:04I'm going to show you two examples here
1:46:06so my first example here for outside int
1:46:08tracking is actually the original Oculus
1:46:10the original Oculus required sensors to
1:46:12be placed on a table usually in front of
1:46:15you so these sensors here uh were
1:46:19actually pointing uh towards you and
1:46:22then they are creating a sensor field
1:46:25and they can see you in this range and
1:46:27you had to make sure that these sensors
1:46:29are overlapping so there's some
1:46:31triangulation happening and what they're
1:46:33actually looking for is light patterns
1:46:35emitted from uh the controller and also
1:46:38from the headset so my second example is
1:46:41the Oculus Rift s uh which is basically
1:46:45the new version of the rift it actually
1:46:47has the cameras inbuilt into the
1:46:49device now this makes it possible to
1:46:53actually do both the head tracking and
1:46:55also tracking the controllers in six
1:46:57Dees of freedom without any external
1:46:59sensors so the cameras are on the device
1:47:02and basically this is the field of view
1:47:05that these cameras have I mean it's an
1:47:07approximation it's not absolutely
1:47:09correct here and you can see that um the
1:47:12controllers are actually captured uh
1:47:14this way in fact the Oculus uh Rift s
1:47:18and the quest even they also have
1:47:20additional cameras on the side so even
1:47:22when you have the controllers next to
1:47:24the device or even below the device uh
1:47:27the device can still see the controllers
1:47:30this is actually quite an advancement
1:47:31over outside in tracking so when you had
1:47:33these stands on the table they could
1:47:35only see so and so far and usually the
1:47:37table would include things so as you
1:47:40were doing things below the table you
1:47:42couldn't be tracked anymore now with
1:47:44Inside Out tracking you actually have
1:47:46these sensors on on the device and so
1:47:49anything that is visible so in a certain
1:47:52area around around them can be tracked
1:47:54so when you put your hands behind your
1:47:56back obviously this will not work
1:47:59but but that's fine so here I have an
1:48:02example of the HTC VI with outside in a
1:48:06form of outside in tracking it's
1:48:08technically not absolutely correct but
1:48:11this is what it usually looks like you
1:48:12have a sensor in the environment and
1:48:15then uh the controller here is coming
1:48:18towards the sensor and it's being
1:48:20tracked in six degrees of freedom
1:48:24so here's an example of me demonstrating
1:48:26the rift s and I often do this um so I'm
1:48:29pointing out the different cameras on
1:48:31the device and I am explaining uh how
1:48:34the rift s can actually see has cameras
1:48:36on top downwards to the left and right
1:48:39in front and uh has a really good field
1:48:44of view for these cameras to actually
1:48:46track and I'm making some comparisons to
1:48:48the Oculus Rift uh original sensor
1:48:51stance that you see in the back
1:48:53so let's talk about hand tracking hand
1:48:55tracking now has actually become a
1:48:57standard on the latest generation of VR
1:49:00devices I'm going to introduce you to
1:49:01hand tracking in two steps first I'll
1:49:03show you what this used to be like with
1:49:06a device called The Leap Motion which I
1:49:09have here so the leap motion is a USB
1:49:12tethered device that would be mounted on
1:49:15top of the headset and it creates a
1:49:18field of view a window in which uh you
1:49:22can actually do hand tracking so I'm
1:49:24going to show you this first and then
1:49:26I'm going to show you how hand tracking
1:49:27actually works on the Oculus quest which
1:49:30we have there and um now it's
1:49:33essentially inbuilt so built into these
1:49:36devices so this is Del motion it's
1:49:39really not very big and creates this
1:49:41window here then I can actually see my
1:49:45hands nicely
1:49:47articulated so um in this example I'm
1:49:50just like moving my hands up and down
1:49:53and I can actually draw uh if you
1:49:56will um with left and right hand or all
1:49:59the
1:50:00fingers um so this is kind of cool like
1:50:04first person kind of
1:50:06view and this is like from the up so
1:50:11looking
1:50:12down um up facing device from the
1:50:17side
1:50:19um this is kind of like makes sense
1:50:22soorry this example I'm going to
1:50:24show um Oculus hand tracking so here on
1:50:28the settings in the device we do
1:50:31have uh hands and controllers and you
1:50:34can see hand tracking and auto enabling
1:50:37hand tracking is actually toggled to on
1:50:41so switched on now that means I can just
1:50:43put down my controllers look at my hands
1:50:46look at my hands and there they
1:50:48are and I'm going to do pass through so
1:50:51you see what I'm looking at so here are
1:50:55my hands and um going to go back into VR
1:50:59and it's nicely mapped it's relatively
1:51:03accurate it feels pretty good um I am
1:51:06touching the correct fingers and this
1:51:09would actually be a gesture so it lights
1:51:11up becomes active and what we're going
1:51:14to do is well we can like scroll and we
1:51:19can scroll sideways as well so I'm going
1:51:22to close this and can scroll
1:51:25sideways and it feels relatively good
1:51:28you can see the rendering of my hand it
1:51:29still seems relatively
1:51:31accurate and um yeah so scroll up and
1:51:35down with this pinch to drag gesture and
1:51:38one thing that is a little bit uh if an
1:51:41app doesn't support hand tracking like
1:51:43here this is the browser so I can
1:51:45actually browse with hand tracking um
1:51:49most normal websites would work but then
1:51:51when you go on to look at whe this not
1:51:53works in A-frame then uh at this stage I
1:51:55have to disappoint you have to grab the
1:51:56controllers so we're going to do the
1:51:58path through controllers are over there
1:52:00going to grab them and um we're going to
1:52:04go into the VR immersive view there we
1:52:07go there it is so Hand tracking and uh
1:52:12A-frame per default doesn't work yet but
1:52:14this is probably going to happen very
1:52:15soon just not at the time of this
1:52:17recording I'm going to exit out with the
1:52:19Oculus button and this concludes my
1:52:21little demo on of um hand tracking and
1:52:23controllers so you can actually
1:52:24seamlessly go back and forth between
1:52:28hand hands and controllers pick up the
1:52:30controllers and press a button and there
1:52:32they are again and uh give it a little
1:52:36bit of rest and then look at your hands
1:52:38give it a bit of time for the
1:52:40recognition to work so we do have hand
1:52:42tracking now built into these devices
1:52:45and I talked about VR gloves earlier and
1:52:47the reasons I don't have them um but uh
1:52:50the point of VR gloves is not necessary
1:52:52the tracking of the hands was actually
1:52:54the creation of the heptic sensation so
1:52:57that you actually feel like you're
1:52:59touching something that can only be
1:53:01accomplished through through some tricks
1:53:03by actually mapping the physical to the
1:53:06virtual environment so that for example
1:53:09you are perceiving in VR a surface here
1:53:12and this happens to be also your desk
1:53:14that you are touching and if you can
1:53:17create these kinds of nice overlays
1:53:19between the physical and the virtual you
1:53:21actually do get a sensation of physical
1:53:25feedback even without varing additional
1:53:28things like gloves and
1:53:31whatnot so most of my examples were
1:53:34actually visual so what does it take to
1:53:36create a virtual reality and virtual
1:53:38reality is predominantly visual so in
1:53:41that sense it's not wrong however a very
1:53:44very very important part of creating
1:53:46immersion is actually the audio around
1:53:48the user so my last example will focus
1:53:51on Spa allio and here I could have
1:53:55chosen many examples one that I like is
1:53:57actually tiltbrush and I want to show it
1:54:00to you in two ways first without sound
1:54:03and then with sound
1:54:14[Music]
1:54:38so now we have seen the main ingredients
1:54:42of virtual reality both on the visual
1:54:44side and then on the tracking side so
1:54:46stereoscopic views or alternatively
1:54:48creating like a cave like monitor wall
1:54:51siiz displays around the user so inside
1:54:53a virtual reality headset if you will or
1:54:56for the headset we talked about
1:54:57steroscopic views we've learned a little
1:54:59bit about interactions and also hand
1:55:01tracking and then uh spatial or
1:55:04positional audio as well so these
1:55:06together actually create a rich set of
1:55:08Sensations and this is really what you
1:55:10want to accomplish uh so that uh your
1:55:13users feel immersed in virtual reality
1:55:16to close things I'm going to revisit
1:55:18beatsaber and we're going to look at it
1:55:20uh from the perspective of all these
1:55:22kinds of technologies that we've just
1:55:24learned
1:55:25about so in terms of the VR technologies
1:55:28that we' have just learned about beat
1:55:30saer really actually chooses those it
1:55:32creates a stereoscopic display through
1:55:34the headset it supports six degrees of
1:55:36freedom tracking which is really
1:55:37necessary uh you need to be able to move
1:55:40in space and also it does work actually
1:55:43with both outside in and Inside Out
1:55:45tracking but the example that I show you
1:55:47here is using the Oculus Rift s with
1:55:50Inside Out tracking and and down
1:55:55down yeah so I have to follow these
1:55:57directions when slicing oops and we
1:56:00should have used the other saber but
1:56:02that way you see what happens when
1:56:04things go wrong avoid
1:56:07those and
1:56:09then just
1:56:11blue Now
1:56:16red and this one you can cut into any
1:56:19direction and the synchronous cut are
1:56:23fun but I do like it when you get to
1:56:26slice like crazy like
1:56:28this and now comes my favorite part of
1:56:30the
1:56:34[Music]
1:56:37song any any down down down right down
1:56:43down left right down
1:56:46down
1:56:48right any
1:56:50any and you made one mistake so far but
1:56:53it's
1:56:54okay you're not
1:57:01pro and I have to be careful that I
1:57:04don't hit the cable in
1:57:06case I don't want it to disconnect
1:57:11again and
1:57:13[Applause]
1:57:15down and this one I always find it funny
1:57:17when they turn around and then you have
1:57:20to do like last minute changes
1:57:23and this one is cool slce SL SL
1:57:51[Music]
1:57:55we choose to go to the moon in this
1:57:57Decay and do the other things not
1:58:00because they are easy but because they
1:58:02are hard the Saturn 5 was a three-stage
1:58:04rocket each stage had a part to play to
1:58:07make the journey from the Earth to the
1:58:08Moon possible when you're ready to see
1:58:11how it all worked touch the glowing base
1:58:13of the Saturn 5 to initiate the launch
1:58:20sequence ignition sequence starts all
1:58:23engine running lift off we have a lift
1:58:26off Tower
1:58:28cleared reporting their roll and Pitch
1:58:31program which puts Apollo 11 on proper
1:58:32heading the third stage fired once to
1:58:35get Apollo 11 into a parking orbit
1:58:37around the Earth and after a few hours
1:58:38for system checks fired again to send it
1:58:40on its way to the
1:58:42Moon ready to move on just press the
1:58:45glowing Command
1:58:50Module I follow 11 this is Houston your
1:58:54go for
1:58:55separation as the lunar module was
1:58:57designed to fly in the vacuum of space
1:58:59there was no need for it to be
1:59:00aerodynamic weight was kept to an
1:59:02absolute minimum the walls of the cabin
1:59:04were no thicker than a few sheets of
1:59:06aluminum
1:59:07foil the astronauts didn't even have
1:59:09seats and piloted the ship standing
1:59:11upright in what was essentially a
1:59:13controlled fall from orbit
1:59:15[Music]
1:59:22when you're ready to proceed press the
1:59:24hatch on the
1:59:28[Music]
1:59:31eagle 7 hours later the commander made
1:59:34his way down the ladder to stand upon
1:59:35the foot of the lunar module okay we can
1:59:38see you coming down the ladder now the
1:59:40combined 360 lb weight of the commander
1:59:42and his spacit was a mere 60 lb in the 1
1:59:4516th gravity of the Moon making movement
1:59:47extremely easy
1:59:54all that remained to do was for the
1:59:56commander to take one small step for man
1:59:58one giant leap for man kind onto the
2:00:01surface of the
2:00:08Moon this footprint represents the
2:00:10culmination of a decade's work by almost
2:00:12400,000 people in what was one of the
2:00:15greatest periods of Discovery in human
2:00:17history and is still probably there to
2:00:19this day
2:00:33press the footprint when you are ready
2:00:34to end this demo and return to planet
2:00:36Earth
2:01:16in this segment we're going to talk
2:01:17about the key concepts of augmented
2:01:19reality I've done a similar version of
2:01:21of this for Concepts about virtual
2:01:24reality and now we are looking more
2:01:26specifically into Concepts to do with
2:01:28augmented reality I'll actually show an
2:01:31example scene here on the Hollens 2 um
2:01:36this is the hand interactions example
2:01:39that they have and it's actually really
2:01:40quite cool and you know a lot of things
2:01:43are going on in such an example so I can
2:01:45grab things I have hand tracking I can
2:01:49track multiple hands I can implement
2:01:51gestures I can resize objects I can use
2:01:56you know kind of like what you know from
2:01:58the phone is pinch to scale or
2:02:01Zoom um you can do these things in AR
2:02:03with your hand here I'm playing the
2:02:06piano um with a little bit of relatively
2:02:10uh good finger tracking so not just hand
2:02:14tracking but actually fully articulated
2:02:16finger tracking although the small
2:02:21finger here causes some
2:02:23issues this is an example of really
2:02:27going through a scene full with widgets
2:02:30and things to play around that Implement
2:02:33different kind of behaviors most
2:02:36important thing is obviously how I'm
2:02:38experiencing this content in the real
2:02:40world it's placed it's anchored it's
2:02:42like stable it is there I'm using a
2:02:45augmented reality headset the hollowland
2:02:472 is like on my head and I'm
2:02:50experiencing this merged view of the
2:02:54virtual content it is kind of like an
2:02:56overlay onto the real world it's a
2:02:58composite View and this is really also
2:03:01some of the key definitions um of
2:03:03augmented reality so here I'm using a
2:03:06similar structure to this same uh type
2:03:09of information on virtual reality so
2:03:12we're going to start with the five uh
2:03:14W's you know the who what why and so on
2:03:18um but in this case specific to AR we're
2:03:21going to talk about the key Concepts
2:03:23what makes something AR augmented
2:03:25reality so there are some key
2:03:27characteristics we're going to learn
2:03:28about Technologies and we see lots of
2:03:30demos uh throughout I also have a video
2:03:33focused on Technologies um that is
2:03:35really separate from these Concepts and
2:03:37more more dedicated to demonstrating how
2:03:39these concepts are implemented in
2:03:41current
2:03:43Technologies so I think of this content
2:03:46here is like the primer to AR and before
2:03:48we actually jump into the Technologies
2:03:50it's important to get a of you know what
2:03:52augmented reality is really about and
2:03:54what are some of the key Concepts cuz
2:03:56frankly the technology doesn't always
2:03:58live up to some of these uh Concepts and
2:04:01um so it's important to separate those
2:04:03two concepts and Technologies are really
2:04:06the implementation of these
2:04:08Concepts so here's the five W's for AR
2:04:12the who the what the when the where and
2:04:14the why who interestingly uh is similar
2:04:18to some of the uh VR who in the space
2:04:22but I do mention Google and Microsoft
2:04:25again but apple and Magic leap are like
2:04:28two uh different companies here and so
2:04:32Google obviously had Google Glass and
2:04:35glass was like really created a lot of
2:04:38hype around augmented reality although
2:04:40technically uh if you really go strictly
2:04:42by the definitions by the key
2:04:44characteristics of AR Google Glass is
2:04:46actually not really AR it's just a it's
2:04:50like a heads up display just like a
2:04:52little c a little projector in front of
2:04:54your eyes U but it doesn't the content
2:04:57is not really like part of the real
2:04:59world it's more like an overlay an
2:05:02information overlay so um it is very a
2:05:05very weak form of AR I'm going to talk
2:05:07about that concept of weak AR as well
2:05:10and obviously some of the latest uh
2:05:12examples are a Kade and a core on our
2:05:14smartphones from Apple and Google and
2:05:17then Hollens 1 and two from Microsoft
2:05:20this is at the time of the recording and
2:05:23the magic leap one was still
2:05:26around
2:05:27so when you ask me when did this whole
2:05:30AR thing actually start well I would say
2:05:32there was a really early start then
2:05:34nothing and then this like wave of
2:05:37things suddenly it's everywhere so uh
2:05:41one of the very first experiences was
2:05:43the sort of demo class um
2:05:471968 uh kind of like a heads up display
2:05:50it had to be m mounted really to the
2:05:53ceiling so it was not head mounted it
2:05:55was ceiling mounted H in many ways like
2:05:59the submarine Periscope kind of thing
2:06:02and uh well then uh many many many years
2:06:06later in 2013 Google Glass came out and
2:06:09there was really a hype around oh AR is
2:06:11going to is going to be revolutionizing
2:06:13everything and as I said Google Glass is
2:06:16technically in many ways uh not really
2:06:19AR uh it doesn't abide all the uh
2:06:23definitions uh and the characteristics
2:06:26but okay let's just say it was and um
2:06:30Holland one I think was
2:06:33really
2:06:35another and I would
2:06:37say on the and I would say on the
2:06:39headsets uh space Holland one is
2:06:42definitely an interesting and key
2:06:44example here when it comes to AR and now
2:06:48the hland 2 is even uh more advanced and
2:06:51uh we'll talk a little bit about some of
2:06:52the differences as well and then in
2:06:552017 we suddenly uh saw like make AR
2:06:59making it into our smartphones and there
2:07:01was this idea that the vendors are using
2:07:04our smartphones to figure out all the
2:07:07kinds of issues with tracking and
2:07:10display uh perception so for example
2:07:14that virtual content should be occluded
2:07:16if it is occluded by a real physical
2:07:18object or a a human and this is
2:07:21something something that is only now
2:07:22being available in
2:07:2420 uh 19 2020 it's people occlusion with
2:07:28air kit 35 um is finally there and um so
2:07:33on the smartphones and so there was this
2:07:35idea that um the main vendors and the
2:07:38key players in the space are using the
2:07:39smartphone uh platform to figure out and
2:07:43iron out all these issues um and then
2:07:45really make it to their headsets and
2:07:47maybe wearable and always on AR is going
2:07:50to happen once once we have figured out
2:07:52all kinds of issues including obviously
2:07:53the display technology uh and battery
2:07:57for
2:07:59example so when you ask me where is AR
2:08:02well it used to be just in research Labs
2:08:03you could come visit Michigan and I
2:08:05would show you some uh AR things but now
2:08:08it's actually really in your pocket uh
2:08:10if your smartphone happens to be capable
2:08:12of AR core and AR kit um running those
2:08:16kinds of platforms and um it also used
2:08:19to be just an Enterprise context now
2:08:20Enterprise context are being retargeted
2:08:23um for business uh uh business
2:08:26opportunities so magic leap is now is
2:08:28the company is now targeting um the
2:08:31Enterprise context just like hollowland
2:08:33and Microsoft has done before um there's
2:08:36still not a lot of consumers that
2:08:38actually have access to AR headsets um I
2:08:40would say it's predominantly the
2:08:41smartphones but I would say now it's
2:08:43also with the
2:08:44consumers why I think there are some
2:08:47really cool opportunities uh that uh
2:08:49when it comes to different kinds of
2:08:50domains
2:08:52like for example education when you
2:08:53blend the the real and the virtual
2:08:56content and you do it in a way so that
2:08:58it like the the the real content and the
2:09:00virtual content they feel to be part of
2:09:03the same world I think that's when it
2:09:05really becomes a very effective
2:09:08experience so I said Google Glass is not
2:09:10really augmented reality and I often do
2:09:12this
2:09:13to you know have an interesting
2:09:15discussion around it and have my
2:09:17students figure it out so I give you the
2:09:19definition by the book reality we said
2:09:22it's combining real and virtual content
2:09:24it's a composite view not just uh visual
2:09:27objects I always say that although the
2:09:29paper on which this is based is a
2:09:31literature review from Zuma ATR 199 1997
2:09:35so really long time ago um and they
2:09:37mostly talked about uh visual content
2:09:40but I just say not just visual objects
2:09:42here uh it's interactive in real time
2:09:45this interaction don't think of it as
2:09:47like you can click and and talk not that
2:09:50kind of interaction but actually the as
2:09:52you the rendering is interactive in real
2:09:55time you you you looking around so the
2:09:57implicit uh part of the interaction that
2:10:00works uh and and obviously we the expit
2:10:04interaction is also supported like you
2:10:06clicking things but originally uh it's
2:10:09interactive and real time really meant
2:10:11it's like real time Graphics uh overlay
2:10:14on the real world it's according to the
2:10:16right perspective which means we have to
2:10:18figure out your pose uh the pose uh so
2:10:22orientation and position of the headset
2:10:24relative to the real world and it's
2:10:26registered in 3D and so this means it
2:10:29really is able to determine when a new
2:10:313D object comes in we we can place it in
2:10:35the world and it'll be aligned uh it'll
2:10:39be a p it's like anchored um fixed it'll
2:10:42be rendered according to where it is in
2:10:45the physical world and this uh is called
2:10:47registration
2:10:51so these are the three key
2:10:54characteristics now Google Glass
2:10:56obviously fails to meet which one well
2:10:59the uh registration in 3D it is really
2:11:02just a heads up display um the content
2:11:05that it displays to you is not anchored
2:11:07in in the real world it doesn't actually
2:11:09it's often times it's not 3D even uh the
2:11:12content itself and so um so think about
2:11:15it that
2:11:16way so in my mind a key characteristic
2:11:19of AR is really blending of environments
2:11:22both the physical and the virtual uh
2:11:24world I can actually see those virtual
2:11:26objects as if they were part of the uh
2:11:28physical world that is also my my
2:11:31understanding of um the registration so
2:11:34I think of registration as hologram
2:11:36illusion and for VR I said in my mind VR
2:11:39has to do with an immersive task for AR
2:11:43in my mind it's about an information
2:11:45task um so the key thing is not
2:11:47immersion the key thing is actually
2:11:49giving me access to virtual information
2:11:52virtual objects um that really are
2:11:55placed in an intuitive and a meaningful
2:11:57way uh with the real world so me looking
2:12:01at some kind of virtual content that has
2:12:03no relationship to the real world is not
2:12:06a good use of AR and I would discourage
2:12:09from those kinds of experiences there
2:12:11really has to be a connection to the
2:12:12real world otherwise we might as well do
2:12:14it in VR uh or just as a 3D interface um
2:12:20on a screen so that's something to think
2:12:22about so next let's quickly talk about
2:12:26AR displays uh so for VR we had the
2:12:29headon display and we have this idea of
2:12:31the cave and for AR we have a few more
2:12:33options so we can have a headworn
2:12:35display like the hollow lens 2 uh we can
2:12:38have a spatial or projective display so
2:12:41really this is like projectors
2:12:43projecting into the world and usually
2:12:45some kind of camera or sensor tracking
2:12:47you to render the virtual content
2:12:49according to your perspective
2:12:51that gives you like this 3d effect this
2:12:54illusion uh this depth illusion and The
2:12:58Parallax and all those things that come
2:12:59with it according to your distance um to
2:13:03the
2:13:05content um we can have handheld AR so
2:13:07that's smartphone based or we can have
2:13:09monitor based ARS but this is more than
2:13:11just looking at a desktop screen that is
2:13:133D it actually um registers uh you and
2:13:19your location your pose relative to the
2:13:21screen and actually changes the
2:13:23rendering on the screen uh accordingly
2:13:25I'll show you some of these I show you
2:13:27examples of each of these so um in the
2:13:32next video we're going to look at a
2:13:33quick example from our XD project which
2:13:36has really focused on bridging multiple
2:13:38this AR display Technologies uh you will
2:13:40see a user wearing a holens um you will
2:13:43see a user uh playing with a kect based
2:13:47projection based we call this proam
2:13:49setup projector and Camera so the
2:13:51connect here is the camera so let's look
2:13:54at the first three AR displays I just
2:13:56mentioned um so in the following you
2:13:58will see a video uh of three of my
2:14:01former students are playing kind of like
2:14:03a laser shooting game inspired from Robo
2:14:05raade but using there three different
2:14:07kinds of a displays so you will see uh
2:14:11one student is wearing the hollow lens
2:14:13uh so it's a headworn display another
2:14:15student is playing with a room alive
2:14:17setup so this is actually a Pro Cam so
2:14:20projector and cam Cera setup with
2:14:22projection mapping so lots of things
2:14:23coming together that I'll explain a
2:14:25little bit more in in in in the
2:14:27Technologies Parts as well uh as a form
2:14:30of spatial augmented reality so she's
2:14:32she's not wearing anything she can the
2:14:34content is is rendered according to her
2:14:36perspective and then we see a smartphone
2:14:38user at the time um actually using
2:14:40Google Tango a platform that is now been
2:14:43superseded by a
2:14:46core so let's uh look at this uh example
2:14:49video there's quite a number number of
2:14:51things going on you see again a Hollens
2:14:54user more or less in the middle on the
2:14:55left we see the student with the
2:14:58projector and connect setup looking at
2:15:01the projection on the wall and you see
2:15:03inlays of what the Holland users sees on
2:15:06the left you obviously see the
2:15:07projection on the wall but it's not
2:15:08according to your perspective it's
2:15:10rendered according to her perspective
2:15:12and then on the right you see a Google
2:15:14tangle user is basically a form of
2:15:16smartphone based AR and so now they can
2:15:19play together it's networked but the
2:15:21point here is to illustrate these three
2:15:23um AR displays let's quickly focus on
2:15:26just the projection uh the pram setup
2:15:29the projection and the camera setup here
2:15:31the camera is a connect which can do
2:15:32skeletal tracking I'll give you a demo
2:15:34of this data it's really quite cool and
2:15:37that way we can figure out your
2:15:39basically your pose relative to uh the
2:15:42projector and then we can do the
2:15:44projection
2:15:45accordingly so now I'm going to show you
2:15:48uh the fourth type of air play which is
2:15:51Monitor based but keep in mind uh it
2:15:54still has to you know follow the key
2:15:56definitions the key characteristics so
2:15:58all three have to go on it has to be
2:16:00some merging of the real and the virtual
2:16:02world um it has to be interactive in
2:16:04real time and it has to be registered in
2:16:073D so I call this example AR mirrors um
2:16:12this is really what it is each of these
2:16:13displays that you see here on the wall
2:16:15this is in my lab and early days of my
2:16:17lab um is accompanied by a connect
2:16:19mounted on top and that te connect uh
2:16:22tracks me and then I can actually render
2:16:25uh a 3D model of the lab according to my
2:16:29well position and orientation towards
2:16:32the display this actually mimics the
2:16:35effect that happens when you look into a
2:16:36mirror I mean you looking around the
2:16:39mirror like this you would see more or
2:16:41less of the room behind you and that is
2:16:43exactly the kind of experience I wanted
2:16:45to create in this AR mirror examples now
2:16:48the the cool thing about monitor-based
2:16:50AR is that I don't actually have to wear
2:16:52anything but I have to look at a screen
2:16:55and that screen kind of like has to well
2:16:58show a version of me in this case there
2:17:00was a green ball if you noticed it was
2:17:02my head I wasn't really done with this
2:17:04research project at the time of the
2:17:06recording um obviously with the connect
2:17:08we can render a full skeleton and and
2:17:10show you some of these things later but
2:17:12so monitor based AR is also uh quite an
2:17:15interesting uh way the most common way
2:17:18to experience AR for us is on smartphone
2:17:21these days or if you have access to one
2:17:23of these guys behind me the H lens one
2:17:26or two or magic leap which I actually
2:17:28don't have
2:17:43here so now let's talk about some of the
2:17:46key Concepts I will feature augmentation
2:17:48and mediation um I will talk about
2:17:51strong AR and weak AR so I will
2:17:54distinguish those two uh two terms that
2:17:56maybe you have heard of they're very
2:17:58popular a mar based and Mar less AR and
2:18:01in fact I do think there's value in each
2:18:03so don't think of them as one
2:18:04superseding the other so after marab
2:18:07based and Mar L we becoming a little bit
2:18:09more Technical and I'll introduce you to
2:18:11two well acronyms here slam and Vio so
2:18:14slam simultaneous localization and
2:18:16mapping it's really this idea of the
2:18:18device figuring out both uh building a
2:18:21map of the 3D space and then finding its
2:18:23location within that 3D space so it's 3D
2:18:25position and bio which is really visual
2:18:29inertial odometry is this idea of using
2:18:32both the camera and the inertial
2:18:33measurement unit of your smartphone so
2:18:35the gyroscope and Exel and trying to
2:18:37figure out how uh the phone actually
2:18:40moves through 3D space so these are some
2:18:43key Concepts that I thought I would
2:18:46introduce you to because they're
2:18:47actually really quite important for Mar
2:18:49less AR and then I'm going to uh talk
2:18:52about tracking and registration and the
2:18:54field of view and plane detection and
2:18:57object recognition um because these are
2:19:00some of the things we can do based on
2:19:02some of the algorithms and techniques
2:19:04I'll introduce
2:19:08earlier so let's focus on augmentation
2:19:10and mediation
2:19:13first augmentation is really this idea
2:19:16of bringing in Virtual content as an
2:19:19overlay onto the real world so here I
2:19:22drew my own variation of mri's reality
2:19:25virtuality continuums so from reality to
2:19:28virtual reality and I drew in augmented
2:19:31reality I drew in augmented virtuality
2:19:34and I also Drew in mixed reality so
2:19:37augmentation is really the amount of
2:19:40virtual content that you overlay over
2:19:42the real world at some stage maybe
2:19:45roughly here is when it's mostly virtual
2:19:48content and not so much real world more
2:19:50and then and virtual reality we say is
2:19:53when you actually don't see the real
2:19:54world anymore it replaces the reality
2:19:56around you this is a simplification
2:19:58obviously now mediation is really along
2:20:02this spectrum and so the more virtual
2:20:06content I bring in the more I could
2:20:07actually mediate how you perceive the
2:20:10world so it is important to not just
2:20:14think about augmented reality as adding
2:20:16virtual objects into your view I mean
2:20:19this is what we most commonly do but
2:20:21there's also an idea of diminished
2:20:23reality so I can actually remove objects
2:20:26or uh have the world around you appear
2:20:29completely differently so Black Mirror
2:20:31if you will and uh this is when it
2:20:33becomes um extremely exciting but also
2:20:36very dangerous so augmentation always
2:20:39sounds like adding something and
2:20:41mediation really uh emphasizes the fact
2:20:45that we can mediate how you perceive
2:20:47your surroundings and we will see more
2:20:50and more of these things in the future
2:20:52when we're starting to see ads in
2:20:54augmented reality for example and uh you
2:20:57should be aware of that concept so just
2:21:01to give a few more examples uh on the
2:21:04augmentation side so typical examples
2:21:06are information overlays any kind of
2:21:08spatially anchored objects virtual
2:21:10objects that are like spatially anchored
2:21:11and appear like as as if they were part
2:21:13of the real world this idea of hologram
2:21:16illusion that I mentioned earlier
2:21:18spatial audio so even without anything
2:21:20visual uh it could also be we could also
2:21:23talk about augmented reality um if it is
2:21:27an audio that uh still uh you know
2:21:29abides by the rules of these three
2:21:32characteristics from before so you're
2:21:34just listening to background music on
2:21:36your um on your phone uh that doesn't
2:21:40actually it's not spatial audio but if
2:21:42that spatial audio reacted how if there
2:21:44were like a virtual audio Source in in
2:21:47the room then we would actually talk
2:21:48about augmented reality as well most
2:21:51however augmented reality is kind of
2:21:52like visual but don't forget about the
2:21:54audio as well on the mediation side I
2:21:58have beautification now that is nice
2:22:00right so think of all the Snapchat
2:22:01filters that make you look better and
2:22:04more beautiful I should have used one of
2:22:05those um or those that make you look
2:22:07younger um which I think in my case
2:22:10would be an example of diminished
2:22:11reality so apart from Snapchat filters
2:22:14um diminish reality as I said you could
2:22:17for example remove objects around me
2:22:20like the shelf behind me just gone and
2:22:22replace it with something else so we're
2:22:24really starting to mediate and
2:22:26manipulate uh the the perception of the
2:22:29world around us again this is Black
2:22:31Mirror material if you will and then
2:22:34this leads to this idea of dark patterns
2:22:37uh in the design oriented parts of this
2:22:40XR moo I will talk more about some of
2:22:42these T dark patterns and really when it
2:22:44comes to design ethics as
2:22:48well the next con I wanted to talk about
2:22:51is strong AR and also weak AR so this
2:22:55actually I can illustrate quite nicely
2:22:57here on this graph so where do you think
2:22:59strong AR is going to be along this
2:23:01graph it's not here because we would
2:23:03then not talk about AR anymore so it's
2:23:05roughly here so um weak AR is there and
2:23:08strong AR is there so weak AR is like
2:23:11really um what what Google Glass I would
2:23:15say it does you know it shows some
2:23:18content uh overlay on the Real World um
2:23:21but it doesn't actually really register
2:23:24it in
2:23:253D and so strong AR is really like all
2:23:28those things that I talked about earlier
2:23:30so here strong AR highly accurate
2:23:33tracking full semantic understanding we
2:23:35really understand the world around us um
2:23:37it supports natural or as they say with
2:23:40the holl to instinctual
2:23:42interaction which is an interesting
2:23:44concept and head-mounted um AR display
2:23:47is really strong AR these are examples
2:23:49of strong AR and when I look at
2:23:51something and it has no or imprecise
2:23:54tracking like Pokemon go for example is
2:23:57definitely weak AR it has no knowledge
2:23:59of the environment and that's something
2:24:01that they are improving now in the
2:24:03process of improving since the antic
2:24:05actually acquired 60 AI which was really
2:24:08all about reconstructing the environment
2:24:10so um it will be better and then no
2:24:13mostly implicit interaction so implicit
2:24:15really mostly camera based uh
2:24:18interaction that feels kind of like weak
2:24:21AR you're just looking around it doesn't
2:24:22actually feel like AED reality to many
2:24:25people and a handheld AR display a
2:24:28smartphone where
2:24:29realistically real where
2:24:32realistically um the only AR it just
2:24:36functions as a lens as a virtual lens
2:24:39into as a lens into this virtual world
2:24:42and everything else around you is the
2:24:43real world so it's like really not a lot
2:24:46of virtual content that you can see at
2:24:47any one time so um unless it's really
2:24:50well done in terms of like the size of
2:24:52the content it really fits that maybe
2:24:54tabletop AR experience um you would if
2:24:59any you compare it to a headset based AR
2:25:01it would consider it as more as of a
2:25:03weak AR experience next I'm going to
2:25:05talk about marab based versus maress AR
2:25:08we're going to look at an example of me
2:25:10using the holand one I'm going to use uh
2:25:14this photo as a basis for a few
2:25:15illustrations here and I'll show you Mar
2:25:19based AR so I'm drawing in a coordinate
2:25:21system because that's something I'll
2:25:23need for some of the examples I talk
2:25:24about so you really out of my head
2:25:27basically the the the the camera of the
2:25:30Hollin one of the main cameras you
2:25:32actually see um the z-axis so me walking
2:25:35forward would increase Z or backward
2:25:38decrease Z now that depends obviously on
2:25:41what kind of coordinate system it is
2:25:42Left-Handed or right-handed but um let's
2:25:45just say we're using a left-handed one
2:25:47here and then um X
2:25:50and Y now keep in mind that the Y I mean
2:25:53we don't know actually from which
2:25:55perspective this is rendered but one
2:25:57thing that I should say uh in this
2:26:00illustration um when you move your head
2:26:04like when you move your head like this
2:26:06or like that that doesn't bend the Y AIS
2:26:10accordingly this is something that I've
2:26:11seen with a lot of my students that
2:26:13causes some confusion no the Y is always
2:26:15like upwards okay so whether I have my
2:26:18head like this Y is still like this it
2:26:20doesn't go with the head um that's why
2:26:24we actually have a magnetometer and a
2:26:27gyroscope and uh those kinds of things
2:26:30in these devices so that they know where
2:26:33up and down is so keep that in
2:26:35mind and then I have the marker here so
2:26:39I use the example of the hero marker so
2:26:42it's like um basically this kind of
2:26:45marker and then me looking with a
2:26:46hollowland add that marker so that's
2:26:48what I'm kind of like illustrating here
2:26:50and so as I look at this marker with a
2:26:54device or with an app that actually uses
2:26:57marker based tracking what will happen
2:26:59is okay I'll start out at this location
2:27:01of 0 1.6 usually I mean I'm a little uh
2:27:05higher than 1.6 but the device initially
2:27:07thinks maybe
2:27:081.6 and um and then I did from that it
2:27:13establishes a coordinate system and then
2:27:15it figures out okay so the marker is
2:27:17really roughly from here it's the Z is
2:27:21an increase of 50 cm so we update this
2:27:24and it's a little bit lower than my head
2:27:26position so maybe 10 cm although that
2:27:28might be a little bit of an exaggeration
2:27:29here and um a little bit to the left and
2:27:32the marker is a little bit to the left
2:27:35and so because we we can establish this
2:27:38coordinate frame here we can also
2:27:40understand how the marker that we are
2:27:42seeing now uh we can actually figure out
2:27:45it's there and so most importantly the
2:27:48app would then use algorithms that are
2:27:50trained on this marker so the computer
2:27:51is trained on this and if this marker is
2:27:55like further away from the camera it's
2:27:57like smaller than this original one to
2:27:59one scale that was trained on or when
2:28:01it's closer to the camera then obviously
2:28:03it appears larger and um if it's like
2:28:07somehow distorted at an angle the camera
2:28:10can figure that out as well because it
2:28:12was drained on this marker now I don't
2:28:14really illustrate that all that here the
2:28:17most important thing is that there is a
2:28:18marker we call this a fidu
2:28:20because it's in the view of the camera
2:28:23it's actually visible to both the user
2:28:25and the camera usually sometimes maras
2:28:27are designed to blend a little bit more
2:28:29into the environment but uh often when
2:28:31we prototype and design things and using
2:28:34things like air toolkit based
2:28:36Technologies like arjs for example um we
2:28:39use these kinds of hero
2:28:41markers so they are quite popular so now
2:28:44marker L AR obviously gets rid of the
2:28:46marker how do we do that well the device
2:28:49kind of like as it moves through 3D
2:28:51space it actually Finds Its Own markers
2:28:56um if there were something like this
2:28:57let's say I have a hero marker poster in
2:29:00my uh room but I don't actually treat it
2:29:03as a marker and the device looks at this
2:29:05and builds like little feature points
2:29:07this is by the way also how it works
2:29:09based off a marker but in this case the
2:29:11device uh finds sent pixels that it sees
2:29:15like it finds like interesting points
2:29:17like let's say you are the computer for
2:29:19a second and you have to figure out how
2:29:22the camera if it were moving how it
2:29:24moved so you would pick a few points
2:29:26Behind Me Maybe you like the colored
2:29:27pens there maybe you notice those or you
2:29:29like the red shelf and you're looking at
2:29:31specific like corners of that red shelf
2:29:34you're doing some kind of edge detection
2:29:36to figure out how do these pixels move
2:29:38between frames and this is really how
2:29:40Market tracking actually works and you
2:29:43can if you don't have a prein marker
2:29:44that we trained on and you can use a
2:29:47combination of things like also the
2:29:49position and the pose or like the
2:29:51orientation of the camera and then all
2:29:54it all sums up and you can figure out uh
2:29:57how this uh device what we actually want
2:30:00to find out is how the device moves
2:30:02through space or how the marker comes
2:30:04closer to the camera and uh we can do
2:30:06both with some of the techniques um that
2:30:09I'll talk about in the next few slides
2:30:11so one concept that is often associated
2:30:13with augmented reality is this idea of
2:30:15scale of an augmented reality experience
2:30:17and the popular type of scale is the
2:30:20tabletop experience uh especially
2:30:22popular with marker based AR um because
2:30:25all what you often do is you have a
2:30:26marker right in front of you you put on
2:30:28a tabletop uh Mar L you could also do uh
2:30:32obviously just uh relative to a tabletop
2:30:35and then the user wouldn't be seated and
2:30:37that is a form of scale it's a very
2:30:40popular scale now with marker less AR we
2:30:42can actually be more free we don't have
2:30:45to place multiple markers in the room to
2:30:47figure out where the user is keep in
2:30:50mind that the device always needs to see
2:30:52these markers in order to understand how
2:30:53it moves through 3D space when it comes
2:30:55to marker based AR so now with marker L
2:30:58are we can find our own little features
2:31:00uh in the environment let's call these
2:31:02uh key points and um we can track them
2:31:06and so we actually enable a much larger
2:31:08scale up to the extent that we can
2:31:10actually make the whole world augmented
2:31:12reality and that is something that is
2:31:13currently being worked on uh through
2:31:16something called the AR cloud and this
2:31:18idea of mapping the entire space in real
2:31:20world sometimes uh with uh through games
2:31:23like games with a purpose like the idea
2:31:26of extending Pokémon go now and using
2:31:29all these Pokémon go players to map out
2:31:32the world for us um so it would be
2:31:35interesting to see who will own this
2:31:37data and how the data is actually being
2:31:40used in the future but we can
2:31:42essentially enable world scale AR
2:31:44applications as well the most popular I
2:31:46would say marker less is around the room
2:31:48and marker based is usually around the
2:31:49tabletop
2:31:50um although that is uh not a complete
2:31:54mental model but it's like a very
2:31:56popular way to think about uh these of
2:31:59these kinds of AR
2:32:14experiences so let's look a little bit
2:32:16more under the hood I just thought I
2:32:18need to introduce these two con Concepts
2:32:20slam and vile um so because uh visual
2:32:24inertial odometry is like this key thing
2:32:26that was associated with ar core and AR
2:32:28kit and so uh people have been talking
2:32:31about this all the time and if you want
2:32:33to participate in some of these
2:32:34discussions you want to be cool you
2:32:35should say things like slam and visual
2:32:38inertial odometry so slam is
2:32:41simultaneous localization and mapping um
2:32:44the algorithm works by tracking key
2:32:47points over a sequence of camera frames
2:32:50like image by image by image and then it
2:32:53actually tracks them so it really
2:32:55determines the trajectory of them and
2:32:58then it develops an estimate of uh these
2:33:00points in 3D positions so basically
2:33:02triangulation through multiple frames
2:33:04and it can actually from these uh
2:33:07estimates of these 3D positions it can
2:33:09actually determine the camera pose um
2:33:14which could have observed those points
2:33:16and if you do this for a sufficient
2:33:18number of points you you can solve both
2:33:20for the motion so how the camera moves
2:33:22through space and structure so you can
2:33:25do both localization and mapping of the
2:33:27space around you now this is an
2:33:29intuitive explanation of Slam slam has
2:33:32really been active research area for
2:33:35many many years actually dedicated
2:33:37conferences research conferences to that
2:33:40and um to be honest I'm not a computer
2:33:43graphics person or robotics person so
2:33:46I've have never really implemented a
2:33:47slam system myself now luckily have all
2:33:50kinds of Frameworks and tools that help
2:33:51us work around this so having this
2:33:54intuition for how slam works and what it
2:33:57does at the algorithmic level even
2:33:59though it's maybe not technically fully
2:34:00accurate is fully sufficient at this
2:34:02time to understand the concept so visual
2:34:05inertial adometry works by tracking uh
2:34:08the pose via the camera which actually
2:34:11does match a point in the real world to
2:34:13a pixel on the camera sens sensor and
2:34:15actually doing this each
2:34:16frame it tracks also the pose VI the
2:34:20inertial measurement unit um now the
2:34:22inertia measurement unit is not as
2:34:24accurate when it comes to position but
2:34:25it's very accurate when it comes to
2:34:27orientation so it processes both the
2:34:29exomer and the gyroscope data from your
2:34:31smartphone then it combines those two is
2:34:33the visual information uh of the pose
2:34:36and the IMU acquired pose and it
2:34:39analyzes those measurements from both
2:34:42systems H over time so it does this like
2:34:44a little bit over over over time uh and
2:34:47then determines the best estimate of
2:34:49three position when I say time this is
2:34:51still real time right this between
2:34:53frames but we have like 60 frames per
2:34:55second so we can do this really really
2:34:57quickly uh on uh smartphones these
2:35:02days so now that I've listed out slam
2:35:05and voo uh slam really can be
2:35:09implemented just like with the camera
2:35:12and doesn't the latest slam algorithms
2:35:14don't need a special Hardware like a
2:35:16depth camera we can just use the RGB
2:35:18camera the normal camera of your phone
2:35:23um and Vio actually so visual inertial
2:35:27odometry in many ways makes this even a
2:35:30little bit more robust because it also
2:35:31considers the IMU and it makes slam work
2:35:33on mobile which used to be a challenge
2:35:36um so we can do this really quickly
2:35:38really fast we can recover like one
2:35:41thing you should try out to mess up the
2:35:43tracking is like shake your phone during
2:35:45an AR experience and then see when it
2:35:47actually recovers so when it refin
2:35:50the the positions uh another thing that
2:35:53you'll notice when you run an AR
2:35:54application on your phone is that it
2:35:56always asks you to move around a little
2:35:58bit initially uh to establish tracking
2:36:01is what they say well this is when it
2:36:02actually builds these key points and and
2:36:04so you basically helping the application
2:36:07build a well a geometry uh so you're
2:36:12actually helping the application build a
2:36:13coordinate system of the world around
2:36:15you and then it'll work um as expected
2:36:19so this brings me to the two concepts of
2:36:21tracking and registration which I've
2:36:22just used as if they were clear but um
2:36:26let us illustrate this a little bit so
2:36:27tracking is locating the position of the
2:36:30user's Viewpoint when the view so is
2:36:33actually moving so you're moving like
2:36:36through the 3D space so like walking
2:36:38like here so and we can track how the
2:36:41position of your headset and therefore
2:36:44your head has changed and we can render
2:36:48the scene differently according to your
2:36:50New
2:36:51Perspective now so we can track how you
2:36:54move through 3D space and now
2:36:56registration in 3D is what I said is one
2:36:58of the key chars characteristics so of
2:37:01AR so registration is really the process
2:37:03of positioning virtual objects with
2:37:05respect to the real world so let's say
2:37:07you bring in your hand you're doing this
2:37:09hence gesture and then we're bringing in
2:37:11this Cube and we can use the hand
2:37:14position which we were tracking before
2:37:16as a reference point can relative to the
2:37:19that point can is can bring up the cube
2:37:22maybe so that it appears above your hand
2:37:24and we can use the last known position
2:37:27of the hand and we do have a coordinate
2:37:29system already established and so we can
2:37:32place the cube at this location so
2:37:34slightly above on Y and then copy paste
2:37:37the rest and it'll appear it'll be
2:37:39registered there and now if we continue
2:37:41to track the device we can obviously
2:37:43render the perspective on that Cube all
2:37:46the time um accurately so try and
2:37:49registration really work together and I
2:37:52will quickly jump to the field of view
2:37:54now the field of view is one of the most
2:37:57disappointing things that you will
2:37:58probably have with a headset uh even the
2:38:01latest generation of a headsets don't
2:38:03have a very large field of view so
2:38:05rather than you seeing the cube like
2:38:07this what would actually happen on a
2:38:09Hollens one where you have a 35° uh
2:38:12field of view like uh this angle here
2:38:16roughly you would see only por of that
2:38:20Cube it would be cut off and you would
2:38:21have to move your head upwards to
2:38:23actually see the rest of the cube and
2:38:25this can be a little disappointing
2:38:27because obviously everything else that
2:38:30is not in the field of you is just the
2:38:32real world so um so while in the
2:38:38promotional videos it always looks like
2:38:40while in a third person or promotional
2:38:42videos when they introduced the Holland
2:38:45for example it always looks like you
2:38:46would see this whole uh content what you
2:38:49you actually do see is only this cutof
2:38:51portion and so you need to think about
2:38:53how to build in visual cues into this uh
2:38:57area the field of view that is actually
2:39:01can actually be augmented with virtual
2:39:02content so you could like for example if
2:39:05this were an important object that
2:39:08should be in the entire view um you
2:39:11could for example draw in little markers
2:39:14that would say that would visual cues
2:39:16that would actually make the user move
2:39:18up their head
2:39:20but to be honest if I look at 3D object
2:39:21like this this is a 3D primitive I
2:39:24expect it to be a cube it could be a
2:39:25building it could be larger it may not
2:39:27be a cube it might actually be a larger
2:39:30box and I would probably explore it more
2:39:34by looking upwards in general the
2:39:36smaller the content the better uh it can
2:39:39be viewed with ar headsets and larger
2:39:41content is just like really difficult to
2:39:45um View and therefore also to conceive
2:39:47like really like how big the object
2:39:49really is um
2:39:53so what would be very hard right now is
2:39:56this really this idea of diminished
2:39:58reality like really porting transforming
2:40:01everything around me uh blending in ads
2:40:03or uh whatever I'll show you some
2:40:05examples later we're looking at hyper
2:40:07reality as one of those uh concept
2:40:10videos that I think are very interesting
2:40:12concepts of the future also a little bit
2:40:14scary to be honest when it comes to
2:40:16augmented reality in the future um but
2:40:18so many ways we can be happy that the AR
2:40:21headsets right now are actually uh quite
2:40:23limited and break uh this suspension of
2:40:25disbelief all the time just by being
2:40:27Limited in terms of the field of view
2:40:29for example but um this concept will
2:40:33still be around for a while the field of
2:40:35view really having uh perfect
2:40:37augmentation if you will like all around
2:40:39you with ar that will require a little
2:40:42bit more research I would say and also
2:40:44techn technological advances because
2:40:47even from the Hollands one to the
2:40:48Hollands 2
2:40:49I'm not even sure like I've seen
2:40:52different ways of calculating the field
2:40:54of view but and and one way is to say
2:40:56it's twice as large but um that's more
2:41:01like in terms of the area that you see
2:41:03but overall it doesn't actually add a
2:41:05lot to it so it's not like double or
2:41:08something like that so that's just
2:41:10something to keep in mind so for the
2:41:12next few years you might still be a
2:41:14little disappointed initially but then
2:41:17uh when we finally have all these fully
2:41:19augmented AR views we can also we have
2:41:23to be careful of all the scary scenarios
2:41:25that this
2:41:26enables finally um key Concepts uh are
2:41:29also plane detection and object
2:41:31recognition I think you you need uh to
2:41:34understand some of these Concepts to to
2:41:36understand how current AR devices
2:41:37actually work so I'll show you an
2:41:39example here of me using uh the Google
2:41:43measure app and what I'm going to do now
2:41:45is I'm actually going to measure the
2:41:47size of the screen that you see
2:41:50sometimes behind me in some of the
2:41:52angles so it's supposed to be a 57 in
2:41:55display so I'm going to measure and
2:41:59while the measurement is interesting as
2:42:01an app it's also an AR application uh
2:42:03what I wanted you to focus on is how it
2:42:05visualizes the plane it actually
2:42:07understands the orientation of this
2:42:10display so it sees a surface there it
2:42:12also sees a surface on the floor
2:42:14sometimes and it understands the normal
2:42:17so the orientation of that surface and
2:42:19it actually draws that into the view uh
2:42:24for me into the air view and this is
2:42:25actually good design and so we'll talk a
2:42:27little more about this in in other parts
2:42:30of this Muk
2:42:32specialization my uh second example here
2:42:36is Google Lens so when it comes to
2:42:38object recognition um I was playing
2:42:41around a little bit obviously here in
2:42:42the studio Google lens is often
2:42:45associated with like this way of
2:42:47translating menus or
2:42:49like translating from one language into
2:42:51the other but you can actually just like
2:42:53make this text on my screen you can copy
2:42:56paste it you can read it out um so you
2:42:58can do some of these things with Google
2:43:00Lens um which is really an impressive
2:43:02tool if you will um some basic AR okay
2:43:07so you can do that but you can also
2:43:09really detect objects uh so you can walk
2:43:11around and use the camera and in order
2:43:14to make AR really a thing so that we can
2:43:18start to interact with these objects
2:43:19meaningfully we need to understand what
2:43:21they are in this case it was able to
2:43:23detect uh the type of wheel that I was
2:43:25using and if the computer actually knows
2:43:28what kind of um equipment what kind of
2:43:31physical objects are around the user the
2:43:33augmented reality experience can be a
2:43:35lot more meaningful and so this kind of
2:43:39semantic understanding of the
2:43:41environment around us is still one of
2:43:43the key barriers right now to
2:43:46technological innovation it has to do
2:43:49both with really like building this uh
2:43:52understanding of the world so machine
2:43:54learning uh is like uh the term that you
2:43:57will probably think right now so that
2:43:59the computers can actually see we we
2:44:01call this computer vision they actually
2:44:03understand the computer actually sees
2:44:05like a human understands builds a
2:44:07perception and semantic labeling and
2:44:10understands that it is a chair and even
2:44:12understands what kind of chair it is so
2:44:14some of these things are still a little
2:44:16bit away I mean we see early examples of
2:44:18this but we not there yet when it comes
2:44:20to really semantic understanding I'll
2:44:22show you some examples of where we are
2:44:25on the iPad for example now uh we can
2:44:29detect people and we can use those as
2:44:32information to for example occlude
2:44:34content or we can like walk around I
2:44:37will walk around in the studio and I'll
2:44:39show you how it detects and understands
2:44:42what some of these objects are there a
2:44:44wall behind me there's a window opposite
2:44:46there's this seat or a chair that I'm
2:44:48sitting on
2:44:49so this is the end of the AR Concepts
2:44:52lecture and um I'm going to talk about
2:44:54AR Technologies really that Implement
2:44:56some of these Concepts um in a separate
2:44:59video so um I chose key Concepts that I
2:45:03think are key Concepts we started from
2:45:04the characteristics and then I went
2:45:06through a couple of key terms including
2:45:08down to some of the algorithms I hope
2:45:10that you thought this was interesting
2:45:12and useful I think understanding these
2:45:14Concepts is really important so that we
2:45:16also get a better sense of these
2:45:17Technologies and their limitations once
2:45:20those limitations are overcome however
2:45:23we may enter a very different uh world
2:45:26and uh discussion around some of these
2:45:28Technologies so in later portions of
2:45:30this Muk we really talk about ethics and
2:45:33also privacy and security concerns which
2:45:35really have to do a lot with both AR and
2:45:39VR so um I'll see you there
2:45:56in this video we're going to talk about
2:45:58augmented reality Technologies we're not
2:46:00just going to talk about them we're
2:46:01actually going to look at them okay so
2:46:03the whole idea is and we're not going to
2:46:04start with the Hollin but like this is
2:46:07one of the latest devices the whole idea
2:46:09is to understand uh all the technologies
2:46:12that are actually coming together in a
2:46:13device like this uh in order to get this
2:46:16to work it's actually been years of uh
2:46:18research and development before a device
2:46:20like the Holland 2 which this is appear
2:46:23can appear on the market and actually um
2:46:27push uh push us just a little bit
2:46:29further again in this space so I'm going
2:46:32to leave that here and we're going to
2:46:34talk about augmented reality
2:46:36Technologies so we're going to learn
2:46:38about a few AR Technologies and these
2:46:41are actually all enabling and building
2:46:43on top of each other we're going to
2:46:44start out with Market tracking we're
2:46:46going to look at motion tracking um in
2:46:49this case like tracking a device through
2:46:513D space without markers actually so
2:46:53markerless if you will we're going to
2:46:55learn about body tracking really uh
2:46:57learn tracking bodies we looking at the
2:47:00connect uh understanding uh the skeletal
2:47:03tracking of the connect and also looking
2:47:05into its depth sensing capabilities
2:47:08we're going to lead over to spatial
2:47:09mapping because that is something that
2:47:11the hollow lens can do it actually
2:47:12creates a 3D mesh of the world around
2:47:15you basically a copy a 3D copy if you
2:47:17will a 3D scan people often like to say
2:47:193D scan and then it actually does a
2:47:22little bit of SE scen understanding on
2:47:24top of it but I'm going to show you also
2:47:26where we are with some of the uh mobile
2:47:29platforms at the moment I'm going to
2:47:30show you AR kit uh and some of the scene
2:47:33understanding there we're going to talk
2:47:34about projection mapping because if you
2:47:36have all these things you can do also
2:47:39projection based AR so not just I I
2:47:41always want you to not just think in
2:47:43terms of mobile or headworn ar but also
2:47:47let's not forget about anything to do
2:47:49with spatial AR or monitor based AR
2:47:52finally I think we're going to talk
2:47:54about a key ingredient to actually make
2:47:55AR content appear as part of the
2:47:58environment and that is actually light
2:48:00estimation I'm going to show you an
2:48:02example there as well it's not exactly
2:48:04what I had in mind um but you will
2:48:06really see the difference that way so um
2:48:09let's uh get started so this lecture is
2:48:11really built around number of
2:48:13demonstrations for each of these
2:48:15Technologies and so I'm just going to
2:48:16get started with the first demonstration
2:48:18on marker tracking in this video I'm
2:48:21going to show marker based AR so I'm
2:48:23going to actually basically show this
2:48:25marker to the camera I'm using arjs here
2:48:30uh quick and simple and um going
2:48:34to just give you the demo so this is now
2:48:38me in a web browser and um it's also
2:48:42it's using a slightly different camera
2:48:43the one that I have over there and as I
2:48:47now bring in this marker into the view
2:48:49you can see how the camera is kind of
2:48:51like tracking that marker and relatively
2:48:55uh accurately detects the the
2:48:59pose and the orientation of that sheet
2:49:02of paper I have to pay attention that
2:49:04I'm not occluding it as soon as I'm
2:49:06occluding it it is has a hard time
2:49:08obviously seeing the marker so the way
2:49:10I've printed it here allows me to move
2:49:12it and it's also at a really nice size
2:49:15for the camera so then the tracking
2:49:17works pretty pretty well well uh
2:49:20relatively robust so we can even let it
2:49:22fall and well that didn't work so well
2:49:27but uh so this is at the core of a lot
2:49:30of the uh AR things that we show in fact
2:49:33I believe that MAA based AR um is very
2:49:37good for prototyping so there's
2:49:38something that we'll explore in in later
2:49:41segments of of the course this year as
2:49:44part of the
2:49:45xao and um yeah I just I'm just still
2:49:48excited uh they don't have to look like
2:49:50this they can actually be
2:49:52customized um they can look and blend in
2:49:55they can look more like the environment
2:49:57they can blend in a little more uh they
2:50:00they can look more like QR codes if
2:50:01that's what you want uh so you could EMB
2:50:04better URL and then have uh people both
2:50:07use it to open and launch your uh
2:50:10product or interface or whatever it is
2:50:13and then also use it to augment The
2:50:15Experience um so yeah I think this is
2:50:19pretty pretty cool stuff still and even
2:50:21though we have more advanced Mar less AR
2:50:24uh Technologies and devices I don't
2:50:27think that as a designer you can live
2:50:29without uh marker based um prototyping
2:50:34and Rapid exploration is just so much
2:50:36faster so that was my first demo on Mark
2:50:39tracking and now we're going to look at
2:50:41motion tracking what I mean by motion
2:50:43tracking is like understanding how a
2:50:45device like the smartphone or the hollow
2:50:48lens how they really understand where
2:50:50they are in 3D space without holding
2:50:54markers up in front of them so let's
2:50:56look at
2:50:58this so in this example we see my
2:51:01student Rob uh former PhD student
2:51:04actually go through the school of
2:51:06information AR core just came out and we
2:51:08started playing around with ar core and
2:51:10getting it to work on and in the web so
2:51:13this is a 3js scene those of you
2:51:15familiar with these technologies will
2:51:17notice anyway we going through the
2:51:19building we're actually walking up some
2:51:22steps we're walking around with a
2:51:23smartphone and we're dropping um
2:51:27landmarks if you will we call them
2:51:29breadcrumbs uh so that you can actually
2:51:31go back and find your way back um and
2:51:34you what this really demonstrate is how
2:51:36demonstrates is how accurate the
2:51:39tracking actually is and uh how well AR
2:51:43core as a smartphone without any special
2:51:46cameras um it's really just software
2:51:48based
2:51:49uh augmented reality and the tracking
2:51:51how well it can actually maintain the
2:51:53tracking uh let me show you a second
2:51:55example here uh so this is uh Rob again
2:51:59demonstrating one of his prototypes in
2:52:00the lab in this case we're just
2:52:02calibrating uh this display to the
2:52:05smartphone and what we can then do is
2:52:07actually write on it so the way this
2:52:09works is we are actually uh moving the
2:52:12smartphone just like the way we want to
2:52:14write and so it becomes kind of like a
2:52:17laser pointer if you will so we've
2:52:19played around with this quite a bit he
2:52:21was quite proud of this little prototype
2:52:24so um that was Rob and um that was one
2:52:28of his uh projects AR managerie that we
2:52:31worked on we were really exploring
2:52:33mobile based AR and uh what can we do
2:52:35with it and now that we have these tools
2:52:37so that was really interesting so so far
2:52:40I've shown you marker and motion
2:52:42tracking so so far we can understand a
2:52:44little bit of the environment not too
2:52:46much yet um for good AR we also need to
2:52:50understand quite a bit more actually we
2:52:52need to understand people um and I'll
2:52:56explain in a little bit more depth later
2:52:58why this is important so I'm going to
2:53:00show you quite an extensive demo here of
2:53:03the kect um I've pre-recorded this a
2:53:07little bit of you know karate I still
2:53:09got it in me and a little bit at least
2:53:13and uh this is mavashi Gary uh and um
2:53:17anyway so so I'm going back to the
2:53:19computer and I'm now going to replay um
2:53:23this recording this is actually a
2:53:24connect recording so the connect is a
2:53:28well a depth camera if you will and it
2:53:32can also do this kind of Skeletor
2:53:34tracking so what you'll see if you look
2:53:36at the left that's the original image on
2:53:38the right there is actually um a
2:53:40skeleton drawn in and that is uh the
2:53:44estimated skeleton sometimes these lines
2:53:46are a little bit thinner that's when
2:53:47we're not exactly sure because we can't
2:53:49see the joints and we can't uh be sure
2:53:52um but in any case the connect actually
2:53:54can do quite a few things um it also has
2:53:57a depth camera you see a big Shadow
2:53:59behind me that is because the camera
2:54:01couldn't see behind me um but you do see
2:54:04a kind of 3D reconstruction frame by
2:54:06frames in Real Time 3D reconstruction of
2:54:10uh my body and I'm going to play around
2:54:12with a few settings here and one thing I
2:54:14wanted to show you is this kind of what
2:54:17they call the surface meas with normals
2:54:19it'll uh it'll look very
2:54:23interesting it'll look like a plastic
2:54:25version of Michael uh what you also see
2:54:27me do on the right I'm actually moving
2:54:29around this 3D scene this is not like
2:54:32I'm not actually moving the kect I'm
2:54:34just moving the perspective on that
2:54:37recording and obviously I cannot see
2:54:39behind me because we didn't actually
2:54:41record it we didn't have a connect
2:54:43behind me so we couldn't actually see
2:54:45behind me and so the connect can
2:54:47actually do quite a bit of stuff here
2:54:49and quite a lot of
2:54:51things so what do we have for now well
2:54:54we understood Mo marker tracking and
2:54:57then we did motion tracking so basically
2:54:59the devices find their own markers and
2:55:01they're going through 3D space and doing
2:55:03slam or visual neral odometry that we
2:55:05learned about in the concepts then we
2:55:07learned a little bit about connect the
2:55:08connect is actually quite Advanced the
2:55:10reason I wanted to show you the connect
2:55:12is because if you think about it in this
2:55:14hollow lens is actually a connect in
2:55:16there now a connect with with which we
2:55:18are like walking around the uh demo that
2:55:21I just gave you the connect was
2:55:23stationary and it was pointing at me and
2:55:25basically did a 3D reconstruction of me
2:55:28and the World Behind me now if you have
2:55:31a connect in there and you're walking
2:55:32around with it on your head you're
2:55:35actually scanning the environment all
2:55:37the time and we are getting what is
2:55:39called a point Cloud uh of this or the
2:55:42spatial mesh uh we call the spatial mesh
2:55:46and this is just the 3D information so
2:55:47the hard part would actually be making
2:55:50sense of this kind of like spatial mesh
2:55:53and uh I'll show you spatial mapping
2:55:55first and then we actually also going to
2:55:57talk about scene understanding so really
2:55:59the sense making part I have x-rays
2:56:03coming out of my hands and when I tap it
2:56:08really reads uh the environment nicely
2:56:12I'm going to show you a little bit of a
2:56:13different demol later with this we'll
2:56:16see much better how that meshes being
2:56:18built but it really is quite impressive
2:56:23and of course the color effects they
2:56:24shows look really nice and this can also
2:56:27be uh done up into this area here and it
2:56:33really when you're like at a distance
2:56:35and you're doing this at the on the
2:56:36floor it's just nice how it travels
2:56:39through the entire room like a little
2:56:42wave we can
2:56:44actually uh go here into the software
2:56:49and we can save this uh mesh it'll
2:56:54download it for me and then I can open
2:56:58it and I'm going to quickly recolor this
2:57:01in a color scheme that I think makes
2:57:03sense and then when we zoom in here it
2:57:07does look a little
2:57:08bit um chaotic uh the traces of where
2:57:13the lights are and here's my capture
2:57:15station the seat is a little bit visible
2:57:19um and then the
2:57:21Shelf not so well captured over there
2:57:25the light is really interesting we're
2:57:27going to insert uh this chair that's the
2:57:30example I wanted to show so now we have
2:57:32a virtual chair can put it on the
2:57:35floor um okay so the hand obviously
2:57:39needs to stay in view of the device
2:57:41otherwise you canot track it and then
2:57:45make it a little bit larger
2:57:48and that's roughly where we going place
2:57:50it
2:57:53so now it's like a good chair it has it
2:57:57still seems a little bit higher than
2:57:59this one but it's fine and now what I'm
2:58:01going to do is I'm going to drag it and
2:58:03actually going to place it over there on
2:58:06the floor and a little bit like this and
2:58:10what I want to demonstrate now is when I
2:58:12sit down and I'm behind this chair you
2:58:15can see how it renders the occlusions
2:58:20around the black
2:58:22chair and this is is one of the hard
2:58:25things to keep with the illusion that
2:58:27this is a an object that is part of the
2:58:30real world we need to to be occluded by
2:58:32a virtual and physical objects obviously
2:58:35like this chair here so that was a first
2:58:38application of the spa of mesh we can do
2:58:40occlusion now and that is really
2:58:42important so that these virtual objects
2:58:44really feel like they are part of the
2:58:45world uh around us and front of us as
2:58:48well and included by both virtual and
2:58:51physical objects so that's uh pretty
2:58:53fundamental what I haven't uh figured
2:58:55out yet at this stages we okay we have a
2:58:583D reconstruction of the environment we
2:58:59can do quite a bit but do we really
2:59:02understand what we see well in the the
2:59:04answer to this at the moment is that the
2:59:05devices are well they're getting better
2:59:07at this but they don't actually yet at
2:59:10least at this stage and know exactly
2:59:12what they're looking at uh but they are
2:59:15getting better at this every minute so
2:59:18here I'm going to show you an example of
2:59:19scene understanding all right so we're
2:59:22trying to just walk a little bit through
2:59:24the environment here with the iPad and
2:59:28it can see all these different kinds of
2:59:32um surfaces so this is using the
2:59:37lar and has a
2:59:40relatively
2:59:41good understanding of these different
2:59:44kinds of environments and it does have
2:59:48basic semantics so this was
2:59:51characterized as a seat it's probably a
2:59:52bit hard to read the
2:59:55label um so this should be a
2:59:58table this is none none none doesn't
3:00:04know um basically when I look up there
3:00:08this is a window it gets that despite
3:00:11the light source which is unusual since
3:00:13this is a studio and here we have a seat
3:00:18so it gets dead right it knows where the
3:00:21floor is and this is hard because it's a
3:00:25glass
3:00:26table and it just doesn't know exactly
3:00:29what all these things
3:00:31are um
3:00:34otherwise pretty good so it's a wall
3:00:36that's pretty easy um so it does look
3:00:43nice and this is the semantic uh SE
3:00:48understanding and labeling so in this
3:00:51example I'm
3:00:53showing the iPad AR kit people
3:00:57segmentation and then occlusion as a
3:00:59result of this so there is a virtual
3:01:00base in front of me that you can see is
3:01:02not real but you see it on the iPad and
3:01:05then my hand actually like trying to go
3:01:07through this and obviously my hand
3:01:10should actually occlude um the virtual
3:01:12object when my hand is in front of it
3:01:15and then as I reach through the vase um
3:01:18at some stage my hand should appear
3:01:20behind it so what happens here behind
3:01:22the scenes is actually analyzing uh what
3:01:25the camera sees trying to find uh a
3:01:28person in there and then segmenting out
3:01:31this person so basically extracting this
3:01:33person from every image and figuring out
3:01:36the depth the distance of that well part
3:01:39of the human body to the camera and then
3:01:42adjusting the rendering accordingly so
3:01:44that the flower or the vs are actually
3:01:46appearing behind or in front of um the
3:01:51arm my arm so that's cool right quite
3:01:54impressive uh set of technologies that
3:01:56we have so far uh with that we can
3:01:59actually do a lot um so we have seen
3:02:01understanding and that will improve in
3:02:02the next few years so that's exciting uh
3:02:06what we're going to do last is actually
3:02:07look at two more things I wanted to show
3:02:09you a little bit of projection mapping
3:02:11um just so that we also understand uh uh
3:02:15spatial AR a little bit better that I
3:02:17introduced you to in the concepts and
3:02:19then finally we're going to look at
3:02:20light estimation have a little bit of a
3:02:22demo there as well so in this example
3:02:24we're showing two things we're showing
3:02:26we're actually in the lab here it's a
3:02:27little bit of an older recording on the
3:02:29hollow lens one there is a projection
3:02:32setup that is actually using projection
3:02:34mapping to go uh to render um between
3:02:38these two walls um so the image is
3:02:41actually distorted to make it fit if you
3:02:43think about how geometry works that's
3:02:45how it works in any case we also see a
3:02:47Hol lens a rendered cube in front of it
3:02:49and what we're actually using we're
3:02:51using the connect that is also there uh
3:02:54to track the position of the user adjust
3:02:56the projection accordingly so we can
3:02:58actually have these virtual cubes appear
3:03:00just like you would view them on a
3:03:02hollow lens but you don't actually have
3:03:03to wear anything um so I'm showing here
3:03:07a projection mapping uh solution uh
3:03:11Microsoft room alive to Kit coming out
3:03:13of Microsoft research and we set this up
3:03:16in the lab and it's actually quite cool
3:03:18so um this was a short demo but it
3:03:22showed a few things coming together and
3:03:25what you could also see is how uh with
3:03:28projection mapping and a relatively nice
3:03:30setup we can make it possible for people
3:03:32to see augmented reality even without
3:03:34wearing a hollow lens and the experience
3:03:36can be very similar if you noticed how
3:03:38the hollow lens cubes were basically
3:03:40occluding uh and working in a very
3:03:43similar fashion uh that as was the
3:03:45projection in the background
3:03:48in in reality of course you wouldn't
3:03:49want to have both like there's no need
3:03:51to have a headset and the projection uh
3:03:53but as a concept uh and so but I wanted
3:03:56you to be able to see these two
3:03:58technologies and compare them a little
3:04:00bit all right finally we're going to do
3:04:03light estimation this is really the last
3:04:04part we're going to look at and here I
3:04:07have um an example and all I'm going to
3:04:11do is place my phone on the table and
3:04:14then I'm going to find a light switch
3:04:15this is AR core elements as an example
3:04:19so I'm trying to find the light switch
3:04:20and you know as I switch it it actually
3:04:23does change uh it detects this so the
3:04:27phone can actually detect the light the
3:04:29amount of light in the room and can
3:04:30adjust um the content uh to make it like
3:04:33a night
3:04:35scene
3:04:36um and this actually makes it now fit
3:04:39better into this environment when it's
3:04:41dark and the brighter version as you can
3:04:44see here Blends in more nicely during
3:04:46the day and that's actually all there is
3:04:48to it in the future there will be better
3:04:50examples and more examples of light
3:04:52estimation right now not a lot of
3:04:54applications actually make use of light
3:04:56estimation this is also something we're
3:04:57still really figuring out I mean it's
3:04:59very hard like if you're inserting
3:05:01content over there but your phone is
3:05:03here and it has a light sensor it knows
3:05:05what the light is here it doesn't know
3:05:07what the light is over there so it has
3:05:08to estimate it how do you estimate this
3:05:11and why do we have to estimate it so
3:05:12that we can adjust the lighting and the
3:05:15rendering essentially of the objects
3:05:16that be the viral objects that we want
3:05:18to insert into the scene over there to
3:05:21make it fit the light conditions over
3:05:23there and so with that uh we could also
3:05:27talk about spatial uh audio um I did
3:05:30show that in the VR Concepts and
3:05:32Technology section so that applies here
3:05:34as well so I've walked you through quite
3:05:37a number of fundamental Technologies if
3:05:40you put them all together you really
3:05:41have a nice AR experience and uh that
3:05:44was the point of this whole lecture
3:05:59in this video I'm going to present well
3:06:01some ideas towards an XR decision tree
3:06:05we've just been introduced to VR and a
3:06:09both the concepts and the Technologies
3:06:11in a little bit more detail and uh as
3:06:14you will experience if you're working in
3:06:16this space um people will ask you for
3:06:19your advice on which technology
3:06:22technology to choose uh when and
3:06:26why and um now I happen to be somebody
3:06:29who who has been working on a lot of
3:06:31projects already and I have kind of like
3:06:34developed uh some intuition towards when
3:06:37to use AR and when to use VR and maybe
3:06:40when not to use either of them and this
3:06:43is something that I'd uh like you to
3:06:46develop as well so the idea for this
3:06:49lecture is to give you an a decision
3:06:52tree now it's not that simple I've been
3:06:55thinking a lot about how to best teach
3:06:57this and what what is this decision tree
3:07:00and uh well here is a version of this
3:07:02decision tree so the whole idea behind
3:07:06this lecture is to be able to answer
3:07:08some of these questions the ones that
3:07:09you have to ask yourself when you engage
3:07:11in a new project should it be VR or ar
3:07:15or is neither a good idea
3:07:18uh what kind of AR VR display and
3:07:21tracking Technologies should we choose
3:07:24what are the benefits of each and what
3:07:26are the challenges and what are the
3:07:29limitations uh which types of platforms
3:07:32and devices should we
3:07:35support um this is not straightforward
3:07:38um this is a increasingly complex and
3:07:40diverse technology landscape if you
3:07:42remember looking uh at this together
3:07:45with me it is really quite the
3:07:49worse and finally which tools should we
3:07:51use uh and then both for design and
3:07:53development there's a number of tools
3:07:55out there there's actually so many tools
3:07:57out there it is uh quite difficult to
3:08:01well maintain an overview of you're in
3:08:02the space and then for somebody new to
3:08:05this space it's even
3:08:07worse so when it comes to the question
3:08:09of whether to do VR or ar we already
3:08:12have some advice from very experienced
3:08:14people like Mark binghurst for example
3:08:17so so he says VR is ideal if there is
3:08:20lots of visual elements and I agree with
3:08:22him there uh 3D spatial interaction uh
3:08:25that makes a lot of sense if there is
3:08:27physical manipulation then he says
3:08:29procedural learning um but um I wanted
3:08:33to talk about interaction at infinite
3:08:36resolution so let's uh think about this
3:08:39for a second okay so here I have the
3:08:41quest and one of the controllers so lots
3:08:44of visual elements that makes sense
3:08:46right we're like going in there and it
3:08:48makes us feel like wow we can see so
3:08:51much content even though as you know
3:08:53some of the VR displays are smartphone
3:08:56based um essentially the screen real
3:08:59estate that we have is not larger but
3:09:01because of the agency in autonomy and
3:09:03this idea of really being able to look
3:09:05around and immerse ourselves with this
3:09:08content even walk through it and then
3:09:10interact with it espcially in 3D that
3:09:13really makes VR quite a platform for
3:09:16these kinds of experience this is really
3:09:18where it is IDEAL Physical manipulation
3:09:21that makes a lot of sense now we even
3:09:23have hand tracking even though I mean
3:09:26unless this like I'm not sure whether I
3:09:29want to rotate cubes and charts and
3:09:32whatever like with my hands I mean it
3:09:35looks cool in my Nority report but
3:09:37frankly it is not ergonomic and it's not
3:09:39really the most use usable experience in
3:09:42that way anyway we can do a lot with
3:09:44these controllers and so that's fine so
3:09:46physical manipulation if you will or
3:09:48direct manipulation is definitely
3:09:50something that I can see as well and
3:09:52then I said interaction at infinite
3:09:54scale now when it comes to AR there's
3:09:57usually some relationship between this
3:09:59virtual content and the real world that
3:10:01we see now in VR we don't actually have
3:10:03that connection to the real world and so
3:10:05I say interaction at infinite scale
3:10:07because we can make something super
3:10:09large and still kind of comprehended and
3:10:11see it despite the limited screen screen
3:10:14real estate that we actually have just
3:10:16because of how free we are with moving
3:10:18around in the space and looking at
3:10:20it and so yeah that actually then
3:10:24enables interaction at really really
3:10:26large scale we can also make something
3:10:28super tiny and super small and then
3:10:30still interact with it um and so I think
3:10:34that is where virtual reality is really
3:10:36fascinating on the other hand AR is
3:10:38ideal if there are some visual elements
3:10:42not obviously everything Visual and
3:10:44virtual because then it would be VR
3:10:47three spatial interaction and physical
3:10:48manipulation are just as good with ar if
3:10:50not better and then interaction is
3:10:53either at one:1 scale so between the
3:10:55virtual content and the real world
3:10:57really fits in there like life siiz
3:10:58content or less larger than a one to one
3:11:02scale usually doesn't make a lot of
3:11:04sense just because of the limited field
3:11:05of view that we usually have so it's not
3:11:08as immersive as we we can do it with
3:11:10uded displays like the the Oculus Quest
3:11:14and so that's why I say interaction not
3:11:17at infinite resolution but at one:1
3:11:20scale I think this is where it is really
3:11:21interesting because you get a you get a
3:11:23sense for what um like when I buy
3:11:25furniture I want to really have one to
3:11:27one scale I want to make sure it
3:11:28actually fits in here and that's where I
3:11:30think AR is really good now VR wouldn't
3:11:32be a good idea for furnish placement um
3:11:35if I'm in this room it doesn't make a
3:11:37lot of sense um to not see the room so a
3:11:41one to one scale is is I think the best
3:11:44application of AR at least in this
3:11:46example
3:11:48on the other hand if you're looking at
3:11:50when things are not ideal so VR is not
3:11:53ideal if there are many non-visual
3:11:55elements um yeah what do you actually
3:11:57show to users is it a white or black uh
3:12:00screen mostly that you show well you
3:12:02think about it uh lots of text to read
3:12:05or input um both reading and input are
3:12:08very cumbersome and tiring actually uh
3:12:11even with some of the best input methods
3:12:13that we have now including speech and
3:12:16it's also not IDE if there is a need for
3:12:18haptic feedback and this is something
3:12:20that Mark binghurst and I also agree on
3:12:24um I would just say it based on the fact
3:12:26that um unless there is some kind of
3:12:29mapping between the virtual quent and
3:12:31the physical world um any need for
3:12:34haptic feedback usually requires some
3:12:36additional user instrumentation or some
3:12:37additional device like blowing air at
3:12:39the user or something uh to produce this
3:12:42kind of heptic feedback and um these
3:12:45devices require a lot of calibration and
3:12:48need to be instrumented and installed
3:12:50very carefully so usually not ideal and
3:12:54then if there is a need for connecting
3:12:57with the real world then this is really
3:13:00a case for AR not for VR so keep this in
3:13:04mind and then AR on the other hand is
3:13:06not ideal if there is also many
3:13:08non-visual elements um although I do
3:13:11think that AR experiences can work that
3:13:12are mostly audio- based um we haven't
3:13:15seen so many examples of that
3:13:17when there is lots of text to read or
3:13:20input is also not good with
3:13:22ar and unless you have for example uh
3:13:27character recognition like I can write
3:13:28on a sheet of paper and that would be
3:13:30captured automatically in AR and then I
3:13:32can process it further uh that is
3:13:34actually exciting and in fact something
3:13:36we're working on at the
3:13:37moment uh when there is a need for
3:13:40haptic feedback I would also say AR is
3:13:42not ideal unless um there is actually a
3:13:46connection with the real world
3:13:48so you can map uh obviously if you're
3:13:50interacting with the real physical world
3:13:52around you um then I think it's fine um
3:13:56if if that AR application is really
3:13:57designed for that space you will
3:13:59basically have haptic
3:14:01feedback when it comes to htic feedback
3:14:03I like some of the design choices the
3:14:05Holland 2 made so for example now when
3:14:07you're opening a start menu you're
3:14:08looking at your wrist and then you're
3:14:09actually touching there so you have like
3:14:12this self actuated kind of force
3:14:15feedback because you're touching uh same
3:14:18with this uh gesture it gives you some
3:14:20resistance as you as you're touching
3:14:21your thumb so I think that's quite smart
3:14:24so we're using our own body to provide
3:14:25haptic feedback I think uh that's very
3:14:28interesting and that's where actually AR
3:14:30can work and finally if there is no
3:14:34connection to the real world around you
3:14:37um then really this is not a case for AR
3:14:40I think you should look at a VR instead
3:14:43so um before we really come to our
3:14:45decision Tre let's uh first review what
3:14:47kinds of Technology choices we actually
3:14:50have what options do we have before we
3:14:51make any decisions well obviously on the
3:14:54display side I've introduced you to a
3:14:55few different types of displays from
3:14:57head monitor to room
3:14:59size um AR displays are a variety
3:15:03handheld headworn spatials or projective
3:15:05or monitor based we covered those we
3:15:08also talked about tracking um 3 degrees
3:15:11of freedom versus six degrees of freedom
3:15:13we discussed about outside in and inside
3:15:15out we understand had tracking and um i
3:15:19gaze I mean so head gaze is really like
3:15:22most VR unless the device actually has
3:15:25built in ey tracking or you extended it
3:15:27with an ey tracker it actually really
3:15:28just tracks your your head uh and so if
3:15:32your application needs ey tracking then
3:15:34you have to think about that uh how do
3:15:36you actually provide that
3:15:39support and uh
3:15:42then uh well you can also think about
3:15:44what other things to track obviously you
3:15:45have controllers and in some devices
3:15:48like the Oculus Quest you have hand
3:15:50tracking built in you may choose things
3:15:53like leave motion as I showed you
3:15:54earlier uh so that you can attach to the
3:15:57VR headset and then provide a form of
3:15:58hand tracking as
3:16:00well on the AR side you have to think
3:16:02about on the AR side one of the most
3:16:04important decisions you have to make is
3:16:06whether you want to do Marker based or
3:16:07marker less AR it really has a number of
3:16:10technological implications and then you
3:16:13have to think about how much of the real
3:16:15world around you do you have to
3:16:16understand and so is plane detection
3:16:18sufficient or do you need 3D spatial
3:16:20mesh this usually requires a little bit
3:16:22more advanced technology sometimes even
3:16:24additional types of cameras including
3:16:26depth cameras like we have with the KCK
3:16:28the holens or the real
3:16:31sense and uh whether you need to track
3:16:33physical objects that's something you
3:16:35also have to think about so either you
3:16:37train the system beforehand you attach
3:16:39markers to those physical objects or you
3:16:41use some specific machine learning based
3:16:42libraries that allow you to detect some
3:16:44physical
3:16:45objects and and there actually
3:16:47additional considerations so I
3:16:48simplified a little but as you can see
3:16:51lots of choices to be made so these are
3:16:55the Technologies considered by
3:16:57themselves for each AR and VR and how do
3:16:59they actually translate through
3:17:00different kinds of platforms and devices
3:17:02if you're using offthe shelf devices so
3:17:05obviously on the VR side we can choose
3:17:08between something that is phone based
3:17:10like cardboard which is really on the
3:17:12low end up to something like maybe the
3:17:15latest Rift s Quest um Oculus Quest 2
3:17:19coming out soon at the time of this
3:17:20recording so um these are some of the
3:17:24most advanced uh virtual reality head
3:17:26mon displays that we have you could also
3:17:29look into caves but I just left that out
3:17:30for
3:17:31now so for AR we can do the same we can
3:17:35uh rank the Technologies rank order the
3:17:37Technologies and from the lowend
3:17:39Spectrum to the highend Spectrum so I
3:17:41would put marker based usually on the
3:17:43phone or some kind of webcam based
3:17:45applications um on the left maress if
3:17:48I'm thinking on on the phone it's kind
3:17:51of like medium end and then on the high
3:17:53end I would put varable AR so head
3:17:56monitor displays with six degrees of
3:17:57freedom so the hollow lens and the magic
3:17:59deep this is probably also the
3:18:01understanding that you have at this
3:18:02stage uh and this is what I would expect
3:18:04if I ask you how to how would you rank
3:18:06order the following devices and then I
3:18:08give you in some random order uh like
3:18:10Windows mixed world the Aras go and then
3:18:13Hollens and um smartphone
3:18:18so the technology landscape is even
3:18:20larger I said we need to think I said we
3:18:22need to pick tools so we can pick tools
3:18:26from the uh More Design oriented so 360
3:18:30authoring then maybe a little bit of 3D
3:18:33uh immersive authoring so working
3:18:35directly in VR or then using more like
3:18:37development tools uh that I list down
3:18:40here so then on the AA side um we can
3:18:44also look at libraries that are more
3:18:46dedicated get to specific types of
3:18:48tracking like marker based body based uh
3:18:50tracking uh libraries and tools here we
3:18:53can also look into immersive authoring
3:18:55so these I would say this is where I
3:18:56would draw the line maybe this is more
3:18:59designer oriented uh tools and then more
3:19:03the developer oriented tools so AR
3:19:05Foundation XR interaction for the tool
3:19:07kid for Unity AR cor AR kid or mrtk and
3:19:11some other like uh some other Solutions
3:19:13as well so these are obviously down here
3:19:17are really the Frameworks for
3:19:19development and up there are the tools
3:19:22that will help us in the earlier stages
3:19:24of design many times they actually
3:19:26Target
3:19:41designers we getting closer to the
3:19:43decision tree before I draw the decision
3:19:45tree I'll draw a technology Tre so I was
3:19:47thinking a little bit about this I'm
3:19:49going to divide this um on the left I'm
3:19:51going to put up AR on the right I'm
3:19:52going to put VR uh oriented things so
3:19:55first we talked about display and this
3:19:57is a little bit repetitive so I just
3:19:59went over those display types and then I
3:20:02characterize the a space into marker
3:20:04based marker less uh and then the VR
3:20:07space in terms of tracking into three do
3:20:09six do outside in inside out each of
3:20:12these are some combinations of these are
3:20:14associated with different kinds of
3:20:15headsets um and then between AR and VR
3:20:19we have often uh this in common that we
3:20:22want to uh track uh controller at least
3:20:25receive input from controllers even the
3:20:27Holland has a controller like a little
3:20:28clicker I usually don't use it I usually
3:20:30just rely on their hands um so
3:20:33controller andh hands both VR and AR I
3:20:36tracking becoming more and more
3:20:37pervasive now on these headsets and then
3:20:39tracking additional objects for which
3:20:41you often actually need additional
3:20:43libraries and tools so I don't really
3:20:45cover that um in this moo um too
3:20:50much and then on the tools side well
3:20:53I've just like laid out a little bit of
3:20:56a mental model a version of it more on
3:20:58the development front so from arjs and
3:21:01voria which are really around marker
3:21:03based application development to AR
3:21:05Foundation more like around smartphone
3:21:08based AR um and then for air kit and air
3:21:11core in this case and then we have mrtk
3:21:15so this is really the toolkit from
3:21:16Micosoft the mixity 2 it that's you know
3:21:18actually spans both AR and VR space so
3:21:21put in the middle and then we have the
3:21:23more VR oriented um both the sdks for
3:21:28some of the main headsets and then steam
3:21:31VR um yeah so these are some of the SDK
3:21:36the software development kit some of the
3:21:37options you have there so as I was
3:21:40drawing out this diagram I was actually
3:21:42thinking about two things so I call this
3:21:44an XR technology Tre uh I was thinking
3:21:47about strategy games like for example I
3:21:49like to play Starcraft and in order to
3:21:52uh give my units uh some certain
3:21:54abilities um I have to research some of
3:21:57them along the way so let me illustrate
3:22:00this quickly so one way to look at this
3:22:02Tech Tre in order to end up with ar
3:22:04Foundation you have to research AR you
3:22:07have to choose handheld and you have to
3:22:09research maress and then you know uh you
3:22:13actually have access to AR foundation
3:22:16and and
3:22:17um or let's do AR handheld and then Mar
3:22:22based and then you can end up with arjs
3:22:25or voria and then if you have Unity
3:22:28Affinity towards Unity you would pick
3:22:30voria and if you have an affinity
3:22:31towards A-frame web XR and the web based
3:22:34approach then you would choose
3:22:36arjs H there are many other tools here
3:22:38like ar2 kit I could have chosen as well
3:22:41in any
3:22:42case so this is the first way to look at
3:22:45it but the second way uh and the other
3:22:48thing that I was thinking as I was
3:22:50creating this diagram was actually
3:22:52Jeopardy so why do I say Jeopardy well
3:22:55if you know the game um the game is
3:22:57actually one where you actually provide
3:23:00you see answers and you actually have to
3:23:02ask the right questions so if you look
3:23:05at this tree this tree actually has all
3:23:07the answers but um in order to make use
3:23:11of the tree you have to know how to ask
3:23:13good
3:23:15questions and um so the best people at
3:23:18Jeopardy ask the right question so give
3:23:21you an example again so if you're
3:23:23looking at a 800 um it would show us AR
3:23:27foundation and so the question that you
3:23:29would ask is what is a markerless uh
3:23:33smartphone based or handheld AR toolkit
3:23:38and then you could say AR foundation for
3:23:40Unity and so that would the be the
3:23:42correct answer well if you're opening up
3:23:45VR 800 and it shows you as the answer
3:23:48steam VR then the question would
3:23:50be uh what is a headworn uh what is a
3:23:55platform for then the answer would
3:23:58be what is a virtual reality platform
3:24:02that allows you to program against hmds
3:24:06that support both three do or six do and
3:24:10outside in or inside out
3:24:12tracking um so basically all the popular
3:24:15headsets out there and then this would
3:24:18be uh the answer steam VR so this is
3:24:22opposed to the idea of showing you VR
3:24:24let say 600 and it shows you Oculus and
3:24:26then you would say and you're thinking
3:24:28maybe Oculus Rift okay or Oculus Quest
3:24:31and then you would say uh what is a
3:24:34software development kit that allows me
3:24:36to program against headmount displays
3:24:38that support 6 degrees of freedom Inside
3:24:41Out
3:24:43tracking and you could see the
3:24:44difference here steam VR really covers
3:24:46like this entire space uh just because
3:24:48it's a crossplatform but still just
3:24:51VR so and if the answer were mrtk so
3:24:55mixed reality toolkit the question would
3:24:57be what is a toolkit that allows me to
3:25:00program against both AR or VR handheld
3:25:05headworn um
3:25:07maress or 60 degrees of freedom virtual
3:25:11reality
3:25:13headsets um and then that would actually
3:25:16be mrtk because it does support from my
3:25:19experience at least uh I have
3:25:20successfully worked with mrtk programmed
3:25:23against a core and air kit smartphones
3:25:26um program against the hollow lens uh so
3:25:29headworn AR and also headworn Oculus um
3:25:33actually the rift s so that's why I said
3:25:3560 degrees of freedom and inside
3:25:37out so it is fun when you know all the
3:25:41answers and therefore can ask the right
3:25:43questions and that's really what I
3:25:45wanted to illustrate with this tree this
3:25:47tree contains all the answers and many
3:25:49of them are actually a little implicit
3:25:51and I can make these connections
3:25:53relatively easily because I've worked in
3:25:54this space for a while so in order to
3:25:57navigate this space I was thinking what
3:26:00are good questions that we could ask
3:26:02like in Jeopardy Style
3:26:17so finally here we are uh at the
3:26:19motivation for this exra decision tree
3:26:21that I was promising so I'll start by
3:26:23stting some of my goals before I talk
3:26:25about some of the non- goals my goal is
3:26:27first and foremost to provide a
3:26:29framework for for you to think about uh
3:26:32XR Technologies and actually to think
3:26:33through these Technologies and actually
3:26:35make technological choices uh I wanted
3:26:38to be systematic and technically sound
3:26:40but I didn't want to require a lot of
3:26:42technical background uh so you going
3:26:44through my AR and VR both Concepts and
3:26:47Technologies uh lectures should be
3:26:49sufficient for you to understand how
3:26:51this decision tree works you don't have
3:26:53to be a designer or developer
3:26:55necessarily I wanted to provide concrete
3:26:57examples but make sure they are
3:26:59representative now the problem with
3:27:01concrete examples is there are often
3:27:02some super specific tools and toits that
3:27:05a hardcore developer would point out to
3:27:07you um and I know many of these but I'm
3:27:10not sure how long those will be around
3:27:12often these things will be incorporated
3:27:14into larger Frameworks driven by the
3:27:16more key players and so if you're
3:27:19missing something here doesn't mean that
3:27:20I don't know about it just means that I
3:27:22decided not to pick it as an example
3:27:23because I thought it was not that
3:27:26representative in in in in the sense
3:27:28that it wasn't that well known or
3:27:30popular non Golds are actually to cover
3:27:33all possible technology choices in terms
3:27:35of platforms devices and tools to
3:27:38provide one universal decision Tre that
3:27:39works for every project so really this
3:27:41is not what I do here so my Approach
3:27:44here is to treat this as a design
3:27:46problem and uh I was thinking uh what is
3:27:50one of the tools that I like to teach in
3:27:52interaction design and I came back to
3:27:54this paper from the '90s on questions
3:27:56options and criteria um as a tool to
3:27:59think through a design space so this is
3:28:01a tool for design space analysis and so
3:28:03in fact I wanted to suggest possible
3:28:05questions options and criteria for you
3:28:06to think through as you're making
3:28:08decisions around an XR project so here
3:28:12we're going to start from a question so
3:28:14the Quee how to display something and
3:28:15then obviously we have a number of
3:28:17options so let's say something could be
3:28:18permanently displayed or something could
3:28:20be appearing based on some event from
3:28:22the system or some user interaction and
3:28:25then you can think about uh okay
3:28:27criteria for each of these um and then
3:28:30you make some connections and you
3:28:31actually assess each of these so for
3:28:33permanent if something is permanently
3:28:35visible then there is no user input
3:28:37required therefore low user effort um
3:28:40it's probably not that helpful if it's
3:28:42always visible so it might actually
3:28:44clutter the screen so I wouldn't
3:28:46actually positively assess screen
3:28:47compactness with this um option uh but
3:28:50it does provide continuous user feedback
3:28:52because it's always visible to the user
3:28:53so I would say yes that's a positive so
3:28:56appearing is um well it may require um
3:29:00more effort on the user so something
3:29:01needs to trigger this um it does help
3:29:05screen compactness and maybe I have so
3:29:07many other things that I need to display
3:29:08anyway so maybe that's the choice uh to
3:29:11go
3:29:12with um which is interesting uh it
3:29:15seemed like the other option seem to
3:29:17outweigh um this one but I go for this
3:29:20one because I consider this the right
3:29:21trade-off um and so this is uh depending
3:29:24on the situation so the next question
3:29:26then is how to make it appear and maybe
3:29:28as you try to answer that question you
3:29:30can also mitigate some of the concerns
3:29:32you may have around user effort and user
3:29:35feedback so I was thinking how can we
3:29:37turn this XR technology tree around by
3:29:41providing a set of questions options and
3:29:44criteria for XR for you to think through
3:29:46now this would even work better if you
3:29:49were looking at a specific scenario or
3:29:51actually example uh like a project of
3:29:54yours so I introduce you to these ideas
3:29:57a little bit more conceptually and
3:29:59generally speaking for now but then
3:30:01we'll actually apply some of these to
3:30:03some of our scenarios that I have
3:30:05prepared for you so here on the left I
3:30:07show a set of questions these are
3:30:09questions uh I'm not sure how generic
3:30:12they are but they are really some of the
3:30:14ones that often come up in my projects
3:30:16at least how do I display things so what
3:30:18kinds of uh display Technologies do I
3:30:20choose how do I track things so what
3:30:22kinds of tracking and sensing
3:30:24Technologies what kinds of pipelines do
3:30:25I have to build um how do I want to
3:30:28enable travel so navigation through the
3:30:30content uh this is especially
3:30:32interesting in VR how do I want the user
3:30:35to be able to manipulate things is it a
3:30:37form of direct manipulation is there
3:30:39some kind of indirect mapping how does
3:30:41that work and how do I enable
3:30:43collaboration a lot of my work uh
3:30:44involves multiple device and multiple
3:30:46users often and sometimes in different
3:30:49locations so I wanted to think about
3:30:52those questions and so this is a
3:30:55question that I often ask so before I go
3:30:57through each of these question in terms
3:30:59of options and criteria uh let's first
3:31:01think about how to actually fill those
3:31:03different kinds of columns so on the
3:31:06questions side as you work through this
3:31:08qoc you really have to ask questions
3:31:10about the design of specific elements so
3:31:13the examples here as I say are a little
3:31:14generic um so not as representative of a
3:31:18good qoc but the main thing is that you
3:31:21get the idea on the options side you
3:31:24really have to list out uh design
3:31:26Alternatives um so um this is not
3:31:29whether you want to make something bold
3:31:31or italic in fact you can make it bold
3:31:33and italic so these are not really
3:31:35design Alternatives these are just some
3:31:37related options for text styling so
3:31:39think about something that is more
3:31:41mutually exclusive really something
3:31:44permanent or something appearing
3:31:46uh that's uh that's a better way to
3:31:48think about options and then on the
3:31:50criteria side really established
3:31:52criteria that can be objectively
3:31:53assessed and weighed because obviously
3:31:56you want to look at these criteria and
3:31:58then you want to actually weigh them and
3:32:00make a informed decision uh about the
3:32:03questions uh and the design options that
3:32:05you have so on the display options side
3:32:08this can actually work at different
3:32:09levels first of all you can think really
3:32:11in terms of the specific types of
3:32:13display technology so headworn should be
3:32:16handheld AR VR and then if you go inside
3:32:20uh so AR and VR so it could be on the VR
3:32:23side um headworn or ar side headworn we
3:32:25could talk about heads up display if
3:32:28it's an AR phone we can actually place
3:32:30things in screen space um so in 2D we
3:32:33could do 360 on the phone or in VR we
3:32:36could do 3D and then the question is
3:32:38like where do we place content in the
3:32:40world so World space somehow attached or
3:32:43fixed to the camera uh or or like
3:32:46something that is following you so
3:32:48basically with a little bit of a delay
3:32:49as you're like looking somewhere else
3:32:50it's like coming with you so examples of
3:32:53follow me would be the main menus on uh
3:32:56a Hollens or an Oculus
3:32:59headset and then you can look at these
3:33:01options and assess them and I spelled
3:33:03out a few criteria here that I usually
3:33:05consider so user familiarity whether
3:33:08some of their previous knowledge
3:33:09translates to this design user Comfort
3:33:12um this includes ergonomic and usability
3:33:15considerations user Mobility so I mean
3:33:17both like the type of context in which
3:33:20they are uh running the interface so are
3:33:22they seated or are they like can they
3:33:24walk around uh I'll also consider
3:33:27accessibility uh uh as a main factor
3:33:31here then instrumentation of the user so
3:33:34how much do I need to put on the user in
3:33:35order to be able to uh you know do some
3:33:39of these things um so and I also need to
3:33:43think about
3:33:44so so I think about instrumentation of
3:33:47the user or the environment I think
3:33:50about uh some key characteristics in
3:33:52terms of display size so especially do I
3:33:55clutter the screen um the
3:33:58resolution um and then obviously how far
3:34:00I can see and things like that the frame
3:34:02rate and any other kinds of additional
3:34:04device requirements and then I make some
3:34:07positive or negative assessment and then
3:34:09I choose the option that I think is the
3:34:10best for a specific design
3:34:13element tracking so here are list out
3:34:15options obviously these are the familiar
3:34:17ones from three do six do inside out
3:34:19outside in marker based marker less face
3:34:21based body based and we have other kinds
3:34:23of tracking options as well I wasn't
3:34:25thinking so much uh in terms of
3:34:27technically tracking so magnetic or uh
3:34:31visual inertial uh or what kind of uh
3:34:35tracking you are
3:34:36thinking um so more like in terms of um
3:34:40how I wanted to track users in the
3:34:42headsets or devices uh that users are
3:34:44holding
3:34:46and then I'm thinking about criteria
3:34:48that we had from before like user
3:34:50instrumentation but also environment
3:34:51instrumentation especially for Mark
3:34:53based for example any kind of
3:34:55constraints physical constraints in the
3:34:56environment device requirements and if
3:34:59I'm requiring some exotic uh
3:35:01Technologies um and whether these
3:35:04devices are uh affordable and um whether
3:35:09like if I have an experience that really
3:35:11runs on the Holland I must admit to
3:35:13myself while it is cool a very small
3:35:16portion of the population will actually
3:35:18have access to it uh it's not so much an
3:35:20adoption question I think a lot of
3:35:22people would use the Holland if they
3:35:24could afford it so it's a Cost
3:35:27question so when it comes to travel I'm
3:35:29thinking about walking redirected
3:35:31walking teleportation menus and there
3:35:33are other options and I weigh them
3:35:35against an almost similar set of
3:35:37criteria when I think of manipulation
3:35:40think about doing with v controllers
3:35:42hands voice do I need eye gaze I can
3:35:45actually manipulate content with my eyes
3:35:46so scrolling for example when I reach
3:35:48the fold um marker based or physical
3:35:52object um so I use a physical object to
3:35:54actually manipulate some virtual content
3:35:56um so a physical prop this is something
3:35:58that is relatively common but then um at
3:36:00least in research but then I have to
3:36:02think about some of the more advanced
3:36:03sensing requirements and maybe some
3:36:06additional Frameworks and tools that I
3:36:07need for that kind of tracking and to
3:36:09support that kind of
3:36:11manipulation so when it comes to
3:36:12collaboration I'm thinking about
3:36:14collocated or REM mode and then I'm also
3:36:17this is in terms of place and then I'm
3:36:18thinking in terms of time so synchronous
3:36:20or asynchronous and for remote and
3:36:23asynchronous I'm thinking about
3:36:24networking requirements and for
3:36:26asynchronous maybe additionally About
3:36:28Storage requirements um because I need
3:36:30to persist some information in between
3:36:33uh the
3:36:34usage so now we actually do have a a way
3:36:38to think through this space it is not
3:36:40exactly um the tree maybe that I was
3:36:43hoping for I mean I drew out this
3:36:45technology tree but think of this
3:36:48decision tree as multiple stacked
3:36:51questions options and criteria trees
3:36:54just like I showed in the original
3:36:55notation I'm going to show you uh some
3:36:58examples based on some scenarios in the
3:37:02remainder here think about this scenario
3:37:05where you have a family uh come together
3:37:07and decide how to decorate the living
3:37:10room using various AR
3:37:12displays so one question we could ask is
3:37:15how to track track in this scenario so
3:37:18we can consider markab based Solutions
3:37:19we know it's mobile so we have Mar based
3:37:22as an option or Mar less although that
3:37:24does have some device
3:37:26requirements and then we can you know
3:37:29lay out some of the criteria here so
3:37:31marab based works on any device maress
3:37:34doesn't work on any device so no need
3:37:35for instrumentation well that is only
3:37:37the case with maress since it just picks
3:37:39its own markers essentially but in a
3:37:41marker based approach I would actually
3:37:43have to put a marker somewhere in in the
3:37:46space and then can I move around well
3:37:48that is a little bit more constrained
3:37:49with marker base because obviously the
3:37:51marker always needs to be in the view of
3:37:53the camera otherwise um I cannot
3:37:55actually really understand the pose of
3:37:57the device and the position and uh so
3:38:01that's why I do the positive assessment
3:38:03here and I decided to go for Mars and uh
3:38:06I don't know whether you would have gone
3:38:07the same way but that's something for
3:38:09you to explore I have a few more
3:38:10scenarios for you to think about so
3:38:13another one is the collaborative game
3:38:15that I showed earlier where we played
3:38:17together and uh we showed it in a
3:38:20collocated scenario but you could think
3:38:22this through a from a remote perspective
3:38:24so it should be synchronous so it's
3:38:26collaborative it's synchronous the
3:38:28people play at the same time at least
3:38:30that's what we had but uh it might be
3:38:32remote not in the same place um how
3:38:35would you design this um what kinds of
3:38:38Technology choices would you make there
3:38:40if you don't know about specific
3:38:41networking uh Solutions that's uh fine
3:38:46um that's always a big question as well
3:38:49so it's just like uh think of the actual
3:38:53questions to uh ask and think about how
3:38:56the qoc would change if I'll tell you
3:38:59that no they're actually not collocated
3:39:02they're actually remote you would
3:39:04actually ask about collaboration
3:39:05differently and in order to enable maybe
3:39:08both collocated and remote you would
3:39:11think about ways to implement that the
3:39:13second scenario is more in a public
3:39:15space here we have one version of it in
3:39:17a classroom where the teacher um wants
3:39:20to show the solar system and then the
3:39:23students uh can use a variety of AR
3:39:26displays for example or you just go for
3:39:28some of them but like um the idea is you
3:39:31would need to choose what kind of
3:39:33Technology you would put in place and
3:39:35then you would also need to be able to
3:39:38answer uh questions around who can see
3:39:41this how do we display things how do we
3:39:43track things you have a right of options
3:39:46you could go marker based or marker less
3:39:48if you go marker based you could also
3:39:49think about where you would place the
3:39:51markers how would this actually work for
3:39:53all the users so lots of things to Think
3:39:55Through um given that these are mostly
3:39:58seated um you might lean one way or the
3:40:01other um so there are some interesting
3:40:04trade-offs
3:40:07here my last example is also in a public
3:40:11space think uh in terms of a group
3:40:13meeting you're just going to a room on
3:40:14campus or like in the building that you
3:40:17work in and you want to work together
3:40:19collaboratively on some exercise and
3:40:22viewing some error content together so
3:40:25again how would you display things uh
3:40:27how would you track things how would you
3:40:29capture things how would you manipulate
3:40:32things can I actually move some of the
3:40:34things can I point to things is there a
3:40:36connection it's still an AR scenario
3:40:39here in fact most or all of these um are
3:40:43actually AR users in this examp example
3:40:45we also think about VR users as an
3:40:48alternative and could these VR and AR
3:40:50users even work together um it's I leave
3:40:54this up to you so think about the people
3:40:57think about uh VR and air displays and
3:41:01um if you really wanted to work on this
3:41:03this is actually what we'll do in the on
3:41:05Earth Track we're going to go I give you
3:41:07this exercise of actually thinking
3:41:09through some of these scenarios and uh
3:41:12putting together a an XR decision tree
3:41:15in the form of multiple stacked uh Q's
3:41:18thinking about the display and tracking
3:41:21Technologies thinking about
3:41:22collaboration and maybe other aspects
3:41:26that you want to think through in fact
3:41:28you are coming up with the questions and
3:41:30you are coming up with the options and
3:41:31the criteria and it's a good way to kind
3:41:34of like brainstorm and have some fun
3:41:36with the knowledge that we have so far
3:41:38and also soliciting some feedback from
3:41:41our PE Learners so I would really invite
3:41:43you to this uh exercise as part of the
3:41:46Honest
3:41:59track welcome back to the Honor track in
3:42:02this exercises part of the Honor track I
3:42:04would like you to consider a few
3:42:06different scenarios and then come up
3:42:09with a recommendation in terms of AR or
3:42:12VR technologies that you would be using
3:42:15or that you would suggest we should be
3:42:17using to support and Implement some of
3:42:20these scenarios so that's a little bit
3:42:21of fun and also it's actually a very
3:42:24pragmatic way for you to explore the
3:42:27whole design spectrum that we have in
3:42:30terms of XR Technologies and so I
3:42:33introduce it to one specific way in fact
3:42:35I've already talked about it in the
3:42:36lecture but now we are actually going to
3:42:38try it out and see how well this works
3:42:41how this is a little bit more of a uh
3:42:44formalism but it it should make the
3:42:47approach more systematic so I have the
3:42:49task description here for you so our
3:42:52task is to recommend XR Technologies for
3:42:55a given scenario in fact there are
3:42:57several scenarios to choose from and you
3:42:58can work on one of the scenarios so as a
3:43:00first step you should consider all four
3:43:02scenarios and pick the one that is of
3:43:04immediate interest to you you can then
3:43:07also later work on some of the other
3:43:08scenarios if you find that's fun I hope
3:43:10it's fun so you should consider
3:43:12implementations along the reality V
3:43:13reality Continuum but now we're talking
3:43:15in terms of Technology okay so work
3:43:17through the EXO technology tree which
3:43:19I'll show you in a second again and
3:43:21collect your options so the options is
3:43:24part of what we are going to consider as
3:43:26part of our qoc our questions options
3:43:29and criteria analysis so this stage you
3:43:31have obviously the options and now in
3:43:34step four I would like it to really
3:43:36strongly think about question
3:43:37formulations so what are the right
3:43:39questions to ask and then what are the
3:43:41criteria and this is when you actually
3:43:43do the qlc as systematically as possible
3:43:47finally in the fifth step and that is
3:43:49optional you can pick another scenario
3:43:51do the qoc analysis for that scenario
3:43:54and then compare and for most of the
3:43:56scenarios you have a few different ways
3:43:57of doing things so there's no generally
3:44:00right or wrong each of these choices
3:44:02that you can make as you walk through
3:44:04the exra technology tree will have an
3:44:07impact on reach and maybe user
3:44:10experience and scale and scope and so
3:44:13these are things to consider and so it
3:44:15would be interesting for you to consider
3:44:17alternative Solutions as well so in
3:44:20terms of what I would expect you get out
3:44:22of this exercise I feel like okay you
3:44:24kind of learn about a scenario based
3:44:25approach and that is the human computer
3:44:27interaction style to this kind of
3:44:29Technology selection and decision making
3:44:31and thinking but it's I think it's a
3:44:32good approach and so you would learn
3:44:34about a scenario based approach to
3:44:37selecting EXO
3:44:39Technologies you would also get a better
3:44:41sense of what questions to ask so as
3:44:43part of the Q I really Force you to
3:44:45think about the questions it's fine if
3:44:47you know all the Technologies but asking
3:44:49your clients the right questions if you
3:44:51are going to make recommendations to
3:44:54them and working with the client it's
3:44:56very good if you know what kind of
3:44:57questions to ask and so that allows you
3:45:00to really navigate that space and then
3:45:02obviously you can also consider
3:45:04different options and these questions
3:45:06will drive you and how you will assess
3:45:09these options so the criteria I would
3:45:12say that at the end of this activity you
3:45:13have a more comprehensive list of
3:45:15criteria to a decision making it's not
3:45:17complete it's not exhaustive but it's
3:45:19let's say more comprehensive and
3:45:21obviously you can build this out so here
3:45:23are a few pointers first of all this is
3:45:26the first scenario so home decoration we
3:45:29are looking at potentially a
3:45:30collaborative solution we are looking
3:45:33potentially at something in the
3:45:34augmented reality space you want to
3:45:38preview different kinds of home
3:45:40decorations you can think about bringing
3:45:42in new kinds of carpet furniture or wall
3:45:45decoration how would you do this what
3:45:48would be the kind of technology that you
3:45:50would want to use and what kinds of
3:45:53criteria should you think about so what
3:45:55are the options what are the criteria
3:45:57and what are the right questions to ask
3:45:58about this second scenario it can still
3:46:01happen in the living room but here the
3:46:02focus is on playing a game it's a
3:46:04collaborative game and potentially
3:46:05evolving different kinds of Technologies
3:46:08here I'm showing spatial AR actually in
3:46:10the middle so projection based AR with
3:46:13the connect this is is a hollow lens so
3:46:16a wearable ar headworn ar solution and
3:46:19this is a handheld AR solution so what
3:46:22if you wanted to support these they
3:46:24could play this game these players could
3:46:26be in the same room they could be
3:46:29collocated in the same room or maybe
3:46:30even in different locations just to mix
3:46:34up things a little bit would that change
3:46:35anything about the technologies that you
3:46:38would pick you would probably need
3:46:39additional Technologies to enable
3:46:41obviously the remote collaboration but
3:46:43that's something I'd like you to think
3:46:44about
3:46:45let me introduce you to the third
3:46:47scenario the third scenario is a
3:46:49classroom lecture so here there is a
3:46:51teacher or an instructor up front for
3:46:53example they talk about the solar system
3:46:56laying out a few planets for you to
3:46:58think about and obviously they would
3:46:59want their users so their students with
3:47:02any kind of AR or VR technology so we
3:47:05are showing a few different headsets
3:47:07here my student care daily prepared this
3:47:10sketch for us to give us a little bit of
3:47:12a sense of what kinds of Technologies we
3:47:14might might be looking at this could be
3:47:16in the AR space You could also think
3:47:19about a version of this scenario where
3:47:22you just like thinking you know what I
3:47:23think we should be going with virtual
3:47:24reality it would make things easier or
3:47:27not and so I'm not saying it would so
3:47:31something for you to think about anyway
3:47:33the whole goal of the scenario is to
3:47:36have an instructor up there talk about
3:47:39the solar system and have the students
3:47:42in this classroom be able to look at it
3:47:45in a version of AR or
3:47:48VR the fourth scenario and this is our
3:47:51final one for now and maybe we can have
3:47:53more scenarios in the future this is a
3:47:55team meeting okay so it's a little bit
3:47:57smaller in terms of the number of people
3:47:59it's maybe more collaborative so think
3:48:02about that ask questions about that and
3:48:05then also using both VR and AR devices
3:48:11uh could be mostly headsets but you
3:48:13could also expl expore how this could
3:48:15work with handheld AR devices or even
3:48:18spatial AR so I leave it up to you this
3:48:21is just a team meeting that's the most
3:48:23important thing you want to have
3:48:24collaboration there might be physical
3:48:26assets like these posters here from
3:48:29which you want to actually have
3:48:30information projected on a physical
3:48:33object that could be one way to
3:48:35interpret it this skeleton here could
3:48:37also be completely virtual I'll leave it
3:48:39up to you um you can interpret this
3:48:42scenario a little bit and the idea is
3:48:44really to make make a recommendation
3:48:45what do you think would be the best way
3:48:47to implement this so again you should
3:48:50consider the entire Spectrum so the
3:48:53entire reality virtuality Continuum all
3:48:55kinds of display Technologies and not
3:48:57just what you see explicitly in the
3:49:00figures that we Ed to illustrate these
3:49:02scenarios you can go broadly you can
3:49:05really explore each of these things that
3:49:07you learned about in the AR Concepts and
3:49:10Technologies as well as VR Concepts and
3:49:12Technologies lectures and so so really
3:49:15consider the entire Spectrum here and
3:49:17make a good technology recommendation go
3:49:20through the tech Tre both on the a side
3:49:22and on the VR side consider these
3:49:25different kinds of tracking Technologies
3:49:27on both sides not just the displays you
3:49:30don't have to think too much about the
3:49:32tools and how you would implement it
3:49:33this usually also makes a really really
3:49:36big factor in when I recommend
3:49:38Technologies but for now let's focus on
3:49:40some of those up there and not so much
3:49:42on development since we haven't really
3:49:43learned too much about it
3:49:45yet and finally the way I'd like you to
3:49:48approach this is with a systematic qoc
3:49:51analysis approach so questions options
3:49:53and criteria so you lay out the
3:49:55questions you lay out the options that
3:49:57you have this is an example that is
3:49:59really focused on a user interface
3:50:02design and then here are the different
3:50:04criteria and then how you would assess
3:50:06both positive or negatively assess I
3:50:08talked about this in the lecture and
3:50:10then finally obviously you would make
3:50:12your decision your recommendation based
3:50:14on on some of these criteria again the
3:50:17paper we are talking about here is the
3:50:19mle at all from 91 paper so questions
3:50:21options and criteria elements of design
3:50:23space analysis which I think works
3:50:26pretty well as I Illustrated in my
3:50:28lecture a couple of example questions
3:50:31options and criteria so ask questions
3:50:34about the design of specific elements
3:50:36this will help you so consider specific
3:50:38elements in each of the scenarios you
3:50:40just have to work on one scenario but
3:50:42obviously nobody's going to stop you if
3:50:43you want to consider Alternatives and
3:50:45also some of the other scenarios then
3:50:48for each of these questions you list out
3:50:50the design Alternatives so really like
3:50:52looking for really opposing ends when it
3:50:55comes to options not just related
3:50:57options as I explained in the lecture
3:50:59and then also really you establish a
3:51:01criteria that can be objectively
3:51:03assessed and weighed and then that way
3:51:05you make your decision so here was one
3:51:08example for laying out the options for
3:51:10different display Technologies and what
3:51:13that means in ter terms of user
3:51:15familiarity Comfort Mobility
3:51:17instrumentation Etc and these are just
3:51:20examples of how you can approach this
3:51:22task so I would really encourage you to
3:51:25approach uh this technology
3:51:27recommendation a little bit more
3:51:29systematically following the qoc
3:51:32analysis it might uh feel like we are
3:51:35overdoing things but that's important I
3:51:37think it's it's important you can start
3:51:39to cut Corners when you have done it
3:51:41picture book style that's how I always
3:51:44what I always say to my students in
3:51:45interaction design if you don't know all
3:51:47the steps in the process how are you to
3:51:50know which of these steps is really
3:51:52important and which of these steps you
3:51:53can Skip and often in because every
3:51:58project is different it might be that
3:52:00there is a step that you would usually
3:52:01skip but in this particular scenario in
3:52:05this particular case you really need to
3:52:06consider the step and that's why I'd
3:52:08like you to approach this in a more
3:52:11formalized more systematic way it'll
3:52:14help you train your thinking get you
3:52:16into the right mindset and more like a
3:52:19design thinking mindset which I think
3:52:21will be very useful as we're starting to
3:52:24think more about how we can shape the XR
3:52:27space and so there will be another
3:52:30exercises part of the honors track I
3:52:32have one more for you after this and
3:52:35then obviously also more exercises in
3:52:36the other courses that come with their
3:52:39own honor stack so I hope you enjoy this
3:52:42uh fine to complain I mean it's okay I
3:52:45can make changes uh but it would be
3:52:47great if if we could if you could engage
3:52:49with these scenarios and come up with
3:52:53some well interesting technology choices
3:52:57and again it's it's an exercise designed
3:53:00for you to really navigate this entire
3:53:03design space of EXA Technologies getting
3:53:05a better uh grip on on on on all the
3:53:08choices you can make it should it should
3:53:10be fun so I look forward to seeing what
3:53:13you're coming up with and uh I'll see
3:53:15you hopefully in the next exercise as
3:53:17part of the honors track as
3:53:34well so in this video we're going to
3:53:37watch me actually go and attend a
3:53:40virtual conference on uh virtual reality
3:53:43actually in in virtual reality and so
3:53:46that's interesting so the i e virtual
3:53:49reality conference i e VR in 2020 was
3:53:52supposed to be in Atlanta and then it
3:53:53actually had to be moved uh to online
3:53:56and remote uh due to covid-19 and so
3:53:59this was one of the first conferences
3:54:01that I have attended in this way and it
3:54:03was very interesting it's obviously
3:54:05interesting to go to a virtual reality
3:54:06conference in virtual reality so if any
3:54:10kind of conference is the one that seems
3:54:12to be well equipped for this kind of
3:54:14stuff then at least that's what we all
3:54:16thought then it's the virtu reality
3:54:18conference and I want to say that
3:54:19obviously the organizers did a really
3:54:21really good job it was interesting uh to
3:54:24see how it was attended and actually
3:54:27they had way more attendance at least in
3:54:29numbers uh than in previous years of
3:54:32this conference and um so as part of
3:54:34this video I want to share a little bit
3:54:36my my experience of what it was like to
3:54:37attend this conference I'm going to show
3:54:40you some excerpts of of of that and I'm
3:54:43going to reflect a little bit critically
3:54:45on that experience um at the end so I'll
3:54:49basically have a little bit of a diary
3:54:51study here so you can see some sequences
3:54:54um that I was recording now um keep in
3:54:58mind that I was attending the conference
3:55:00and the way I provide the footage is
3:55:01like as me going to a public event and
3:55:05there are some concerns about um filming
3:55:09at venues like this and so I was very
3:55:12careful and I was BAS ly uh just uh
3:55:17working with a student of mine we were
3:55:18going to some parts of this conference
3:55:20we're going for a walk I did meet some
3:55:23people I was very careful I did not
3:55:25include uh any uh anybody really and um
3:55:30but it is it is an interesting way uh to
3:55:34actually uh capture my little trip to
3:55:37the it EVR conference that's very
3:55:39important for me um and obviously there
3:55:41are issues with how this footage was
3:55:43produced this allows us to actually talk
3:55:45about some of these issues as well this
3:55:47is kind of like me looking at the
3:55:49website and then I'm like picking an
3:55:52avatar now this is the interesting part
3:55:55right where you addess yourself before
3:55:57you go to a conference in this case
3:55:58you're actually picking an
3:56:00avatar and I remember I was looking for
3:56:02a very specific Avatar so then this was
3:56:04like one of my first experiences here uh
3:56:07going to some of the rooms and exploring
3:56:10essentially in Mozilla hubs so it chose
3:56:12Mozilla hubs exploring Essen
3:56:14um what uh the world is like attending
3:56:19some of the speakers seeing some friends
3:56:21um and colleagues and then trying to
3:56:24make it up these virtual staircases and
3:56:28that was a little bit of a tricky thing
3:56:30to do and there are some reasons why
3:56:32this um venue is designed this way but
3:56:35it's funny that even in VR at least to
3:56:37me it's funny that even if you are
3:56:39things are not accessible that's to try
3:56:41to make contact eye contact and hand
3:56:43contact with Blair was a friend of mine
3:56:46that didn't work and then I met this
3:56:48person here and we kind of like shook
3:56:51hands in VR and that was a very
3:56:53interesting experience um it's kind of
3:56:56like meeting people was was trying to
3:56:57get a sense for the program what kind of
3:57:00sessions are on so somebody from the
3:57:02community actually contributed these
3:57:03screenshots and put them in VR bad
3:57:06design if you if you know what I mean
3:57:08and this is something I'll bring back as
3:57:10a negative example in in the design
3:57:12oriented course course 2 and then me
3:57:14picking a venue I mean actually a room
3:57:17so that's funny right you kind of like
3:57:19um have to still find your room even in
3:57:21virtual reality as you're going to these
3:57:23conferences in virtual reality like you
3:57:25would think that wow now that should be
3:57:27straightforward all we need is a menu
3:57:28but this menu is obviously significant
3:57:31Please be aware that visitors in hubs
3:57:33Lobby can be anywhere in the room
3:57:34without being seen by you so that is
3:57:36important knowledge there and then also
3:57:39they had some instructions around
3:57:41recording um so what I was trying to do
3:57:44here is going actually to a coffee break
3:57:46having a little bit of an interaction
3:57:48with some of the people there and I
3:57:50thought uh that was interesting I was
3:57:52trying to you know make a joke uh around
3:57:55uh tea and coffee and then I was reading
3:57:57my student uh sha and we actually had
3:58:00quite a bit of fun and so we were
3:58:04exploring a few things together here so
3:58:06I'll share a few
3:58:10moments for example we were looking at
3:58:13posters and and I was trying to find her
3:58:15and in fact I was actually occupying the
3:58:18same space as her that was really weird
3:58:20and I couldn't see her I mean um that
3:58:24never happens usually and um yeah it was
3:58:28flying around a little bit these are
3:58:29things you can do in Virtual Reality I
3:58:31was trying to get a different
3:58:32perspective on uh things and uh this was
3:58:35an interesting moment cuz sha was kind
3:58:37of like getting embarrassed for what I
3:58:38was doing just like when we would go to
3:58:40a real conference uh you know I am the
3:58:43kid in the room kind of thing and um
3:58:46yeah so I was playing around with some
3:58:48of the uh tools we have in vrs we have a
3:58:51Loza pointer here and I saw other
3:58:53students present some posters to me that
3:58:55was really cool they made good use of
3:58:56these tools so I wanted to try it out
3:58:58myself and and then we were going a
3:59:00little bit on on a walk and I thought
3:59:02that was interesting so I was trying um
3:59:04I think we had some audio issues here sh
3:59:07couldn't hear me and this is funny
3:59:09because these basic issues happen all
3:59:10the time
3:59:15yeah and then weig when we figured it
3:59:17out finally so yeah this was me
3:59:19attending i e VR in VR some you know
3:59:23excerpts and some sequences from from
3:59:26some of the recordings I've actually
3:59:27attended over multiple days which was
3:59:29interesting right getting up dressing up
3:59:32kind of like looking U professional and
3:59:36U both in VR and then also in in real
3:59:38life because I was know I was treating
3:59:40it as a job I I thought I wanted to
3:59:43record that EXP experience a little bit
3:59:45for my students so that they know what
3:59:47it might be like to to go to a virtual
3:59:49conference um this was probably a little
3:59:51bit special because it's a virtual
3:59:53conference in uh but it's a virtual
3:59:56conference about virtual reality so it
3:59:58was it was interesting um so when I
4:00:01compare the experience um I mean the
4:00:05things that I'm usually excited about
4:00:07which is meeting people at conferences
4:00:09or seeing friends I mean this was to be
4:00:11fair a conference that I don't usually
4:00:13attend and so I did know quite a few
4:00:15people and I connected with some of them
4:00:17it was an interesting setup there was
4:00:18like a slack kind of like Chat thing in
4:00:21parallel to some of the sessions and you
4:00:22could obviously then watch the sessions
4:00:24in Mozilla HS where you saw a lot of the
4:00:26footage but you could also just go to a
4:00:28twitch stream so was just live stream to
4:00:30Twitch and YouTube I think and uh so
4:00:33multiple different ways of actually
4:00:35accessing this uh conference which I
4:00:37think the organizers did a really great
4:00:38job but then really then divided the
4:00:41attention between some of these channels
4:00:43and some issues uh so when I met people
4:00:46they weren't always using VR headsets
4:00:48for example which is fine but some
4:00:51people had really issues with muting the
4:00:53microphon so you could hear everything
4:00:55uh some people were speaking very loud
4:00:58uh and so that was interesting just like
4:01:00in the real world I guess and so in in
4:01:03many ways um this virtual reality
4:01:06experience recreated a lot of the issues
4:01:07we have and you go to a normal physical
4:01:10conference so you might say it was
4:01:11actually kind of like close to the real
4:01:13thing right that a lot of people aim for
4:01:15that in virtual reality making it like a
4:01:19reality um so I mean um I did meet a few
4:01:23people it was interesting um the
4:01:26encounters were different definitely I
4:01:28didn't um necessarily see I I couldn't
4:01:31get a face right I just saw an avatar um
4:01:35it was interesting I I met I went to
4:01:38this meetup group that was cool met a
4:01:41few people from Europe did not include
4:01:43that footage cuz it was kind of like
4:01:45obviously private discussion and nobody
4:01:47was aware um of that so I did not
4:01:50include that the coffee breaks were
4:01:51freest style and I asked and announced
4:01:53whether it's okay and um so that's kind
4:01:57of okayish um so obviously my whole
4:02:02experience is is interesting both from
4:02:04the impact of covid-19 the use of XR
4:02:08Technologies and then also like
4:02:10attending a conference remotely and um
4:02:13and the fact that there is now footage
4:02:15about this like when I go to conference
4:02:17I take photos all the time um I take
4:02:20videos I have some friends and
4:02:22colleagues smile in there um look at
4:02:25interesting sites and so I was treating
4:02:27a little bit that we are virtual reality
4:02:31in VR uh experience the same way but we
4:02:34don't actually know exactly like is this
4:02:36now did I now do something that is okay
4:02:38or not um I was again careful with some
4:02:41of the footage um and so this is
4:02:45actually really good for us because we
4:02:46can talk about uh some of the issues so
4:02:50I talked I touched a little bit on
4:02:51accessibility I touched a little bit on
4:02:53Equity like having access to these
4:02:55devices the software they chose and all
4:02:57the channels they provided allowed a lot
4:02:59of people to participate
4:03:02accessibility is and was still an issue
4:03:05and in many ways um um wasn't fully
4:03:08solved um but I mean credits to The
4:03:11organizers this was still really well
4:03:13done pry and security I mean there were
4:03:15a couple of warning signs here and there
4:03:16but I don't know what other people did
4:03:17how they recorded how they
4:03:19participated uh and who they were and
4:03:22then on the social and um ethics side
4:03:26obviously it was very interesting very
4:03:28different behaviors in in VR it's not
4:03:30like really meeting people in the real
4:03:32world um I mean in many ways that design
4:03:35of that whole conference venue was
4:03:37stupid right why do we go in VR so the
4:03:39talks were actually the most boring part
4:03:41of the conference cuz everybody's just
4:03:43staring at this huge screen and I wonder
4:03:45whether people just took off their
4:03:46headset or never War One to begin with
4:03:48to attend a talk however posters and um
4:03:52really like posters were some of the
4:03:54most amazing content in this conference
4:03:56now in the real conference uh when I go
4:03:58to I do enjoy talking to students a lot
4:04:01and I like it when I see other
4:04:03professors also talk to students and go
4:04:05to the posters but unfortunately in the
4:04:07real conference experience posters are
4:04:09usually placed in some very bad location
4:04:11where nobody unfortunately nobody
4:04:13actually finds
4:04:14uh them and actually posters are often
4:04:16by early PhD students and and younger
4:04:20researchers and so they should kind of
4:04:22like be promoted they would be the new
4:04:24kind of researchers in our community so
4:04:26for me it was very exciting to see how
4:04:28VR actually had an impact uh in the
4:04:30sense that it made that very uh very
4:04:33cool the talks yeah I mean a few people
4:04:36made some uh more some better use of VR
4:04:38and actually had some slides in VR and
4:04:41gave some Demos in VR um but really the
4:04:44posters did the trick for me and what I
4:04:46also like was a little bit of fun venues
4:04:49like when sh and I went on a walk
4:04:52together I thought it was a cool idea
4:04:53doing that in VR racing each other a
4:04:56little bit so yeah so it did have it did
4:04:59have some fun components um it raises
4:05:01lots of issues though and those are the
4:05:03ones that that we're going to explore
4:05:05more uh covid-19 can be
4:05:09um uh didn't definitely had an impact on
4:05:12on on a lot of uh the research Community
4:05:15obviously a lot of professional work and
4:05:17and and most likely also you so um it's
4:05:22it's it's not the main issue I want to
4:05:24talk about here but it's just
4:05:25interesting because a lot of people now
4:05:27think um there was an extra hype around
4:05:30XR just coming out of it and so in some
4:05:33in total what do I think of that
4:05:34experience well I think the organizers
4:05:36did a really good job I overall don't
4:05:38think that it was the same experience it
4:05:40had the same effects on me and the same
4:05:43quality it was in fact very difficult to
4:05:45integrate it with my real life going on
4:05:47in the real world because it happens in
4:05:48parallel uh to this virtual uh
4:05:51conference so getting up different time
4:05:53zone a little bit and actually there was
4:05:55not so much deal here but I could
4:05:57imagine uh for some of the other
4:05:59conference that I'm going to attend
4:06:00remotely um it's going to be it's going
4:06:03to be challenging in any case I think
4:06:06that gives us lots to talk about and
4:06:07this is where I want to end
4:06:23in this lecture we're going to talk
4:06:24about Trends in XR uh so we're trying to
4:06:28look into the future we're trying to see
4:06:29where things are going what are some of
4:06:32the issues but in this one we really
4:06:34focus more on an Outlook I have a
4:06:36specific lecture dedicated to issues and
4:06:39there's so many issues we need to talk
4:06:40about so in this one we're looking a
4:06:42little bit into the future
4:06:44and when it comes to these things it's
4:06:46always um in important to first
4:06:48establish um a frame of reference or
4:06:52kind of like the coordinate system if
4:06:53you will so how does Michael think about
4:06:56things and given that I'm a human
4:06:58computer interaction researcher I really
4:07:00think of it in terms of three main
4:07:03entities people tasks and Technology I
4:07:07use this illustration here which I have
4:07:09adapted from my colleague Mark Newman
4:07:11and I really like it so much and and so
4:07:13it puts people on one side of the
4:07:16triangle tasks and Technology on the
4:07:19respective other edges and so this is
4:07:21interesting so in each Corner we
4:07:23actually have these entities sitting and
4:07:27we are exploring their relationship so
4:07:28human computer interaction which is
4:07:30again the area that I do research in we
4:07:33ask this question usually how do we use
4:07:35our understanding of people their tasks
4:07:37and Technical capabilities to design
4:07:39better systems so this is a design
4:07:43question but it's also from a technical
4:07:46perspective and so what is interesting
4:07:48about this triangle is how we can
4:07:50explore different sides of it so for
4:07:53example the relationship between people
4:07:55and tasks what do people do how can we
4:07:58support them so in ACI we often study
4:08:01and observe people before we intervene
4:08:03in any way and think about designing a
4:08:06solution we really first need to
4:08:07understand the problem we can also
4:08:09innovate on the people and Technology
4:08:12side so how do people interact what are
4:08:15the human constraints and how do we
4:08:17overcome some of them this area of study
4:08:19is really one that I focus on a lot in
4:08:22my research and it also directly relates
4:08:24to some other areas of research for
4:08:26example human factors engineering this
4:08:28really thinking about constraints so
4:08:30what is motion sickness in VR and how do
4:08:33we overcome this really would would be a
4:08:36very interesting topic here or another
4:08:38one is like text input so how do we make
4:08:40it possible for users to efficiently
4:08:42enter text t on an AR of VR device two
4:08:45really big open issues and I suspect
4:08:47over the next few years we will see more
4:08:50and more research in this space finally
4:08:52there's also the side between tasks and
4:08:55technology so in this area of research
4:08:58it's really more like scenario based and
4:09:00task oriented and it really questions
4:09:02the role of technology and user centered
4:09:05again but it's looking more at the
4:09:07activities of users and less so on the
4:09:10specific types of users so in the
4:09:13following I'm going to explore each of
4:09:15the sides of this triangle a little bit
4:09:17more and I look at it from each angle
4:09:20essentially from the people angle from
4:09:21the task angle and then also from the
4:09:23technology angle and we're talking about
4:09:26Trends so obviously in the arvr space
4:09:29right now what we do see a lot is single
4:09:32users okay so you you get a VR device
4:09:34you put it on your head and then that's
4:09:36it that's that's you now of course
4:09:39increasingly you have multi-user
4:09:41experiences so these users could be
4:09:43most likely somehow remote so it is more
4:09:47likely to see remote users in the
4:09:49multi-user scenario than lots of
4:09:51collocated users doing together a VR
4:09:53application now in classrooms this
4:09:55happens and I've seen it here at the
4:09:56University of Michigan relatively large
4:09:58classes using XR devices at the same
4:10:01time and so yeah we already kind of like
4:10:04there but what's the next step from
4:10:06there well it's social right the idea
4:10:08that when you go outside you see lots of
4:10:11people who own and use these exra
4:10:13devices now I'm not sure that we are
4:10:15there yet I mean right now when you go
4:10:17outside with your holl lens on your head
4:10:20or your VR device on your head and you
4:10:22shouldn't be walking then I wonder what
4:10:25would happen I suspect that Society is
4:10:27actually not ready yet and I also don't
4:10:29think that you will see a lot of other
4:10:30people doing this right now but it
4:10:32depends on the right technology and the
4:10:34right form factor coming out that
4:10:36affords you to do this essentially that
4:10:38enables you to do this the other
4:10:41observation that I have is that
4:10:42obviously right now a lot of these
4:10:43Technologies are very focused on
4:10:45able-bodied users okay we're trying to
4:10:47get this right but in the future I think
4:10:49it should be any user people with
4:10:51special needs and impairments could also
4:10:53be situational impairments right like
4:10:55you're using an arvr device in a
4:10:57specific context like on a bicycle or in
4:10:59a car really has very different
4:11:01constraints while this is more on the
4:11:03task side but basically it turns a
4:11:06perfectly able bodied user into somebody
4:11:09with significant situational impairments
4:11:12and most a your experiences break I mean
4:11:14there's really just no way that you can
4:11:16safely use them in some of these
4:11:18contexts so this brings me to tasks okay
4:11:21so when we design arvr experiences right
4:11:23now we're thinking very it's it's quite
4:11:26cute how we think about it so we can
4:11:28think about it in a seated or tabletop
4:11:31AR experience for example we can think
4:11:33about a standing VR or a room scale AR
4:11:38experience or we don't have that yet so
4:11:40much but we can obviously support on the
4:11:42ghost in scenarios in VR so as you're
4:11:45traveling on a train or airplane I mean
4:11:47we are beginning to see some of this a
4:11:49little bit and then obviously world
4:11:50scale AR like Pokémon go is really a
4:11:53thing and you can walk around and see
4:11:55people play around the world and
4:11:57actually together map out the world and
4:12:00so gaming and entertainment is something
4:12:02that it's not so exclusive to the living
4:12:04room we do do these things more out in
4:12:07public and so we can see how these tasks
4:12:10are actually going to be supported on a
4:12:12large L scale this has a lot of
4:12:15technological challenges as well all
4:12:17kinds of tracking issues that we need to
4:12:18figure out even just like moving from
4:12:21tabletop and seated VR to standing and
4:12:25then tracking users in six degrees of
4:12:27freedom in the room uh and supporting an
4:12:30entire room scale experience and from an
4:12:32a perspective to really understand all
4:12:33the objects that are in the room and
4:12:35then augment them so understand their
4:12:37geometry and maybe also what they are so
4:12:39classify them so in order to really
4:12:42enable these tasks meaningfully I mean I
4:12:45don't really give examples of tasks but
4:12:47I mean the typical scenarios that we
4:12:49often work with are like kitchen
4:12:52scenarios some kind of furniture
4:12:54assembly maybe exercising maybe
4:12:56entertainment in the living room maybe
4:12:58office work any kind of productivity
4:13:00scenarios any kind of virtual or online
4:13:03meetings remote meetings most of these
4:13:06scenarios take place in this
4:13:08configuration right now we don't do too
4:13:10much out there yet in fact most display
4:13:13Technologies like the Hollin work
4:13:14relatively poorly outside in in daylight
4:13:17and so that's really something we have
4:13:18to work
4:13:20on also like thinking a little bit more
4:13:23broadly about tasks context wise we are
4:13:26used to designing for very specific
4:13:28context very limited constraint context
4:13:30but in the future you will see
4:13:32applications embed and incorporate more
4:13:36techniques to actually scale experiences
4:13:38so you could imagine that a user is
4:13:41given a choice whether they want to run
4:13:43the application here on the tabletop or
4:13:45whether they want to bring it into the
4:13:47room and then as they maybe go from this
4:13:49room to the next room or maybe even
4:13:51going to the neighbor's house or whatnot
4:13:54uh we will be able to support
4:13:56transitioning these tasks and continuing
4:13:58these tasks and so we will be able in
4:14:01the long run to support more and more
4:14:03contexts not just the traditional
4:14:05tabletop seated uh environment and again
4:14:08this will require advances in technology
4:14:10but also really on the design side we
4:14:12really need to think this through and
4:14:13give more control to users finally
4:14:15Technologies okay so here I was thinking
4:14:18a little bit about three main trends
4:14:21that I am observing and some of them
4:14:23obviously are already quite evident
4:14:24there are transitions that have already
4:14:26happened so initially everything was
4:14:28kind of tethered just because of the
4:14:30computing power always everything had to
4:14:32be plugged in or using adapters like
4:14:34with cardboard around the phone and then
4:14:37we have started to see more Standalone
4:14:39dedicated devices like the quest is an
4:14:41example here the Oculus Quest or the Hol
4:14:43Dan which is basically a standalone
4:14:45computer so these are still the
4:14:48dedicated AR orvr devices what is
4:14:50happening at the moment is really
4:14:52building mostly AR not so much VR into
4:14:55devices a good example here is our
4:14:57smartphones how arcade and AR core in
4:14:592017 really changed a lot the smartphone
4:15:02and so we're adding around the
4:15:03smartphone so we seeing a lot more
4:15:05built-in Technologies in the next few
4:15:07years you will see robots coming out
4:15:09with ar I mean you will be able to
4:15:11inspect what you Roomba see and maybe
4:15:13guide them and so using AR Technologies
4:15:15I mean the robotics World in general is
4:15:18very close to augmented reality and so
4:15:21these technologies will benefit so more
4:15:23and more cameras and sensors um I mean
4:15:26lar which used to be a very expensive
4:15:29technology is now built into the iPad
4:15:32and so that's an interesting new trend
4:15:35and more and more cameras in the iPhone
4:15:37for example I mean a few years ago when
4:15:39Google came out with Google Tango it
4:15:42required a depth camera on a smartphone
4:15:44and additional cameras and that's why it
4:15:47actually died because most manufacturers
4:15:49couldn't afford to do this I mean afford
4:15:51not necessarily financially but from a
4:15:53design perspective so that really killed
4:15:55it and so then the software-based
4:15:57version of Google Tango is in many ways
4:15:59Google AR core so now that's happening
4:16:01and Google has actually been very
4:16:03careful with how many cameras they put
4:16:05on devices since then and so now apple
4:16:07is in this space and actually like adds
4:16:09more and more cameras to the iPhone and
4:16:11in the next few years we we will see
4:16:14actually devices coming out with all
4:16:15kinds of sensors and cameras and so we
4:16:17will see more of these built-in
4:16:19technology
4:16:20Trends so my second trend is actually
4:16:23closely related to this first from
4:16:25tether to Standalone to build-in which
4:16:26is from Hardware to software and now
4:16:29also to actually happening computation
4:16:31happening in the cloud so the example
4:16:33that I'd like to give here is AR core
4:16:35and AR kit obviously this is a form of
4:16:38software based built-in kind of AR that
4:16:41we not have in smartphones it used to be
4:16:43that you required additional cameras so
4:16:46Google started off with Google Tango and
4:16:47they put in a depth camera essentially
4:16:49like kind of like a Connect buildt into
4:16:51the smartphone and Google Tango was like
4:16:54this relatively interesting and
4:16:55promising really promising project
4:16:57actually but then it died because nobody
4:16:59wanted to have all this Hardware in in
4:17:01this in in the smartphones and so were
4:17:04only very few adopters L noo and Asus
4:17:07were like two of them producing Google
4:17:09Tango devices so then they kind of died
4:17:12and then AR core came out which was
4:17:14really a software based solution I to be
4:17:16fair AR kit came out first and then 3
4:17:17months later AR core came out so Apple
4:17:20versus Google in 2017 was was an
4:17:22interesting summer for our research team
4:17:24let's just say that and now we are at
4:17:27the stage where I mean we are really
4:17:30good with the computing power we have on
4:17:32mobile phones and desktops and all that
4:17:34kind of stuff but there's only so and so
4:17:36much you can do on a single computer no
4:17:37matter how many cores you have and so
4:17:40what you're beginning to see now is more
4:17:41and more cloud-based Computing that
4:17:43means that goes obviously there a lot of
4:17:46data transfer and storing things in the
4:17:48cloud like mapping out the world using
4:17:50building an air Cloud now that is part
4:17:53of that discussion but what I mean is
4:17:55actually that the rendering uh can also
4:17:57happen in the cloud and then uh it will
4:18:00actually just be streamed to your phone
4:18:01it's just like actually you watching a
4:18:03YouTube video but what you're actually
4:18:05doing is playing a game so you're using
4:18:07the sensors on the devices to collect
4:18:09all the information of how the device is
4:18:11moving uh in space then we're sending
4:18:13this uh via software to the cloud in the
4:18:16cloud we have multiple machines
4:18:18Computing and rendering things and then
4:18:20sending this high quality thing back to
4:18:22you we've seen examples of this with
4:18:24unre engine 5 um where really a lot of
4:18:27rendering is happening online uh so in
4:18:30the cloud uh holand uh has examples
4:18:34where um there's actually remote
4:18:36holographic rendering um and so the
4:18:39quality of the uh models is really High
4:18:43polygon count super high resolution high
4:18:45quality that's what Layman would say it
4:18:48looks really really sharp and we can
4:18:51achieve that through um cloud-based uh
4:18:54approaches and then my last Trend that I
4:18:57think is similar to the previous Trends
4:18:59but more focused on an apps uh
4:19:01perspective so right now we're thinking
4:19:04about whether designing to design an AR
4:19:06or VR app on on what you're beginning to
4:19:09see more and more is obviously
4:19:10incorporating a and VR modes into Apps
4:19:13but it's still AR or VR so most of us
4:19:16make that choice as designers and then
4:19:18that's it what I think in the future
4:19:20will happen more is that there are
4:19:22specific tasks in an application that
4:19:24benefit more from AR and then there are
4:19:26specific tasks that benefit more from VR
4:19:29and then so I would imagine that more
4:19:32and more apps are actually building in
4:19:34modes for both and actually thinking
4:19:37really about this cross reality
4:19:39transitioning so an example i' would
4:19:40like to give here is when I teach a
4:19:42lecture in the classroom in the in the
4:19:44in this future classroom this Vision
4:19:45that I have and I want to use XR
4:19:48Technologies and I can imagine that
4:19:50there is a topic let's say I teach about
4:19:52AR VR obviously then you will understand
4:19:54that the VR things I really want to show
4:19:56in VR and the AR things I want to show
4:19:59in AR and so I want to be able to
4:20:00transition to have the students you know
4:20:03look at look at content together with me
4:20:05and and and and and look around in the
4:20:07environment before we go into a VR task
4:20:10uh it could be a VR oriented design task
4:20:12where we really want to have all the
4:20:14degrees of freedom and this High you
4:20:16know field of view and really designing
4:20:19like crazy and then um we might actually
4:20:22later in the lecture we might switch to
4:20:24AR and right now like I mean we need to
4:20:28find specifically different apps I mean
4:20:30there are painting apps designed for AR
4:20:32and painting apps designed for VR um and
4:20:36uh nobody really thinks about bridging
4:20:39these realities yet and I do think that
4:20:42in the future um we will identify it's
4:20:45more like at the app level right now
4:20:46it's either an AR or VR app but in the
4:20:49future again as I say there will be uh
4:20:51within an app um or experience I don't
4:20:54like the term app actually too much but
4:20:55anyway there will be um tasks that be
4:20:58support uh with ar ofr modes and um I
4:21:03think what what this will lead to is
4:21:05that users depending on their current
4:21:09task but also their preference uh will
4:21:12be be able to choose um and obviously
4:21:14also their context so we be able to
4:21:16choose whether they want to use AR or VR
4:21:19and now that's going to be very powerful
4:21:21uh because then we can actually really
4:21:24enable productivity scenarios and we can
4:21:27be productive um in different kinds of
4:21:29contexts now obviously this can also be
4:21:31scary CU you know productive anytime
4:21:33anywhere we don't want to go there
4:21:35necessarily but on the other hand it
4:21:37would be interesting from a
4:21:38technological perspective so let's call
4:21:40this cross reality and I think cross
4:21:42Reality by the way is what I think XR
4:21:45should really stand for the ability of
4:21:47an application to transition between AR
4:21:50and V uh rather seamlessly really
4:21:53triggered by the user uh depending again
4:21:56on their preferences context and
4:21:59task so this was a just a first overview
4:22:02of the people Trends the task Trends and
4:22:04the technology trends
4:22:21so now I thought I'm going to expand on
4:22:23each of these technology Trends a little
4:22:25bit more so from te to stand alone to
4:22:27buildin we really started out from
4:22:29tethered VR air headsets so Rift and
4:22:32Vive and then in the airos space maybe
4:22:35one example is Magic Le chronologically
4:22:37this may not always be correct but now
4:22:39at least for a few years there was this
4:22:41trend around smartphones okay
4:22:42so we brought out the cardboard there
4:22:45was gvr and daydream which are actually
4:22:47both dead now Google and apple pushed
4:22:49both smartphone based AR with their AR
4:22:52core and AR kit platforms a new wave
4:22:55then happened which is Standalone VR AR
4:22:57headsets so the go and the quests come
4:22:59out and they're actually like really
4:23:01mobile they need to be charged but they
4:23:03are self-contained computers the latest
4:23:04and most recent Trends are really into
4:23:07integrated VR and AR capabilities so
4:23:09some early examples of this are
4:23:10spectacles and focal which are really
4:23:13just like glasses on your head
4:23:14spectacles didn't actually have a lot of
4:23:16AR capabilities it's just basically a
4:23:18camera on your head and the idea was to
4:23:20really learn how that might work from a
4:23:22user perspective that's at least what
4:23:24Snapchat said and focal uh really which
4:23:28was acquired by Google and it's going to
4:23:29be interesting where they are going with
4:23:31this but was essentially just glasses in
4:23:34many ways it was similar to Google Glass
4:23:37it didn't have a lot of AR capabilities
4:23:39yet but Google is already with glass in
4:23:42the space so I think we will be
4:23:44beginning to see more and more of these
4:23:46built-in glasses and and contact lenses
4:23:50in the future so on the second
4:23:52technological trend from Hardware to
4:23:54software to cloud-based here I already
4:23:57gave some of the examples I'm going to
4:23:59expand on this a little bit so we
4:24:01obviously have seen dedicated exr
4:24:03Hardware so Google Glass and Tangle and
4:24:06then connect and motion capture suits
4:24:08this is really more like I mean in the
4:24:10larger space of XR obviously this is
4:24:13also used in other contexts so like film
4:24:15making Etc but we have now seen a trend
4:24:18towards software based XR so a kid a
4:24:20core but there are also lots of
4:24:22Frameworks out there that we should
4:24:23really associate with these Technologies
4:24:26for example human pose gesture
4:24:28recognition Frameworks like open pose
4:24:30POS net and body pcks and then obviously
4:24:33also any kind of like face recognition
4:24:35or body base XR with software and
4:24:38machine learning a lot of machine
4:24:39learning going on in this space and so
4:24:42let studio and snap ml so really
4:24:45bringing machine learning into the
4:24:47community into the snap Community is an
4:24:49interesting move and is an example of
4:24:52this trend to cloud-based XR which is
4:24:56really I mean I gave already some
4:24:57examples but connect Azure the idea that
4:25:00Microsoft takes down the connect which
4:25:02was hugely popular with researchers and
4:25:04brings back a cloud-based and Azure
4:25:06based connect it's a depth camera where
4:25:08a lot of the processing and
4:25:11understanding so basically the sense
4:25:13making of what the camera sees the
4:25:16semantic reasoning all that kind of
4:25:18stuff happens in the cloud Holland has
4:25:20remote rendering support and so my last
4:25:23example here is 6D aai and Niantic so 6D
4:25:26AI was really an interesting startup
4:25:28that was really focused on 3D
4:25:29reconstruction on uh smartphones and
4:25:32really rebuilding the world as they see
4:25:34it and it's an interesting move
4:25:35obviously from Niantic which is behind
4:25:37Pokémon go to acquire that company and
4:25:39bringing this talent and knowledge into
4:25:41the into the companies so it would be
4:25:43interesting to see when Pokémon go
4:25:44players map out the world for us I mean
4:25:46not for us for Niantic and therefore
4:25:49Google how that will actually play out
4:25:52and well this is still an interesting
4:25:54example of this move from Hardware to
4:25:56software to cloud-based and so on the
4:25:59last Trend in technology when I said
4:26:02from AR or VR apps to AR VR modes in
4:26:05apps but still AR or VR to actually
4:26:08supporting cross reality so
4:26:09transitioning depending on task context
4:26:11and user reference so examples here are
4:26:13obviously dedicated VR meeting apps like
4:26:16alt space or second life AR furniture
4:26:20placement Ikea Place Wayfair and you
4:26:23have all kinds of other examples there
4:26:25Google Earth and Google Lens are really
4:26:28VR and AR specific examples and they're
4:26:32dedicated apps right so Earth is really
4:26:35like exploring the world in VR and
4:26:38Google lens is really showing you well
4:26:40identifying objects that you're pointing
4:26:42at with your camera and now we're
4:26:45beginning to see more and more
4:26:46integration of some of these features so
4:26:48Snapchat is obviously an allaround AR
4:26:50app and it offers obviously lots of AR
4:26:54modes but people don't think of it as an
4:26:56AR app I mean that's the interesting
4:26:57thing with Snapchat it's actually a
4:26:59messenging app and it integrates all
4:27:01these kinds of lenses as a goodie if you
4:27:03will or as like as an entertainment and
4:27:06content creation tool and so that's
4:27:08smart Amazon shopping has adopted AR
4:27:11Furniture previews for any AR product
4:27:13previews in view in your home kind of
4:27:15thing view in your space kind of thing
4:27:17what I find very interesting is how
4:27:18Google is integrating AR and VR into
4:27:21Maps so you have 360 previews you can do
4:27:24street view on maps and you can actually
4:27:27do what they call live mode now you can
4:27:29actually walk around with Google Maps
4:27:32and explore the world that way seeing
4:27:34like a view of the map superimposed as
4:27:37as you're walking through the world I
4:27:39also think it's very interesting how
4:27:41they are mixing results into Google
4:27:43search so when you use specific keywords
4:27:46where they have content prepared for AR
4:27:49preview you can actually then see 3D
4:27:51models at least and then also seeing
4:27:53like for example a tiger in your space
4:27:57and then we also have examples of apps
4:28:00that are supporting both AR and VR one
4:28:03that I think is promising in this space
4:28:05is actually spatial spatial is a viral
4:28:08meeting and Conference app if you will a
4:28:10meeting room that can be in VR or in AR
4:28:14and in fact users can use different
4:28:16devices and so that is interesting and
4:28:18then a lot of the ongoing resar this
4:28:20space for example one project that I'm
4:28:22working on or I have been working on
4:28:23with Mozilla is XR browser for a while
4:28:26now where we really exploring when
4:28:28you're browsing how to transition into
4:28:30VR and AR modes and how existing web
4:28:34apps and websites would need to be
4:28:36adapted what kind of support would we
4:28:37need to provide in the web browser so
4:28:40this was a an overview of the Trends in
4:28:42XR uh we looked at it from the people
4:28:45angle from the task angle and then from
4:28:47the technology angle and obviously I
4:28:49expanded quite a bit on the technology
4:28:51angle um this was important um because I
4:28:55wanted to think about these issues from
4:28:58all these different angles but when it
4:29:00comes to some of the really
4:29:03understanding really the reason behind
4:29:05some of the issues that I bring up it's
4:29:07important that we get to see more about
4:29:09these Trends and specifically in the
4:29:11technology area so uh for the rest of
4:29:14the week we're going to talk more about
4:29:17these issues and uh you can have more of
4:29:20a discussion around some of these Trends
4:29:22but I really want to uh bring up the
4:29:24issues and what is interesting is none
4:29:28of the trends at the moment indicate
4:29:30that there will be uh fewer issues there
4:29:33will actually be a lot more issues right
4:29:35we're moving from on device to in the
4:29:38cloud computing uh we're building
4:29:41everything into we're fabricating AR
4:29:44into everyday lives uh we will be
4:29:46integrating it into our clothes in the
4:29:48future I mean we already have body cams
4:29:51uh in kind of like in in a lot of
4:29:52settings not just the police wearing
4:29:54them and and so uh there will be very
4:29:58interesting issues um just as a result
4:30:02of some of these new trends as well and
4:30:04obviously with machine learning we can
4:30:06do so many things I only showed a little
4:30:08bit of body pcks which is based on
4:30:10tensor flow which is a way to do machine
4:30:14learning it's a library if you will and
4:30:17um yeah so obviously that we can look at
4:30:21a photograph and see there's a person in
4:30:24there and uh detect the gender uh
4:30:28leaving racial bias aside which we often
4:30:30have in in in current uh computer vision
4:30:33Technologies um yeah so or can estimate
4:30:37body weight and those kinds of things so
4:30:39I've seen all kinds of Solutions in that
4:30:40space
4:30:42um that is interesting and uh those
4:30:45transend themselves very well for a
4:30:48broader discussion around some of the
4:30:49issues so that's why I wanted to have
4:30:52this lecture here and that's why it's at
4:30:54this very uh place in in the book
4:31:13welcome to this lecture on key issues in
4:31:18XR uh I thought we should really bring
4:31:21up and discuss some of the important
4:31:24issues in this space there's actually a
4:31:25number of issues we've talked so far a
4:31:27lot about the concepts the Technologies
4:31:29the trends some of the opportunities if
4:31:32you will and for enthusiasts it's really
4:31:34quite an interesting uh space now um
4:31:38some of us I hope and that is you as
4:31:40well um should also think about this
4:31:43more critically and um I thought we
4:31:48should have a lecture dedicated to some
4:31:50of the issues and not just like the
4:31:52limitations of the Technologies no
4:31:54really the broader issues since these uh
4:31:57Technologies are actually making it into
4:31:59our daily lives we we need to think
4:32:00about uh what does that mean and what's
4:32:02the impact of these kinds of
4:32:04developments that we can witness and
4:32:06observe so before I get into a little
4:32:10bit more organized lecture uh where I'm
4:32:12trying to provide a conceptual framework
4:32:14of how to think about these issues and
4:32:16how to think them through um I thought I
4:32:18start with an obvious one and I hope
4:32:19it's an obvious one so um these glasses
4:32:23maybe you could tell um they were
4:32:25actually real now however um one said
4:32:29this is not
4:32:34real maybe you knew maybe you noticed uh
4:32:37maybe I'm not the kind of beard guy but
4:32:40um in any case
4:32:42this is potentially an issue because uh
4:32:45you never know that what you're looking
4:32:46at is actually the real thing and in an
4:32:50augmented reality application this can
4:32:52actually be uh quite well dramatic let's
4:32:55say uh there is the concept of
4:32:57diminished reality which is really
4:32:59manipulating mediating the environment
4:33:01around you so that it actually starts to
4:33:02look very different it is different from
4:33:04virtual reality though because it is
4:33:06still the real world that you're looking
4:33:07at so the goal is to really mediate it
4:33:10and change it just like for example in
4:33:12this
4:33:13case now you're getting confused right I
4:33:16mean yeah I mean this one is more like
4:33:18for my mom I really wanted to include
4:33:20this maybe it brings back some memories
4:33:23um okay so uh the technology that I'm
4:33:26using here is snap camera if you
4:33:28interested in this and this can be set
4:33:30up um so that it's you know kind of like
4:33:33connected to the software and then I can
4:33:36actually show these different kinds of
4:33:38lenses and uh so this lecture is really
4:33:40dedicated to the issues so let's start
4:33:42with providing uh you a little bit of an
4:33:44overview and then uh jump right into uh
4:33:47the
4:33:48specifics so to start out I was thinking
4:33:51what are the main classes of issues and
4:33:53I identified four different ones first
4:33:57one is design issues there's really a
4:33:59high barrier to entry so it's really
4:34:00hard for anybody who wants to
4:34:02participate in XR design and development
4:34:04to to make it into that kind of space
4:34:07there's actually few guidelines and best
4:34:09practices out there it's a moving Target
4:34:11it's always evolving it's radically
4:34:13changing so U we just don't know yet on
4:34:16the technical side there are lots of
4:34:18issues um here I picked like two
4:34:20prominent ones one is platform
4:34:22fragmentation the fact that we have so
4:34:24many different devices out there and a
4:34:27few key vendors trying to control the
4:34:30market in some way really leads to the
4:34:32same problem that we had with mobile
4:34:34devices in the past and actually still
4:34:36have to some extent um so so with mobile
4:34:39devices if you remember um as a
4:34:42developer or designer really makes a
4:34:43difference whether your user is using an
4:34:46Android kind of device or an iOS or iPad
4:34:50iPhone kind of device and uh most of the
4:34:54time you actually have to create
4:34:55specific apps to each of these kinds of
4:34:57platforms and with XR it's the same
4:34:59thing I mean we have a number of
4:35:02crossplatform solutions uh now but uh
4:35:05there are still these uh vendor specific
4:35:08device sdks that really give you the
4:35:10latest feature access to the latest
4:35:12features and it really takes some time
4:35:14before these features make it into some
4:35:16of these Solutions around Unity or
4:35:18unreal or even webxr which we're going
4:35:20to talk more about in the third course
4:35:24um there's obviously also the issue of
4:35:25device limitations um a lot of these
4:35:27devices are still fairly limited they're
4:35:30often overhyped and there's a lot of
4:35:33overpromise in a lot of the promotional
4:35:35videos out there which actually by and
4:35:37in itself is definitely another issue
4:35:39we're not always honest when it comes to
4:35:40these Technologies
4:35:42the third class of issues I would say um
4:35:45is those that that have to do with user
4:35:47adoption uh XR AR VR uh still fairly
4:35:51unknown it feels very unfamiliar I mean
4:35:53it's not the standard interface we use
4:35:55on a daily basis right it's not like a
4:35:57website or if I took away your mobile
4:35:59phone you would probably miss it if I
4:36:01took away your quest or your hland if
4:36:03you have one probably take you a little
4:36:05while before you realize it's gone you
4:36:07don't actually depend on it yet and
4:36:09there's obviously the larger issue which
4:36:11is really important uh issues to do with
4:36:14accessibility and what I mean by that is
4:36:16that um there are people out there that
4:36:19are um that that usually benefit from
4:36:22screen readers and other kind of
4:36:23software to make um applications and
4:36:27user interfaces more accessible to them
4:36:30and in the arvr space we're really doing
4:36:32a poor job so far basically excluding
4:36:35this whole important group of users and
4:36:38that is is really it is very annoying to
4:36:40me because it ALS directly impacts uh
4:36:43this whole teaching experience here it's
4:36:44very hard for me to make sure uh that we
4:36:47can all make it through the exercises
4:36:49that I have thought of in the Honest
4:36:51track the tools that I wanted to use are
4:36:53actually not why they are really
4:36:56widespread and actually enjoy High
4:36:58adoption um like unity and unreal
4:37:01they're really difficult to use and so
4:37:03accessibility is an issue and Equity
4:37:05obviously as well so uh most of these
4:37:07Technologies are still very very
4:37:09expensive there's a high cost so
4:37:11associated with them if it's not the
4:37:13actual device in itself and you buy it
4:37:15if you can afford the $4 to $500 that's
4:37:17fine there's still additional cost like
4:37:19you learning all these things setting up
4:37:21the equipment and so um Equity is really
4:37:24an issue like for example if I want to
4:37:26teach a class and let's say oh let's all
4:37:28look at this together in AR or VR like
4:37:31how do I do this most environments are
4:37:34actually not equipped um at this stage
4:37:36and then finally obviously social
4:37:38acceptance I mean let's say you an
4:37:40Enthusiast and that's great and let's
4:37:41say you get some cool device and you
4:37:43walk around outside and then and then
4:37:46what like what other people around you
4:37:48going to think they lots of actually
4:37:50ethical issues and social concerns If
4:37:51somebody walks into a cafe with a
4:37:54virtual reality headset what do you
4:37:56think if the pilot says hey good morning
4:37:58this is a flight to Seattle I'll welcome
4:38:00everybody on board by the way this is my
4:38:02first real flight so far I've only been
4:38:05trained in virtual
4:38:06reality I mean yeah I mean we do have
4:38:09flight simulators already but I mean
4:38:11would you trust that
4:38:12pilot I don't know so these are really
4:38:15some interesting questions that we have
4:38:17to well really concern ourselves with
4:38:19and think through finally and this is a
4:38:22big class of issues that's like
4:38:24everywhere privacy and security it's a
4:38:26really big issue um and and so I I'll
4:38:30shed some light on each of these but I
4:38:32think these are the four big classes I'm
4:38:34sure there are additional issues but
4:38:36this is what I came up with as I was
4:38:38thinking through some of the issues and
4:38:39preparing this lecture so I thought we
4:38:42could start by looking at AR and VR uh
4:38:45from a technology perspective and there
4:38:48are a number of issues that are shared
4:38:49and I think many of these issues were
4:38:51actually really making good progress so
4:38:53realism is something that I mean we're
4:38:55starting to relieve some of these
4:38:56experiences we were starting to feel
4:38:58present in some of these experiences
4:39:00many of them are clearly immersive we do
4:39:03have relatively good uh support for
4:39:05natural interaction so that means
4:39:07gesture and speech so body based
4:39:09interaction if you will as opposed to
4:39:11most in keyboard and indirect uh
4:39:13manipulation we have a lot of direct
4:39:14manipulation we do still have the
4:39:16controllers but even that feels kind of
4:39:18natural so we also making good progress
4:39:21on the spatial mapping side I mean this
4:39:22is more like a focus in the AI world
4:39:25like mapping out the environment around
4:39:27the user so that we can really augment
4:39:29the experience but also on the VR side
4:39:31we really have to do a better job Oculus
4:39:33is starting to show some uh obstacles
4:39:36that can identify in the room because in
4:39:37a VR environment these uh things that
4:39:40are in the way are actually not only
4:39:42disruptive to the user experience
4:39:44they're actually very dangerous so
4:39:45spatial mapping I think is something
4:39:47that we shouldn't just associate with
4:39:48one of these Technologies we use it a
4:39:50little bit different so in AR we use it
4:39:52for occlusion and some of these things
4:39:54and in VR we really well we want to
4:39:56avoid danger we want to we want to we
4:39:58want to make sure that all users are
4:39:59safe in in both AR and VR but in VR this
4:40:02can really be an issue given that your
4:40:04view is excluded by the headset um
4:40:07object recognition is something that
4:40:08we're also improving so um we we we we
4:40:11we have a little bit more semantic
4:40:12understanding now we know this behind me
4:40:13is a wall this is maybe a shelf I mean
4:40:16this is already getting hard um but if
4:40:18you look at some of the more detailed
4:40:19examples that I showed like Google lens
4:40:21for example I can bring up a number of
4:40:23objects in front of the camera and the
4:40:25camera will relatively reliably tell me
4:40:28what kind of object it is when the
4:40:30object is moving it is hard do we know
4:40:33whether an object is dangerous or not I
4:40:35mean we can figure out whether it's
4:40:36moving we can probably find out the
4:40:38trajectory and velocity and we can warn
4:40:41users but at this stage we actually
4:40:43don't really have that kind of stuff
4:40:45built into these Technologies just yet
4:40:48it would be very useful though on the VR
4:40:50side more specifically so we're actually
4:40:52making uh quite a good progress in terms
4:40:55of haptics uh so um there's a number of
4:40:58controllers out there like kind of
4:41:00gloves they really it's like exoskeleton
4:41:02they're really attach around parts of
4:41:04your body and may even support you
4:41:07provide haptic feedback so kind of think
4:41:09of it kind of like force feedback so
4:41:10that when you touch something in virtual
4:41:12reality it feels like real um Locomotion
4:41:15yes I mean we can walk around in VR um
4:41:18we have six degrees of freedom tracking
4:41:20now inside out so we don't have to put
4:41:22anything in the environment anymore so
4:41:23we're definitely making good progress
4:41:25we're having more and more Advanced
4:41:26Techniques in terms of redirected
4:41:28walking so that we can actually um
4:41:31create larger virtual experiences in
4:41:33smaller physical spaces just by tricking
4:41:35essentially the the the senses of the
4:41:38user but in a way so that it's not as
4:41:41evident to the user and it's a little
4:41:43bit surprising and so it's still kind of
4:41:45okay but even even that area of research
4:41:48in my opinion has a number of
4:41:50interesting issues um to to to really
4:41:53figure out U given that we do
4:41:56essentially deception of users um so now
4:41:59we're talking about AR specifically so
4:42:00perception refers to all these classes
4:42:03of issues that have to do with how we
4:42:04perceive things when we're looking into
4:42:06an AR interface occlusion is a really
4:42:09important one um so obviously if a
4:42:12virtual object is kind of behind me my
4:42:14body should occlude it just like my body
4:42:16occludes the the real chair um behind me
4:42:19or like parts of whatever is behind me
4:42:22right now and if these were virtual
4:42:24objects they should obviously react just
4:42:26like in the physical world and um the
4:42:29other big issue with perception is for
4:42:31example light estimation so um in order
4:42:33for augmented reality content to really
4:42:36appear like it's part of the real world
4:42:37we need to figure out what's the light
4:42:39condition over there as opposed to over
4:42:40there here and I have interesting key
4:42:42and fill lights here um so that would be
4:42:45quite different and and we need to
4:42:47figure that out to fit in the content
4:42:49and well essentially uh work around some
4:42:51of the perceptual issues finally scene
4:42:55understanding I think that's where we're
4:42:56making a lot of progress because it's
4:42:57this intersection That's so exciting to
4:42:59a lot of people right now with machine
4:43:00learning and AR and so like computer
4:43:04vision this idea that the computer can
4:43:06look at things and actually understand
4:43:08what they're looking at so that this
4:43:10behind me is a wall okay we can do this
4:43:12that I'm sitting on a chair that I'm
4:43:14actually a person um so people
4:43:16segmentation is something that falls
4:43:18into that category um we're making
4:43:21progress there you can see examples on
4:43:23AR kid uh most vendors will adopt
4:43:25similar techniques and build these
4:43:27techniques into their environments um we
4:43:29can segment the sky from the rest of the
4:43:31world we we know the seating we know all
4:43:34kinds of vertical and horizontal
4:43:35surfaces yes but most often on most uh
4:43:39on most for example mobile devices is
4:43:41all we we know is a classified to some
4:43:43extent a few labels that we have there
4:43:46and point cloud and so then we know that
4:43:47this is a chair or something that you
4:43:49can sit on let's say we don't know we
4:43:51don't actually know it's a chair we just
4:43:53know that you can probably sit on it we
4:43:55can tell whether something is a window
4:43:57whether something is a wall and so I
4:44:00already gave some examples when I talked
4:44:02about spatial mapping and object
4:44:04recognition and so uh scene
4:44:06understanding is really the semantic
4:44:07layer uh on top of it and that's really
4:44:09hard um
4:44:11you might think that wow we're making a
4:44:13lot of progress with like mapping the
4:44:15world and that's true outside but indoor
4:44:18mapping so indoor location mapping has
4:44:20always been a little bit of an issue uh
4:44:22well up to the point when street view
4:44:24started to to go into buildings and uh
4:44:26public buildings at least um but so with
4:44:29ar I mean a lot a lot of us are actually
4:44:31using these Technologies at home and um
4:44:35or like in let's say more private
4:44:37settings and so scene understanding
4:44:39really involves a lot of comput
4:44:41and the question is where this
4:44:43computation happens and so you know this
4:44:46okay don't get me started but this
4:44:48raises all kinds of issues I should go
4:44:50to the next uh to the next slide but I'm
4:44:52I'm so happy to talk about some of these
4:44:53issues in more detail as well so scene
4:44:57understanding relates to some of the
4:44:58things I also said when I talked about
4:45:00spatial mapping and object detection and
4:45:03recognition and uh scene understanding
4:45:05is really the semantic layer but these
4:45:08are just the technical issues so I said
4:45:11we should also look at some of the
4:45:13broader concerns that are out there and
4:45:16those are the ethical and social
4:45:18concerns um ethical concerns that you
4:45:21might have but then also um ethical
4:45:25concerns in the sense of like what if
4:45:27you use these Technologies uh in a in a
4:45:29more public setting U so what are the
4:45:32social concerns uh there we're going to
4:45:35talk about that a little bit and then
4:45:37accessibility and Equity a huge class of
4:45:39issues already already said that but I
4:45:41want to expand on that a little bit and
4:45:42then privacy and security so these are
4:45:45some of the three kind of like depends
4:45:48on I've grouped it into three but there
4:45:50are more like six kind of really big uh
4:45:53sets of issues and I want to have a
4:45:56little bit of a discussion around each
4:45:57of these especially in the honors track
4:45:59I've prepared a few exercises we're
4:46:01going to look at a a few cool video
4:46:04examples in each of these when I say
4:46:05cool I mean cool because they support
4:46:08the learning experience but some of them
4:46:09are actually quite not cool and quite
4:46:11scary and that's really uh to poke a
4:46:14little bit into the ethical and social
4:46:16concerns side of issues so I mean if
4:46:19you've seen Black Mirror uh or some of
4:46:21these um like upload or some of these
4:46:24movies uh and series are actually quite
4:46:26interesting how in film at least we are
4:46:29processing the impact uh and actually
4:46:32some of the evil or potentially bad
4:46:34directions um of these Technologies and
4:46:38um and it's important that we really
4:46:40think about this
4:46:56so what I'm going to do now for each of
4:46:58these groups of topics or larger issues
4:47:01uh I'm going to ask uh a few key
4:47:04questions that I think we should concern
4:47:06ourselves with as we are thinking
4:47:08through some of these issues so the the
4:47:11first important uh question I asked uh
4:47:14here is how to promote ethical and
4:47:17responsible XR Design This is really a
4:47:20question that is really important to me
4:47:22um it is inspired and informed by many
4:47:24discussions with colleagues and and and
4:47:27other designers in in in the space and I
4:47:30think it's really important um and we
4:47:32don't actually know that yet I mean I
4:47:35mean there are like when you create an
4:47:37AR or VR app it goes to an app store and
4:47:40then then it will be reviewed by
4:47:43somebody potentially and then it will be
4:47:45determined whether that app is kind of
4:47:47safe to be used and then it goes out to
4:47:50the users and and then we don't know
4:47:52what happens because we can't anticipate
4:47:54all the kinds of context in which uh
4:47:56these apps are being used and
4:47:59so how do we actually establish some
4:48:02kind of process and policy to actually
4:48:05govern some of this um it's really a big
4:48:08question so how do how to to increase
4:48:11control of users over XR
4:48:14apps so another big question that I'm
4:48:16thinking about a lot uh on the ethical
4:48:19uh side is how to increase control of
4:48:23users over XR apps I mean the whole idea
4:48:26of ethic ethical and responsible design
4:48:29is to be transparent and to give control
4:48:31to users and also make it safe obviously
4:48:35uh if you look at the current XR
4:48:37experiences the apps that you can
4:48:39download and try out I mean they're not
4:48:41customizable I mean they just come in
4:48:43some standard configuration basically
4:48:45more or less do what they the best the
4:48:47most they can do in terms of what the
4:48:50devices allow them to do so the the
4:48:53limit is really more like the technology
4:48:54not like whether we should be using all
4:48:57these sensors it's more like oh cool we
4:48:58have access to all these sensors um and
4:49:01so how do we give control to users so
4:49:03maybe I want to trade off some accuracy
4:49:05in tracking if it means that you don't
4:49:07know everything that I have in my
4:49:08household uh I mean you as the developer
4:49:11of the XR app and so I think um
4:49:15obviously there are also U obviously
4:49:18this also plays into accessibility
4:49:19issues that U I'm going to talk about
4:49:22later what are new Norms in Social XR
4:49:25apps that is an interesting question and
4:49:28there's a lot of Social and meting
4:49:30oriented AR and especially VR apps or
4:49:33all space is a popular example it used
4:49:37to be second live and had all kinds of
4:49:39interesting issues around it
4:49:41and so um there a lot of issues in that
4:49:44space uh harassment is a really big one
4:49:47H and how do you actually police this
4:49:49and is it is it okay to have like should
4:49:52we have real people observe what other
4:49:54people do and uh are they allowed to
4:49:57judge them can they police them and
4:50:00should they be visible or not all kinds
4:50:03of issues there and um what happens if
4:50:06you ban somebody from these platforms
4:50:09and should you ban and what's the
4:50:11threshold to Banning it's it's it's
4:50:13really difficult how do you trace
4:50:15identity uh and um how do you make sure
4:50:18that this person that uses that ID and
4:50:21but then also has another ID so all
4:50:23these issues how to mitigate concerns
4:50:26about XR Us in public that to me is a
4:50:28big one because uh right now I mean when
4:50:31you walk around with your smartphone
4:50:33that's fine I mean legally um it's
4:50:36probably not um and it's also like not
4:50:39the smartest thing to do
4:50:41depending on how you walk around with it
4:50:43it's not the safest thing to do at least
4:50:46but um nobody looks at you in a weird
4:50:49way if you take out your smartphone but
4:50:51if you were let's say at the bus stop
4:50:53and you took out your VR headset or you
4:50:56put some kind of glasses on your head
4:51:00more like AR
4:51:01technology what do you think is going to
4:51:03go through people's minds around you and
4:51:07how do we mitigate those concerns now um
4:51:10one way that's is often done is through
4:51:12extensive advertising so I suspect when
4:51:15Apple comes out with their new kind of
4:51:17whatever it is glasses um it'll be it'll
4:51:21look so cool and videos it'll be heavily
4:51:23promoted and at some stage it'll be fine
4:51:26people may want to have that and maybe
4:51:28it'll mitigate some of the concerns but
4:51:31overall this is something that Society
4:51:35um will figure out and uh what we need
4:51:38to do is actually make sure that we get
4:51:40a lot of the other issues right so
4:51:42accessibility Equity privacy and
4:51:44security make those uh safe and
4:51:47customizable and maybe then trust into
4:51:51these technologies will increase also a
4:51:53big issue is um and I'll bring it up in
4:51:55a second is actually what happens with
4:51:57the data that's being collected and who
4:52:00processes it who has access to it and I
4:52:02think that who question is a really big
4:52:04thing if you're a fan of that who
4:52:06company then it'll be fine for you but
4:52:09it may not be fine for other us and so
4:52:11that's something to really think
4:52:13about so looking at the two issues on
4:52:15the right as one example in this space
4:52:18it's really hard to give concrete
4:52:19examples it's so broad but um let's
4:52:22revisit quickly when I went to i e VR
4:52:25conference in virtual reality so this
4:52:27was an interesting um experience because
4:52:31a lot of people knew okay this is one of
4:52:32the first now remote conferences due to
4:52:35covid-19 this is obviously a conference
4:52:37where people are very uh enthusiastic
4:52:40and uh are interested and supportive to
4:52:42some extent at least of these
4:52:44Technologies and um so on the social XR
4:52:49side it was interesting because I did
4:52:51have I met a lot of people in VR I had
4:52:54interesting discussions talking to the
4:52:56avatars um and uh it it felt a little
4:52:59bit like meeting people at a conference
4:53:01at a real conference so that was fine um
4:53:04it there were no weird moments but that
4:53:06was because it was overall a
4:53:08professional event the organizers also
4:53:09made sure that a lot of information was
4:53:12provided beforehand to really make sure
4:53:15this is a professional event and
4:53:18obviously people have a certain
4:53:20reputation and a responsibility when
4:53:22they go to these events um um and so
4:53:26kind of was fine um I am so I'm one of
4:53:30the users of these VR devices I didn't
4:53:32use it all the time so I saw a lot of
4:53:35people overall though that didn't use VR
4:53:38devices I'm not sure whether it has to
4:53:39do with them having specific concerns
4:53:42about these Technologies or just like
4:53:43because it's very tedious and cumbersome
4:53:45right now the way it was set up um was
4:53:48you know lots of little virtual reality
4:53:50rooms limited to like 20 people or
4:53:52something it was really hard to find
4:53:54somebody uh just because of bandwidth
4:53:57and other kinds of issues we had to it's
4:53:59it's hard to scale these kinds of events
4:54:01it's the same with finding a big
4:54:03conference venue um in the real world um
4:54:06but it's much more solvable in in
4:54:10actually in in the virtual world because
4:54:12we can add more compute if you will and
4:54:15then we can support larger uh meetings
4:54:18and and venues um wouldn't be such a big
4:54:21deal I we you still need to figure out
4:54:23some of the technical issues there no
4:54:25doubt but overall it was an interesting
4:54:28uh experience and really played into
4:54:30some of these ethical and social
4:54:32concerns that I was just talking about
4:54:34so one thing that was also noticeable
4:54:36with the VR conference was that actually
4:54:38attendance was virtual attendance was a
4:54:40lot higher normally this is a venue that
4:54:42is limited to I don't know 4 to 800
4:54:45people and uh in this time they had many
4:54:47more H I've never been to VR be to this
4:54:50conference before and so in many ways it
4:54:53actually had some interesting impact on
4:54:56on on the shape and the scale of the
4:54:58conference and and and so maybe because
4:55:03the barrier uh to going there um was
4:55:07relatively low um it was for free as as
4:55:10well and um and so it kind of attracted
4:55:13more people than usual that doesn't mean
4:55:15that these XR Technologies are now
4:55:18enjoying more acceptance uh
4:55:20automatically shouldn't be interpreted
4:55:21this way it's just interesting moving
4:55:23these venues into this kind of like
4:55:25online uh format and and and supporting
4:55:28it with relatively state-of-the-art
4:55:30virtual reality uh Technologies actually
4:55:32did have quite an interesting impact on
4:55:34how this whole conference played out
4:55:52so one big question that goes through my
4:55:54mind is how to make XR accessible for
4:55:58impaired users uh this is really a big
4:56:00one and maybe the one that that people
4:56:02think about initially when I say XR and
4:56:05accessibility oh yeah how do we support
4:56:06people that are not using their visual
4:56:09sense primarily like virtual reality is
4:56:12essentially all about uh the visual and
4:56:15ex and the and the and the AR
4:56:18experiences that we're designing these
4:56:20days are also predominantly Visual and
4:56:23we have relatively poor support for uh
4:56:26users with visual impairments we in fact
4:56:29don't actually really design for them at
4:56:30this stage and how do we make it
4:56:33accessible so that's a big question uh
4:56:35one of the projects I like in this space
4:56:37is seeing VR it's a research project by
4:56:40Microsoft research but actually some
4:56:41colleagues and I would say friends uh in
4:56:44the space are it's human computer
4:56:46interaction research and it's really
4:56:48quite impactful it's a tool kit uh
4:56:50targeted at developers to check their
4:56:52applications um whether like basically
4:56:55you can try out different kinds of
4:56:57seeing modalities so different kinds of
4:56:59impairments can be simulated to some
4:57:01extent and you can check against those
4:57:04and make sure that you're kind of like
4:57:06adapting your interfaces in the right
4:57:08way um I think that is a a great example
4:57:11of uh a tool and an effort it's really a
4:57:16research effort where we need where we
4:57:17need to see more of the other big
4:57:19question in this space so I'd say
4:57:21accessibility is really two things like
4:57:23how do we make XR accessible and then
4:57:27how do we use XR to increase users
4:57:29abilities that's a little bit different
4:57:31right now I'm thinking about giving
4:57:33Vision to people who may not have good
4:57:36Vision right now and we can use given
4:57:39that some of these Technologies are
4:57:40quite Advanced especially in the air
4:57:42space so we talked about we have good
4:57:44spatial mapping we have object detection
4:57:46we have um we have an understanding of
4:57:49geography and uh we also now are making
4:57:52good progress on the semantic layer
4:57:54through scene understanding so we can
4:57:56actually tell people what they're
4:57:58looking at I mean in real time you could
4:58:01provide a screen reader for augmented
4:58:03reality and that might actually I mean
4:58:05designing this will is still quite a big
4:58:08question how would you do this and
4:58:09something that I'm actually thinking
4:58:10about in some of our own research right
4:58:12now but um it could be really powerful
4:58:16um the one example that I wanted to
4:58:17share here on the accessibility side and
4:58:20then also Equity um because it really
4:58:23levels the playing field and allows more
4:58:25people to come together is the iy
4:58:27project I did with uh my colleagues here
4:58:29at the University of Michigan and that
4:58:31project was led by Roland graph who's a
4:58:33good friend and actually a really good
4:58:35researcher as well and a designer and uh
4:58:38so this was a great collaboration so I'm
4:58:40going to share a few uh insights from
4:58:43imim here so imim was really this uh
4:58:46application that Roland and huk uh a a
4:58:50professor in computer science have been
4:58:51working on so it was designing an
4:58:54augmented reality floor projection
4:58:56system um I gym with the idea inclusive
4:59:00gym to Adaptive Sports so allowing
4:59:03children in wheelchairs to play against
4:59:05other children and wheelchairs both
4:59:07manual or powered wheelchairs we
4:59:09designed for both and then also play
4:59:11against able-bodied children and so we
4:59:14had the most interesting user studies
4:59:15and that was really my role like how to
4:59:17help design evaluate um assess the
4:59:20experience how to make it fair um the
4:59:22whole study design and then um we
4:59:24actually had other people helping with
4:59:25recruitment so we brought in a few
4:59:27families and so basically um they came
4:59:31with their children and they were in
4:59:32wheelchairs and they brought a friend or
4:59:35brother or sister along so some siblings
4:59:38and and played around and we we did it
4:59:40in a traditional kind of like user study
4:59:42style where we really thought this
4:59:43through how to who how to play against
4:59:47each other there's some kind of training
4:59:48involved there was actually a whole
4:59:50algorithm that would help adapt the
4:59:52system a little bit so um balancing was
4:59:55really an important issue and we
4:59:57actually didn't get it right completely
4:59:58it was a little overbalanced so the
5:00:00wheel users often beat uh the
5:00:03able-bodied uh children and that was
5:00:05actually uh in some in some situations
5:00:07that was actually not cool cuz this is
5:00:10the first time this happened and
5:00:12normally I let my my brother win kind of
5:00:15thing but hey now he actually beat me
5:00:17and that is a a feeling that I've never
5:00:19had before an experience I've never had
5:00:21before anyway that was really really
5:00:23interesting um and and and AR as so
5:00:26augmented reality the AR component was
5:00:28relatively simple from a design and
5:00:30development perspective but it had a lot
5:00:31of impact and that is also an important
5:00:33lesson actually to be learned it doesn't
5:00:34always have to be complicated or super
5:00:37Advanced a little can do a lot sometimes
5:00:40and so I do think iy is is a cool
5:00:43example uh where we used augmented
5:00:45reality and some additional means um you
5:00:49had like a kick button so that you would
5:00:51basically expand this circle around you
5:00:54this peripersonal space around you to be
5:00:56able to kick the ball virtually um even
5:00:59if you can't use your feet that way um
5:01:02or your arms and uh so yeah I thought um
5:01:07that was a really cool example of how to
5:01:09use XR to increase users abilities so
5:01:11this then also directly leads to some of
5:01:13the issues to do more I would say more
5:01:15with Equity so how do we actually
5:01:17broaden access to these XR Technologies
5:01:19how do we make sure that more and more
5:01:21people have them how can I actually go
5:01:23into a classroom and at which stage is
5:01:26it okay to expect that all my students
5:01:28have an XR device and can try out some
5:01:30of the content to which extent do I
5:01:32still have to provide that I mean when
5:01:34we now go to a classroom in in the
5:01:37Western World at least um we say things
5:01:40like yeah just Google this or like take
5:01:41out your laptop and I want you to use
5:01:44your laptop and I want you to use your
5:01:45mobile phone but I can't say at this
5:01:48stage hey just can we just VR this or ar
5:01:50this I mean this is not even a word yet
5:01:53and um yeah because it's it's expensive
5:01:57there's a lot of concerns about these
5:01:59Technologies um that hinder adoption and
5:02:03um the Technologies by and in themselves
5:02:05are actually still not super convincing
5:02:07yet there are more like lots of toy
5:02:09examples out there I mean IGM was really
5:02:12actually a really cool example and I
5:02:14think we need to see more of these
5:02:16examples so that people see more of the
5:02:19potential and um and that might be a way
5:02:22if if if if the finances are not the
5:02:25primary issue it's more like some of the
5:02:27concerns you have then maybe that's a
5:02:29way to broaden access we need to learn
5:02:31from both positive and negative examples
5:02:33but the positive examples are really the
5:02:35ones that have more impact I would say
5:02:38at least less damaging impact uh how to
5:02:41increase diversity of XR users is
5:02:43another big issue um so finally in in uh
5:02:48finally another question that I wanted
5:02:49to finally another question that I'm
5:02:51thinking a lot about as well is how to
5:02:53increase diversity of our XR users um so
5:02:57I just gave examples from the Western
5:02:58World and that's really where we see
5:03:00most of the Technologies at this stage
5:03:03um U I'm not sure really uh uh I suspect
5:03:06gender balance wise it's probably more
5:03:09male when it comes to arvr Technologies
5:03:11I don't actually have any concrete uh
5:03:14numbers to go on uh it's probably more
5:03:16like the younger generation and I might
5:03:19just be stereotyping here and that is
5:03:21actually dangerous but I feel like given
5:03:23on based on what I've seen so far and
5:03:26based on on the information that I have
5:03:27this might actually be true and so
5:03:29therefore how do we actually increase
5:03:31the diversity of our XI users uh do we
5:03:34do this through design does it need um
5:03:37kind of
5:03:38like does it look does do these devices
5:03:40need to look more fancy do they need to
5:03:43look more like
5:03:44fashion um that's one what some people
5:03:47think um the form factor is definitely
5:03:49an issue but I don't think that's it
5:03:51there are other issues at play here um
5:03:55for example motion sickness uh one thing
5:03:57that is a fact is that female users
5:03:59often uh react more heavily or more
5:04:02likely to experience motion sickness so
5:04:04motion sickness really means that you
5:04:06are like in this headset and you're just
5:04:08starting to feel Buzzy uh you you're
5:04:11experiencing nausea and you're just
5:04:13really uncomfortable and that while it
5:04:16happens with male uses as well it's more
5:04:18likely and actually uh it happens more
5:04:21often um even in my own experience in in
5:04:24in the classroom and teaching with uh
5:04:26female users and and so we can do
5:04:29something about it maybe from a design
5:04:30perspective so we have certain
5:04:31techniques uh for example as you
5:04:33navigate through this space we can this
5:04:36is actually when motion signal happens
5:04:37like when you when the propi reception
5:04:39doesn't actually your so your physical
5:04:41senses of the bodies of your body don't
5:04:43actually match what's happening in the
5:04:44virtual reality and then so if you don't
5:04:47get this match right um um or you're
5:04:50seeing too much at the same time like uh
5:04:52in the periphery if you focus more on
5:04:54the FIA area uh and actually blend out
5:04:57some of the peripheral so if you limit
5:04:59some of the vision for the time as
5:05:00you're moving is for example one
5:05:02technique to make this work a little
5:05:04better and then we need more research in
5:05:06this space as well
5:05:23so the last big topic area uh is is
5:05:26around privacy and security so when I
5:05:28say privacy and security I'm thinking
5:05:30about some of these questions so how to
5:05:31communicate what data is actually being
5:05:34collected that's that's really a big
5:05:36question um I mean I I can I can show
5:05:40you kind of like a consent form like
5:05:42terms of service and you scroll forever
5:05:44before you're like opting into that
5:05:46experience but um do I really inform you
5:05:50uh is that the best way to communicate
5:05:52um how often should I should I ask for
5:05:54your permission lots of issues there is
5:05:56there kind of like a standard interface
5:05:58or should this every application should
5:05:59figure that out for themselves um one
5:06:02thing that I can see with Mozilla and
5:06:04some of the efforts they're doing there
5:06:05in the XR space when it comes to making
5:06:08Firefox or or version of web browsers
5:06:10more XR capable is really thinking
5:06:13through how do we communicate to users
5:06:15how do we ask for permission for access
5:06:17to the camera um how often should this
5:06:20access be granted and what's a good way
5:06:23to ask it h but it's still relatively
5:06:26technical CU most people don't really
5:06:27know what that means what do you mean
5:06:29you need access to my camera okay so
5:06:32what what do you actually then see just
5:06:33the photos oh oh you can actually do
5:06:36live 3D reconstruction of my space
5:06:38identify brand I have in here and
5:06:41estimate the worth of all the objects
5:06:43around me yeah you could do that um
5:06:47so um is that the data that you're
5:06:49collecting so that's a that's a big
5:06:51question uh how to prevent the XR app
5:06:54from seeing sensitive data right so when
5:06:56when you walk around with the Hollin on
5:06:58your head it sees everything everything
5:06:59you see and actually more because it
5:07:01actually has cameras going it has wider
5:07:04Vision than you even though the the FEI
5:07:06of view what it actually can augment is
5:07:08really small don't get trick by this
5:07:10because the cameras actually see a lot
5:07:11more and they need to you know scan the
5:07:13environment around you that's not the
5:07:15same as the fal View and if you didn't
5:07:18know that then that uh relates back to
5:07:20the first uh issue which is how to
5:07:22communicate what data is being collected
5:07:24anyway how to prevent the XR app from
5:07:27seeing sensitive data yeah could you
5:07:28just like ude it could we have some
5:07:30brushing techniques so I was thinking a
5:07:31little about this from a design uh
5:07:34perspective um could you blur out things
5:07:37I mean people always say oh we just do
5:07:39up for ation at the source so um to do
5:07:42both with the the data that's being
5:07:44collected and then also actually
5:07:46transfer to somebody else for being
5:07:48processed somewhere um and that is this
5:07:51off fisc of the source means okay before
5:07:54anybody any application gets to see the
5:07:56data we're going to blur out for example
5:07:57your face we're going to remove we're
5:07:59going to De identify you we're going to
5:08:01anonymize it and um okay what if my face
5:08:05is not the most sensitive data so what
5:08:07if like all the documents that are
5:08:09around here mostly my notes about the
5:08:10moo um but um let's say there were some
5:08:13more important documents um so let's say
5:08:17who I'm going to vote for or some of
5:08:18this data and and while that may be in
5:08:21my office um maybe I don't want the XR
5:08:23device to see this and so how do we make
5:08:25sure that it doesn't see this um the
5:08:27best way is to just like not use it in
5:08:29that space yeah so there are actually
5:08:31lots of issues um just there but and I
5:08:34could go on forever so so how to know
5:08:37whether the XR app is actually secure
5:08:39and here's a hypothetical scenario uh so
5:08:42we can do redirected walking now and
5:08:43that means somebody could remote log in
5:08:45maybe into your device and then uh start
5:08:48tricking your senses in the future the
5:08:50FBI or CIA may actually do this and um
5:08:55um some kind of Secret Service somewhere
5:08:56else in the world but actually use this
5:08:58to kill somebody right we could just
5:09:00navigate them into obstacles and scary
5:09:03thought again but is it is it it is
5:09:06something that could happen and um that
5:09:08is the worst case scenario in many ways
5:09:10but we really need to do a good job of
5:09:14building in safeguards someh kind of
5:09:18control and um maybe some kind of Ethics
5:09:21sport in general uh something on you
5:09:24know when I do a new research study I
5:09:26mean it has to go through a review
5:09:27process and that is uh really experts
5:09:30and that is a and that is conflict of
5:09:32interest three which is also very
5:09:34important uh cuz they're just making
5:09:35sure that I'm doing a good job and that
5:09:37I'm you know that the users I work with
5:09:40uh are not at at high risk and um that
5:09:45it's safe to participate in my research
5:09:47and we should really think about XR the
5:09:49same way in fact in course 2 when I talk
5:09:52about uh the design ethics there um and
5:09:56I bring up some of these issues again um
5:09:59that is one of the approaches that I
5:10:00suggest that we think it through from a
5:10:02kind of like an Ethics review process
5:10:05perspective finally how to develop safe
5:10:08and secure XR apps this is also really
5:10:10like we need more knowledge there so
5:10:12that we can guide designers and
5:10:13developers better uh we need to have
5:10:15some uh process and workflow in place I
5:10:19mean the beginning uh I mean the initial
5:10:21considerations of some of the app stores
5:10:23and app review we have out there is fine
5:10:26and then that obviously it's user rated
5:10:29so the community can also you know alert
5:10:32um themselves and somehow govern this to
5:10:35some extent uh um together
5:10:40but um we really probably but we
5:10:43probably really need uh some more
5:10:45official some kind of more standardized
5:10:47way because there
5:10:49are for because there are things like
5:10:52Side Quests for example I can Sid step
5:10:54the entire Oculus store I can download
5:10:57any app I want uh that may not be secure
5:11:00so I can just download the package
5:11:02directly on my quest if I want just
5:11:03using a tool called side quest for
5:11:06example so even if we have lots of
5:11:08things in place around the App Store and
5:11:10maybe controlled by Aus to some extent
5:11:12there will always be it's kind of like
5:11:14in Jurassic Park right nature will
5:11:16always find its way and so this is also
5:11:20how it is in this
5:11:23space so on the issues of how to
5:11:26communicate whe what kind of data is
5:11:28being collected and how to prevent the
5:11:29XR app from seeing sensitive data I have
5:11:32an example here of using Ikea place in
5:11:35uh in the studio here but it should
5:11:36really be in my living room but I had
5:11:38some con concerns about that and so I
5:11:41recorded it here and it was just
5:11:42interesting to go through that process
5:11:44initially agreeing to all these things
5:11:46and actually it was very different when
5:11:48I tried it on my iPad there was actually
5:11:50this whole process and I have to kind of
5:11:53like agree to all the terms and give
5:11:54access of the camera and then on my um
5:11:57AR core uh enabled phone so on my
5:12:00Android phone I mean nobody warned me of
5:12:02anything or asked for any permission it
5:12:04just it just worked and um and and I
5:12:07find that also interesting that we have
5:12:09these differences between platforms and
5:12:11um it you know one was obviously
5:12:13Overkill like nobody reads all those
5:12:15terms and then on and the other
5:12:17experience was like H that was also like
5:12:21no concerns it was also a bit
5:12:23interesting anyway so that was my
5:12:25example for this part so so far I've
5:12:28actually listed quite a few issues and I
5:12:30was just like rambling and going on
5:12:32forever and it's really something that
5:12:33is really important to me and I think a
5:12:35lot about these issues and I'm sure if
5:12:37nothing else this came through in this
5:12:39lecture quite a bit and so get a sense
5:12:42that this is um it's interesting it's
5:12:45dangerous territory I mean designing in
5:12:48this space using things in the space we
5:12:51do have a lot of responsibility to our
5:12:53users as these as the creators we often
5:12:56lack the knowledge and experience
5:12:57there's a lot of hobbyists and and
5:12:59people out there that are trying to to
5:13:01do to innovate in this space uh we don't
5:13:04have a lot of norms we don't have a lot
5:13:06of established processes and workflows
5:13:08uh um to guarantee some kind of
5:13:11quality in fact to me it's really
5:13:13annoying with lots of the arvr
5:13:15experiences I see out there they're
5:13:16actually really naive and I find that
5:13:19really a little sad it's actually
5:13:20damaging to the whole potential of these
5:13:23uh exper you know the whole potential of
5:13:26these Technologies but I mean it's also
5:13:29that the it's also true that we are
5:13:30exploring this together and we need to
5:13:32learn from both um positive and negative
5:13:35examples and we can often learn more
5:13:37from negative examples we just have to
5:13:38agree on the fact that not every example
5:13:41we see in the space right now is
5:13:42actually a good example and um and we
5:13:46don't actually have all the indicators
5:13:47that we usually have because we don't
5:13:48have equal chances of everybody trying
5:13:50this out and so it'll take some time
5:13:53before we actually really figure out
5:13:55which of these applications make sense
5:13:57and which of these applications are
5:13:59really exploring the wrong direction
5:14:16and so as a Next Step I'm going to think
5:14:18through some of these issues and
5:14:20actually provide a little bit of uh my
5:14:23view on how to maybe address some of
5:14:26them I will not be able to cover each of
5:14:28them I was going on forever but here's a
5:14:30few ways maybe forward so one thing
5:14:34you've noticed is I've talked a lot
5:14:35about mobile and web and all the issues
5:14:38we have there and how we essentially
5:14:39replicate them in this space so apply
5:14:42the lessons that we've actually learned
5:14:44from mobile and web rather than repeat
5:14:46them so I mean to me it's surprising
5:14:49that web XR is only now happening after
5:14:52we already had some of these kind of
5:14:54like the vendor specific doing their own
5:14:57thing kind of
5:14:58sdks s i mean surprising and not
5:15:01surprising obviously it's not surprising
5:15:03because this is how the business World
5:15:05works and and and obviously you push for
5:15:08it and market share I understand but
5:15:10it's surprising because we know from the
5:15:13whole mobile and web world that this
5:15:15didn't play out very nicely and it just
5:15:17caused so much frustration for both the
5:15:20design community so designers and
5:15:22developers and then also the end users
5:15:24uh because you just don't know was it an
5:15:26app was it a web page is it an icon that
5:15:31I need to look for is it a bookmark and
5:15:33so it's like uh I don't know there's
5:15:36actually this uh this whole idea of
5:15:39having a shared virtual
5:15:41ecosystem um in the future is going to
5:15:44be very interesting in fact this leads
5:15:46me to the second consideration which is
5:15:48right now these arvr applications and
5:15:50experiences mean they live in their own
5:15:52little ecosystems we have their own app
5:15:54stores they're not compatible like you
5:15:56buy this thing for the Oculus Quest it
5:15:57doesn't run on your Rift s and I mean
5:16:00Oculus is now addressing some of these
5:16:02issues I don't want to buy the same app
5:16:03multiple times I mean it doesn't make
5:16:05sense to me and um so uh the other issue
5:16:09though is that um these VR and air
5:16:12devices don't really work well with
5:16:13anything else we have like they don't
5:16:15integrate with my TV or my smartphone I
5:16:17mean Oculus for example has an app that
5:16:20allows you to register devices but it
5:16:22doesn't do a lot more uh and it makes
5:16:26some of the process a little bit easier
5:16:27but I mean essentially we have been
5:16:29creating more and more new information
5:16:32cus right now we have some apps that are
5:16:35just on the Ulus or the Vive platform um
5:16:39and and and that's not that's not so
5:16:42ideal because not everybody will be able
5:16:44to have all these kinds of devices and
5:16:46we really need to think about kind of
5:16:47like this whole iot internet of things
5:16:49movement U really needs to also make it
5:16:52into the excess space and I'm not saying
5:16:54I'm not talking about those boring iot
5:16:56applications that I see every at every
5:16:58conference where I go to right now where
5:16:59they give me an iPad and allow me to
5:17:01look under the hood of the car as if I I
5:17:04I were able to repair anything with my
5:17:06car or see what's wrong with it um no I
5:17:09mean maybe they target manufacturing
5:17:10scenarios to be honest obviously but uh
5:17:13no I'm thinking really uh iot in the
5:17:15sense of having these integrated
5:17:17ecosystems like a cloud around us Edge
5:17:21Computing um and there actually like I
5:17:23say this as if there were If This Were
5:17:25cool it is it could be cool but it's
5:17:27really there's lots of issues uh to do
5:17:30with all of it and the AR cloud is a
5:17:33really big one for example so once we
5:17:35have mapped out the entire space um who
5:17:37owns the data who processes where is it
5:17:39stored how do I how do I have control
5:17:41over the data so that's very very
5:17:44interesting from a design perspective um
5:17:49a lot of tools out there are trying to
5:17:51create their own workflows like use our
5:17:53tool and then we have this next tool and
5:17:55then you publish this and then you have
5:17:57that tool and and then you have your XR
5:17:59solution so but then how do you actually
5:18:03get it from there to work on that other
5:18:05platform oh yeah I mean that's not
5:18:06really supported by our workflow and so
5:18:09um yeah I see this a lot and we really
5:18:13need to when we design when you design a
5:18:15new tool for example um never try to
5:18:18replace for example units you are unreal
5:18:21that's like too big a Target so you have
5:18:23to think about where does your tool um
5:18:26fit into this larger workspace workflow
5:18:29and um and then make sure that you're
5:18:32really connecting nicely and integrating
5:18:34um and and don't just you know hope that
5:18:37everybody will just you know change
5:18:39their workflow um it's like with all
5:18:42these to-do lists and task management
5:18:44tools or things that help you manage
5:18:46your email and there a lot of these Buy
5:18:49in and then they're becoming Allin
5:18:52Solutions so it's kind of like almost no
5:18:54way to get out of it anymore and you're
5:18:56stuck in them um and and we should
5:18:59really avoid this so this one here also
5:19:01goes to designers uh and developers out
5:19:04there so a lot of XR applications right
5:19:07now are really just toy examples they
5:19:09are they specifically created to
5:19:10demonstrate some new kind of uh feature
5:19:13technology like people segmentation so
5:19:16we have that as an example I show you
5:19:18the vase uh as an example there other
5:19:21examples like with virtual objects
5:19:23behind people I mean that's cool or like
5:19:26you you look at life 3D reconstruction
5:19:27and then people and you apply a little
5:19:29bit of physics and then people like Drop
5:19:31Like virtual balls into the physical
5:19:33world I'll show you some of these
5:19:35examples as well um yeah so first of all
5:19:38for an end User it's hard to imagine
5:19:40what the real value is uh with each of
5:19:42these applications I mean they're
5:19:43designed to inspire designers and
5:19:45developers to to show their potential um
5:19:48so uh I understand that we have these
5:19:49toy examples but we can really be a
5:19:51little bit more thoughtful when we
5:19:53create uh new kinds of XR experiences a
5:19:56lot more work and but that's something
5:19:58that's definitely I see a lot more uh
5:20:01effort in in my research lab in that
5:20:03space I'm really grateful for my
5:20:04students to really think Beyond some of
5:20:06the you know it's so cool initially to
5:20:08show you a virtual object and and I get
5:20:11it um but you really need to think uh
5:20:14more about it how do you communicate the
5:20:16value of these Technologies and it's in
5:20:18it's in specific domains out there but
5:20:20we have to understand those domains to
5:20:22really understand how we fit in how how
5:20:24does the XR fit in and um you won't get
5:20:28away easily with some kind of toy
5:20:29example in in a in a in a critical
5:20:31domain one is um I mean medicine is
5:20:34definitely a very critical one or crisis
5:20:36simulation or nursing things that we've
5:20:38we've worked on in the past but even
5:20:40just learning an instructional design is
5:20:42very critical where XR will play an
5:20:45important role and even there we see
5:20:47lots of toy examples right now I mean
5:20:49okay again I get it we're exploring the
5:20:50space but uh let's go beyond this
5:20:53somehow finally we really need to lower
5:20:56the barriers to entry and that's
5:20:57something that um I've I've worked on in
5:21:00the past as as part of a lot of my
5:21:02initial research in this space just
5:21:04because of how hard it was for me to get
5:21:06started in this space I thought maybe
5:21:08other people also experiencing this um
5:21:10and and and that's true um lowering the
5:21:13barri to entry also brings me back to
5:21:15some of the accessibility issues I
5:21:16talked about so we really need to be
5:21:18more inclusive we need to make sure that
5:21:21a lot more people can use and benefit
5:21:24from those Technologies and actually see
5:21:26the real benefits um and enable some
5:21:29kind of critical mass and then uh the
5:21:32responsibility that is with designers
5:21:33and developers out there is also a lot a
5:21:36lot more significant so those were
5:21:39really a lot of the issues and remember
5:21:41how we started out um this is almost how
5:21:44we started out and I'm just going to go
5:21:46back so maybe that it will be a
5:21:48memorable lecture talked about a lot
5:21:52about the issues obviously you could
5:21:54probably see how important they are to
5:21:56me how engaged I am with some of these
5:21:58topics and issues I just uh wished that
5:22:01uh more of us uh were able to dedicate
5:22:05uh more energy to this uh space it's
5:22:08really really important it's a really
5:22:10difficult uh set of issues to work on it
5:22:12requires a lot of training in in in a
5:22:15number of areas so in the human computer
5:22:18interaction space which is the research
5:22:20space that I'm active in we do have a
5:22:22lot of people that that that play around
5:22:24with these Technologies and then new
5:22:26cool new interaction Technologies and I
5:22:28think and that's cool and and I'm also
5:22:29one of those uh people as well but um
5:22:33over the last few years I've organized a
5:22:34few uh workshops and um collaborated on
5:22:39on a few research proposals that really
5:22:42try to push into these issues U and work
5:22:45on those that are highlighted here and I
5:22:48I think it it would be great to see uh
5:22:50more appreciation for some of this work
5:22:53um and that people like myself can
5:22:55actually transition and work on these
5:22:57issues because I do have the training
5:22:59and uh and that we can also we have the
5:23:01experience of actually putting these
5:23:02teams together to actually work on those
5:23:04issues um the issues are really too big
5:23:06to to tackle by individual people they
5:23:08require large um efforts and so um yeah
5:23:14I I think there will be a lot of
5:23:16pressure on uh companies in the in in in
5:23:20the future if not already to really get
5:23:22a lot of these things right and we need
5:23:24to continue to push that way we also as
5:23:27designers and developers we really have
5:23:28to figure out more of our responsibility
5:23:31and find ways to you know actually do
5:23:34ethical and responsible design that
5:23:36triggered the lecture that I added to
5:23:38the second course and um accessibility
5:23:41considerations are throughout it's hard
5:23:44it's difficult it it is actually it is
5:23:46really frustrating to me um and I'm sure
5:23:49to a lot of our Learners out there as
5:23:51well um I was trying to do the best I
5:23:54can now the best um is not good enough I
5:23:57can see that and maybe in the next few
5:23:59years this is something that can I can
5:24:01improve as part of uh some of the
5:24:03contents we offer here um so okay now I
5:24:07don't want you to think think um I don't
5:24:10want to get used to this uh too much so
5:24:12I'm actually going to um go back to uh
5:24:16normal Michel look but you know this one
5:24:19last time just one last time okay and uh
5:24:23that's it and here the lecture ends
5:24:42welcome to this last exercise in the
5:24:44first Honor track of our course here
5:24:47that's the first course in the exam
5:24:49specialization so don't be sad I have
5:24:51more exercises as part of the other
5:24:53honor tracks uh in the other courses so
5:24:56I think if you go through all of them
5:24:58you'll become a little expert at least
5:25:00uh that's uh what I hope in terms of the
5:25:03XR space if you're obviously not already
5:25:06an expert so um in this one uh we're
5:25:09going to approach things a little bit
5:25:10differently if you think about the tasks
5:25:12and activities we've done so far so the
5:25:14exercises in the previous modules were
5:25:16really focused on building a better
5:25:18understanding so the knowing XR part but
5:25:22now we want to start thinking more about
5:25:24the issues uh we are entering the last
5:25:27portions of the XR moo uh course one
5:25:33again there's more to come and um now we
5:25:36want to think about shaping the future
5:25:38XR and in order to shape the future we
5:25:41really need to understand uh the trends
5:25:44and the issues and obviously I've had
5:25:46lectures around those topics so now
5:25:48we're going to do some activities and
5:25:50really as part of this exercise reflect
5:25:53on some of the key issues so here is the
5:25:56task that I have for you so perform what
5:26:00I call an XR impact analysis focusing on
5:26:04key issues so you should consider all
5:26:07major classes of issues and I'm going to
5:26:08repeat them in the next slides uh also
5:26:11obviously outlined in the lectures in my
5:26:13lectures focus on key issues stay the
5:26:16issues that you're most interested in
5:26:18and that may be for different reasons
5:26:20but I just want you to be clear which
5:26:23issues are you going to focus on as you
5:26:25are performing the activity I think you
5:26:27should uh pick those issues first and
5:26:30then focus on those issues in the
5:26:32activity uh perform one of the suggested
5:26:34activities using think aloud um we have
5:26:37three activities that I have prepared
5:26:39for you it would be great if you were
5:26:40all doing the same activities you don't
5:26:42have to do it exactly the same apps or
5:26:46this exactly what I show but um these
5:26:49activities are designed a little bit
5:26:51differently and so it would be great if
5:26:53you were sticking roughly to these
5:26:55activities just to ground our thinking a
5:26:58little bit uh use think aoud so as
5:26:59you're performing the activity you just
5:27:02speak out you speak your mind and you
5:27:04say like oh this is weird or this is
5:27:06annoying or wow this this is the
5:27:08information they potentially collect and
5:27:10so these kinds of um wow or surprises
5:27:14are really good indicators that
5:27:16something might uh not be right about
5:27:18the user experience or because I ask you
5:27:21to focus on the issues and actually
5:27:22State them up front you really it guides
5:27:26your thinking it's something you will
5:27:27notice about yourself obviously then I
5:27:29would like you to write your impact
5:27:30analysis so this would be a few
5:27:32paragraphs in text just really capturing
5:27:35your main thoughts and concerns and this
5:27:37is something want to share with other
5:27:40readers uh sorry with other Learners who
5:27:42will then read uh what you wrote and
5:27:44actually respond to it ideally opt
5:27:47optionally you can do another issue or
5:27:49activity so you can stick with the same
5:27:51issue just do another activity but if
5:27:53you're really interested in privacy and
5:27:54security for example you can just uh go
5:27:56through some of the activities and not
5:27:58change your lens but you can also change
5:28:01your lens to a different issue I'll lay
5:28:02out the issues and repeat them again in
5:28:05in in the next few slides so you should
5:28:07add to your impact analysis as you're
5:28:09varying some of these again this an
5:28:11optional step but I think it would be a
5:28:13very valuable one to do uh in terms of
5:28:16expected results so what do what do I
5:28:18think you you get out of this activity
5:28:19and why are we actually doing this well
5:28:22I think you would actually learn about
5:28:24an activity Centric approach to
5:28:25uncovering issues and concerns it's it's
5:28:29fine to talk about issues broadly but
5:28:31it's often better to root it in specific
5:28:34examples concrete examples so you would
5:28:37also start with an IAL brainstorming how
5:28:39to address issues and mitigate concerns
5:28:41and this will then actually develop our
5:28:44design thinking mindset and that
5:28:45promotes for example ethical and
5:28:48responsible design so ethics are
5:28:50obviously a key and they are they should
5:28:54be like everywhere we should con we
5:28:55should concern ourselves with ethics uh
5:28:59as part of a larger design problem all
5:29:01the time um unfortunately as you will
5:29:04also notice if you start working on some
5:29:06of the other courses especially the
5:29:08development oriented one so the third
5:29:09course in the exam Muk we are going to
5:29:11be so busy with the technology it's
5:29:13actually sometimes really hard to um
5:29:16focus on uh ethics and responsible
5:29:20design at least if you do not make those
5:29:23issues uh explicit and that's why I make
5:29:25them explicit I want I'll remind us
5:29:29every single time so I hope you don't
5:29:31find it too annoying it's just a
5:29:33reminder H and this will help our more
5:29:36technical audience as well um and so we
5:29:39don't want to be blind to these issues
5:29:41and just be like Yay this is cool
5:29:43technology we really want to have a
5:29:45critical discussion so what are these
5:29:46issues again social and ethical concerns
5:29:50obviously accessibility and equity and
5:29:54privacy and security so these are the
5:29:56three big classes of issues these are
5:29:57your lenses that you can adopt as you're
5:30:00working on these activities so what are
5:30:02the activities one activity is a social
5:30:05VR meeting activity so this is is from
5:30:08me going to the i e virtual reality
5:30:12conference in 2020 the one that couldn't
5:30:14take place in Atlanta due to covid-19
5:30:17and the one that I intended uh as many
5:30:19others completely virtually here is
5:30:21actually an example of me walking up
5:30:24these virtual stairs and trying to find
5:30:26some of my friends and waving at them
5:30:29and then h no response but I met
5:30:32somebody else and I had a virtual
5:30:34reality handshake which was a cool
5:30:37experience so you can look at this
5:30:39activity and I hope you will have a
5:30:41similar experience as you're trying out
5:30:44social VR hopefully a positive
5:30:46experience but there is no
5:30:49guarantee then the other activity that
5:30:52I'd like you to consider is an object I
5:30:54call it object recognition
5:30:57activity and in the object recognition
5:31:00activity here I'm showing an example of
5:31:02me trying out Ikea place in the study
5:31:06I'm actually performing object Rec
5:31:07recogition of one of my pieces of
5:31:09furniture you can use other apps to do
5:31:11that for example Google Lens allows you
5:31:14to uh pointed objects and ask Google
5:31:17what it think it is so I wanted to
5:31:20perform one of these activities where
5:31:22you are using your camera to in some
5:31:25sense scan the
5:31:27environment and uh then the computer the
5:31:30computer does something but you don't
5:31:32know exactly how they do it it's just
5:31:34magic and they recognize the object and
5:31:37then obviously can uh continue with the
5:31:39activity so here I'm actually now
5:31:40placing really this object in my living
5:31:43room there is additional if this is
5:31:45really done well done in the future uh
5:31:47that would be additional object
5:31:48recognition to make sure that this
5:31:50actually fits nicely into the room at
5:31:52this stage is a little primitive no
5:31:54occlusion whatsoever uh and so these are
5:31:56things that we talk about later but it's
5:31:58interesting when it comes to these
5:31:59activities and uh how a computer you
5:32:02should think how a computer things uh
5:32:05this idea of human computation uh you
5:32:08uh specifying the algorithm that's a
5:32:10very good approach to uncover some of
5:32:13the issues so who tells this Smartphone
5:32:17uh what kind of furniture uh we're
5:32:19looking at where is that database that
5:32:21they are looking up against for
5:32:24example last activity and this one is a
5:32:26little bit of a well let's say funny one
5:32:28here is an augmented reality photo
5:32:30activity so I want you to do something
5:32:33like this uh here I'm actually using uh
5:32:37an air Cor samples project this is not
5:32:39Snapchat but you could just go ahead and
5:32:41use snapchat it would lead to similar
5:32:43results I'm analyzing a little bit what
5:32:45this lens which is clearly a lens just
5:32:47saying right this is not how I usually
5:32:49look it is a lens applied to my face so
5:32:53face tracking it is interesting so
5:32:55essentially you're pointing the phone at
5:32:57your face and obviously there could be
5:32:59things in the background um that you are
5:33:01showing as well so these are some of the
5:33:04things we should concern ourselves with
5:33:07and then again consider it from these
5:33:09different
5:33:11lenses so the social and ethical
5:33:14concerns in each of these activities um
5:33:17are something to think about uh
5:33:20accessibility and Equity right who can
5:33:22try this out is this accessible what
5:33:24kinds of issues uh might users have with
5:33:29special needs is there anything else we
5:33:31should be thinking about when it comes
5:33:32to accessibility and access so equity
5:33:36and by the way access disbility and
5:33:38access are not the same things even
5:33:39though I hear people always confuse the
5:33:41two but Equity is really this idea of
5:33:44making sure that everybody in dependent
5:33:46of uh social and economic status
5:33:50whatever that means has access to an XR
5:33:54experience so uh technology and the app
5:33:58and there are all kinds of of concerns
5:34:00when it comes to equity one of the big
5:34:02recent ones and I just have to include
5:34:04it here because it just happened is
5:34:06Facebook for example example requiring
5:34:09uh users to register with their accounts
5:34:12uh starting October 2020 that is a
5:34:15really big issue uh along all these
5:34:18fronts and something to consider as well
5:34:21uh in particular privacy and security
5:34:23you will see it will be a major theme so
5:34:26in fact you could take any of these
5:34:28three activities I just showed you and
5:34:30run it through any of these lenses and
5:34:33you would learn a lot and that would be
5:34:34the most comprehensive way to do an
5:34:37impact analysis finally when it comes to
5:34:40impact I want you to also consider what
5:34:42happens before during and after the
5:34:45experience uh as a designer in course 2
5:34:49I would guide you in your thinking to
5:34:51make sure that before your users are
5:34:53trying out an experience you should set
5:34:55the expectations of users for this new
5:34:57reality it's very important and there
5:34:59are a few techniques to do so uh other
5:35:02than just text and long terms to accept
5:35:06uh there could be an onboarding
5:35:07mechanism for example uh something more
5:35:10T tutorial based or uh some kind of
5:35:14instructions that would guide users
5:35:15through this and actually prepare them
5:35:17based on examples for example seeing
5:35:20quick snapshots of the experience and
5:35:23potentially even inserting a warning it
5:35:26could be a trigger warning um that is uh
5:35:29something that also comes up in some of
5:35:31my projects in research that I'll share
5:35:32with you in the second course obviously
5:35:35then design impact analysis during The
5:35:38Experience where as a designer you
5:35:40really have to balance the realism you
5:35:42want to make it as cool as possible and
5:35:44experience obviously but Comfort user
5:35:47comfort and
5:35:48safety you can't really compromise on
5:35:50them so think about um how much uh is
5:35:54what I just said how much of this is
5:35:56actually uh violated in some of the
5:35:59activities that I suggest here and then
5:36:02finally uh obviously as a designer you
5:36:04should help users transition back to the
5:36:05old reality so if they is onboarding
5:36:08this would be the offboarding so more
5:36:11like a debrief maybe some kind of
5:36:13support in this transition period
5:36:16obviously the longer we for example
5:36:18spend in Social VR the more we might
5:36:21identify with it one of the big dangers
5:36:23that I see for example uh especially
5:36:26with children and uh maybe different
5:36:29kinds of audiences as well so really
5:36:32take a
5:36:33diverse inclusive mindset here and don't
5:36:38just think about yourself but obviously
5:36:39that's a starting point but as you
5:36:42explore these different concerns and as
5:36:44you're trying out these activity so
5:36:46really as you're navigating this whole
5:36:48uh landscape of issues um I think this
5:36:52impact analysis will help you the
5:36:54techniques behind it um that I wanted to
5:36:57formalize here a little bit so using
5:36:59these lenses and following my protocol
5:37:04the task description to make things fair
5:37:06so really stating the issue first and
5:37:08then stay focused uh you think it l and
5:37:11write up your concerns um this I think
5:37:15will be a good way for us to think about
5:37:17it obviously in the research domain uh
5:37:20there are even more um formalized uh so
5:37:23privacy impact analysis is or assessment
5:37:27uh there are more formalized methods so
5:37:29this is a little bit of an intuition a
5:37:31little bit more um well broad audience
5:37:35uh scoped activity here but I think
5:37:37it'll it will uh get us into the right
5:37:40mindset so let's hope you en enjoy some
5:37:44of these activities uh I have a few more
5:37:46pointers of how to access social VR
5:37:49platforms for example where you can find
5:37:51them what is the software uh but please
5:37:54also help each other this is a in a
5:37:56rapidly evolving space so if someone
5:37:59tries out something new and has a
5:38:01positive or potentially negative but not
5:38:04traumatizing or otherwise dangerous
5:38:08um so uh experience please share and
5:38:11obviously if something goes really wrong
5:38:14then we should just warn each other as
5:38:16well and there might be new products
5:38:19coming out it's hard for me to
5:38:20anticipate uh we don't have a lot of
5:38:22regulations in this space yet so um I
5:38:26hope we're going to learn something
5:38:28treat it as a learning experience be
5:38:30careful um I think the activities I
5:38:33chose shouldn't be too dangerous but um
5:38:36if you watch Black Mirror a little bit
5:38:38uh obviously social platforms are uh
5:38:42actually come with a certain danger
5:38:45harassment is a really big issue there
5:38:48and then there not not going to be
5:38:50better in virtual reality for example
5:38:52and so I hope you have some pleasant
5:38:54experiences and they will still be worth
5:38:57reflecting on you don't have to
5:38:59experience the most negative versions of
5:39:02some of these realities so I hope you
5:39:05enjoy this and we're going to learn a
5:39:06lot together together and we can share
5:39:08our experiences and learn from each
5:39:10other again this is the end of our
5:39:13honors track but only in this course
5:39:15please consider taking some of the other
5:39:17courses focused on design and
5:39:18development would be great to see you
5:39:20back there and then I'll see you next
5:39:22week uh where we're going to think more
5:39:25about strategy and I have a few cool
5:39:28panels uh with both leadership at UFM
5:39:31and people in various domains and
5:39:34positions and then also students which I
5:39:36think is great and and hope to see you
5:39:50there in this video we're going to talk
5:39:53about developing your XR strategy um so
5:39:56this is really designed to be a lecture
5:39:59around this idea of growing your
5:40:02experience your knowledge your team your
5:40:06equipment
5:40:08making a good strategic decisions now
5:40:11I'm not a businessman I mean what do I
5:40:14know about XR strategy uh so I would
5:40:17argue that over the last uh five years I
5:40:20have developed a little bit of a
5:40:22strategy uh so as a professor at the
5:40:24University of Michigan obviously my role
5:40:26is to um do research so I have to grow a
5:40:30team in fact I actually have to train
5:40:33students and I chose to train students
5:40:36in this area of x
5:40:37then I'm also involved in some of the
5:40:40more strategic uh decisions at the
5:40:43University of Michigan So currently I am
5:40:45the XR faculty innovator and residents
5:40:48is part of the University of Michigan
5:40:50wide XR initiative that actually spans
5:40:53all 19 schools and colleges still I
5:40:56don't know too much about business so
5:40:58don't think of this as a business
5:41:00strategy lecture but think of it as a
5:41:03well a lecture where I'm trying to share
5:41:06some of the insights I've gained over
5:41:08the past few years so I wanted to start
5:41:11at the beginning what are really the
5:41:13considerations that you have to make I
5:41:16was thinking well you have to think
5:41:18about knowledge so what what knowledge
5:41:21do we actually have on the team and what
5:41:24do you want to learn so what do you want
5:41:26the team to learn how to diversify where
5:41:29to acquire new knowledge um so we're
5:41:32going to talk about that a little bit
5:41:34and that obviously is a lot about team
5:41:36so strategic considerations have a lot
5:41:38to do with team building and what kinds
5:41:40of talent do you need to have on your
5:41:42team so what roles do you have right now
5:41:45and what is actually a good composition
5:41:46what are New Roads you need to create
5:41:49obviously XR is a lot about equipment so
5:41:52uh basically just buying things when
5:41:53there is a need and a need means there
5:41:55is a project and potentially funding
5:41:58behind the project also like what kinds
5:41:59of tools what kinds of tools do I learn
5:42:02and what kinds of how do I embed them in
5:42:04into the process what is the workflow
5:42:06and then users first of all obviously if
5:42:08you have a product or service in this
5:42:10space who are your users and the target
5:42:12users and then uh what tasks do you want
5:42:15to enable obviously what problems do you
5:42:18want to solve but that's that's
5:42:21obviously uh the first important thing
5:42:23to ask now I'm going to talk about each
5:42:26of these so knowledge team equipment and
5:42:28users and so we're going to go through
5:42:29each of these and thinking uh we're
5:42:31taking a mindset of growth a growth
5:42:33mindset here so on the knowledge side um
5:42:37so it helps to have knowledge obviously
5:42:39in the applications I don't mean like
5:42:41I've played beat saer and Pokémon Go I
5:42:43mean that's maybe the first step no I
5:42:45mean really like having an overview of
5:42:47the kinds of applications that are out
5:42:48there we need to have decent knowledge
5:42:50about the Technologies if for any kind
5:42:52of new solution that we create or want
5:42:54to create we really have to have to have
5:42:56a basic understanding of what kinds of
5:42:58technological options uh do we have you
5:43:00definitely want knowledge in the space
5:43:02of what are the key issues uh right so
5:43:05somebody who thinks through this social
5:43:07the ethical concerns uh the ethical
5:43:09concerns are really important and and
5:43:11how they influence design how they drive
5:43:13design how they inform design you want
5:43:15to have knowledge in design so uh in
5:43:18this space obviously a traditional or
5:43:20let's say a very basic uh ux background
5:43:23is is a good start you want to have
5:43:25knowledge in the development uh tools
5:43:27and environments you want to have an
5:43:30understanding of how to build against
5:43:31some of these Technologies what are fast
5:43:34uh and Rapid test driven development
5:43:37Cycles how do you quickly deploy to
5:43:39these platforms how do debug and what
5:43:41debug actually means and how to inspect
5:43:44XI experiences so that's definitely
5:43:46where you want to have knowledge you
5:43:47also want to have knowledge on on the
5:43:50management side um so building uh a team
5:43:55uh of uh people that can innovate and
5:43:58work in this exis space is is hard it
5:44:02definitely um requires experimentation
5:44:04I'm not sure I have the solution I've
5:44:05been experimenting over the last few
5:44:07years uh obviously management also has
5:44:09to do with uh financial decisions and
5:44:12including also what kinds of equipment
5:44:15and tools you may want to acquire and
5:44:18invest so growing the knowledge well uh
5:44:22it often just takes one person an
5:44:23evangelist like this person here a VR
5:44:27adopter walking around and showing it to
5:44:30other people a strategy that we've often
5:44:32used and this may maybe works best in
5:44:34academic context but in general is a
5:44:36reading group like bringing together
5:44:37people uh from a variety of research
5:44:40groups or research areas so I'm involved
5:44:42in a few reading groups not always as
5:44:44active as I I wished but um there is one
5:44:49uh by academic Innovation here that's an
5:44:51nity at the University of Michigan it's
5:44:53quite broad drawing in a lot of people
5:44:55and we did a reading group around XR one
5:44:58thing that I think is very useful is a
5:44:59community of practice having this
5:45:02opportunity maybe once a month to gather
5:45:04and bring together people from across uh
5:45:07the well institution let's say so the
5:45:10University or the company or the little
5:45:14team that you work on uh maybe bring in
5:45:16people from the outside for some
5:45:18technology demos we've done this quite a
5:45:19few times let's talk about the team so
5:45:23when it comes to the team obviously you
5:45:24can think terms of roles so I need a
5:45:26designer on my team do I need more than
5:45:28one designer I don't know I need at
5:45:30least one developer somebody who really
5:45:32understands that technology maybe
5:45:34somebody who knows
5:45:35Unity um do I need a manager am I the
5:45:38manager so that's something to think
5:45:41about uh in the Exile space uh how much
5:45:43is there a place for artist is actually
5:45:45a really strong place for artists and
5:45:48it's a really good place and then
5:45:50researcher uh researchers are really um
5:45:53both on the academic side or user
5:45:55experience research side so a little bit
5:45:57more practitioner oriented like finding
5:45:59ways of well finding bugs and finding
5:46:02issues and improving the product as
5:46:04opposed to finding evidence for a
5:46:07hypothesis and and doing some more
5:46:09rigorous um scientific research uh how
5:46:12much do you need to have somebody with
5:46:15an entrepreneurial character uh on your
5:46:17team um so somebody who manages and goes
5:46:21beyond and maybe drives Innovation and
5:46:25also thinks about acquiring funding um
5:46:30so I feel like as a professor I'm like
5:46:31in each of these roles a little bit um
5:46:34but I'm going to share a little bit my
5:46:35strategy of how I tried to grow my team
5:46:38at least so the first thing I usually
5:46:41look for still is actually experience
5:46:42with development um I just like these
5:46:45Technologies are quite technical so in
5:46:47some sense it's a filtering mechanism um
5:46:50if if you're not technical that doesn't
5:46:52mean that I will not look at your
5:46:54materials um but I'm looking for
5:46:57experience with XR any kind of
5:46:59development experience with XR if not
5:47:02then at least some kind of mobile web
5:47:04development experience um I'm looking
5:47:07for uh portfolio I'm looking for
5:47:10projects in that portfolio and I'm
5:47:12looking for the complexity of the
5:47:13prototypes that were created and also
5:47:15the the size and the complexity the
5:47:17composition of the teams that somebody
5:47:20has worked in so have they worked
5:47:22actually in teams or is everything
5:47:23individual I'm looking for roads whether
5:47:26there's a lot of independent work or a
5:47:28lot of um well teamw work and then to
5:47:31which extent was it leadership and
5:47:33that's something that I'll just think
5:47:34about I am looking for experience with
5:47:36with design um this is uh often still
5:47:40very necessary but most of the times I'm
5:47:43looking for somebody who can Bridge both
5:47:44design and development I'm very
5:47:46interested in whether somebody has
5:47:48experience with running uh and designing
5:47:49and conducting user studies um so that
5:47:52was very important and finally obviously
5:47:55there's always this adding this any kind
5:47:58of talent to the team so is there
5:48:00experience with X where X is really new
5:48:02and if is if this new thing would be
5:48:04significant is always a place uh in my
5:48:07team at least and then when I bring
5:48:10somebody into the team I'm usually
5:48:13thinking about how to make this a
5:48:16productive uh and uh useful learning
5:48:19experience uh for the new team member uh
5:48:23I'm asking myself how can they grow by
5:48:25working with us like what do they get
5:48:26out of it what what can we provide in
5:48:29fact mentorship is something that I
5:48:31really think about can I actually how
5:48:32can I enable them uh what can I provide
5:48:36uh are there uh things that they uh need
5:48:40to grow in and do I have the knowledge
5:48:43uh to an experience to support them and
5:48:45do I have the time um and then I'm
5:48:48looking at opportunities so what do they
5:48:51actually add to the team if they were to
5:48:53join us and this is a little bit more
5:48:55generic and I do make a risk analysis so
5:48:59I'm thinking about the challenges of
5:49:00bringing somebody into the team so how
5:49:02much overlap is there with existing
5:49:04skills how much do we grow um um
5:49:08thinking about Team Dynamics and all
5:49:09those usual things as well so that is
5:49:11not specific to
5:49:12XR thinking about equipment so
5:49:15throughout this course we really made an
5:49:17effort of bringing in all the equipment
5:49:19or a lot of the equipment that I have
5:49:20access to uh some of it I bought uh some
5:49:24of it um I was given by vendors uh
5:49:28sometimes the school chipped in some
5:49:30money but a lot of it was actually me
5:49:34sourcing if you will the equipment and I
5:49:37just show basic devices here but there's
5:49:39a lot more custom stuff in my lab um so
5:49:42this question around equipment is really
5:49:43an important one so when can you use an
5:49:45off offthe shelf device and and which of
5:49:47these do you actually buy growing the
5:49:50equipment really I would say first of
5:49:53all your phone is a great start I mean
5:49:55if you're not running a team or
5:49:56something if you're just by yourself or
5:49:58exploring this in a little group um
5:50:01really start from what you have which
5:50:03most likely is a smartphone and then
5:50:04explore ways of turning it into an a
5:50:06device or Mar based or even Mar less if
5:50:09it's if it has support for it or into a
5:50:11VR device um my approaches always been
5:50:15to be to go more broad um rather than
5:50:18deep so what I mean by that is I don't
5:50:20buy like x times the same device I'd
5:50:24rather have a variety of devices so I'm
5:50:26oing always for a good mix both of AR
5:50:28and VR
5:50:29devices and uh the beauty is when you
5:50:32have a variety of devices they can be
5:50:35combined in interesting way so I do have
5:50:37a number of just kcks still around we
5:50:39have some Intel re sends so we can use
5:50:42some of these deps cameras um to support
5:50:45some of our more custom kinds of setups
5:50:48I've been building my own displays uh
5:50:50you can also combine your traditional
5:50:52like smartphone and touch with a hollow
5:50:54lens uh tablet or smartphone touch with
5:50:56a hollow lens and can build new kinds of
5:50:58experiences that way um at least that's
5:51:00how it's often approached in research so
5:51:02I've I've usually looked for this power
5:51:05in combination and uh really trying to
5:51:07go complimentary in terms of
5:51:10Technologies and then what is really
5:51:11important to me is that we actually have
5:51:13equipment that is set up that is always
5:51:17available and ready I usually try out
5:51:19the latest I try um try it out uh but no
5:51:23need to buy it yet I never buy in
5:51:25bug and so one thing that I'm often
5:51:28asked and I don't know where I didn't
5:51:29really know where to fit it in but I
5:51:31thought this is a good place uh is when
5:51:33it comes to buying new equipment so look
5:51:35for a a powerful graphics card an
5:51:38example here is the Nvidia RDX series uh
5:51:41you're looking for a decent uh CPU um i7
5:51:45you can get away with less the most
5:51:46important is really the the graphics the
5:51:48GPU and then people always think they
5:51:50need a lot of ram like 32 gigs and well
5:51:53that is cool and it's actually not that
5:51:55expensive I think all you need is 16
5:51:57gigs um at least at this stage unless
5:52:00you really loading a lot of like you
5:52:02know very complex models a lot of stuff
5:52:04that you need to actually keep in memory
5:52:07but my thinking is GPU larger than CPU
5:52:10larger than Ram so higher priority so
5:52:13then let's think about users so
5:52:15internally in your uh Team you really
5:52:18want people to adopt more of these
5:52:20Technologies so often takes one person
5:52:23if you are a user but more like with a
5:52:26designer or developer background then
5:52:28what you really need to do is you need
5:52:29to embrace these XR Technologies you
5:52:31need to learn about the methods and
5:52:33tools how some of your traditional
5:52:34methods and tools need to be adapted
5:52:37uh you need to help create design
5:52:38knowledge and by by doing um not by
5:52:41thinking about it but by doing projects
5:52:43and uh creating helping create design
5:52:45guidelines and revising guidelines and
5:52:48also creating some kind of design
5:52:49patterns as a researcher or instructor
5:52:52uh you really have a place here because
5:52:56you can shape the future of XR you're
5:52:58actually training the talent so if you
5:53:00can somehow bring it into your classes
5:53:02uh as a way of growing users um uh
5:53:07that would be the user base that would
5:53:08be uh be really
5:53:10good I'm always looking for creating new
5:53:13learning opportunities um so even in a
5:53:16class that is not focused on design or
5:53:18development or anything like that as a
5:53:21researcher or instructor Your Role is
5:53:23actually to spread the knowledge as a
5:53:25manager and entrepreneur I think you
5:53:28really need to know uh the players the
5:53:31key players like who are actually the
5:53:34people where is essentially fun fing
5:53:37available um and then develop some Exile
5:53:40strategies so growing the users really
5:53:43uh could be through seminars right so
5:53:45you have an evangelist or somebody who
5:53:47knows the stuff and then talks about it
5:53:50I think better yet is actually giving
5:53:51demos so really uh learning by doing or
5:53:55trying things out I think is very good
5:54:00uh workshops I it's kind of like a
5:54:02combination of seminars and demos a
5:54:04workshop is really much more active I
5:54:07think it's is one of the best ways to
5:54:08learn about these things uh one tool
5:54:11that I think I have used and it's been
5:54:13very successful actually in my little
5:54:15career here at Michigan is design gems
5:54:18side effect of each of these activities
5:54:19is after that somebody will walk away uh
5:54:23that has not seen uh XR yet and they
5:54:26will then go to the next person and
5:54:28maybe become an evangelist and so you
5:54:30know this then creates an interesting
5:54:33cycle and hopefully spreads the
5:54:35knowledge yeah so that um brings me to
5:54:39the end uh at least I wanted to convey a
5:54:42few ways to think about X our
5:54:45strategy uh not just in terms of
5:54:47promotion and marketing but really A
5:54:49much broader view on things so how do
5:54:52you um what kind of knowledge do you
5:54:54have how do you grow the knowledge who
5:54:55do you have on the team how do you grow
5:54:57the team uh equipment how do you make
5:55:00good decisions there and then users and
5:55:03and users was really was really broad
5:55:05not just like the users that you're
5:55:06designing for but also like growing the
5:55:09user the user base of XR okay so let's
5:55:12wrap up here keep in mind um whatever
5:55:15you do in this XR space make it a
5:55:18regular thing so maybe bi-weekly reading
5:55:21groups or maybe even weekly uh some kind
5:55:23of monthly community of practice some
5:55:26kind of ritual that people know is
5:55:28happening and if they miss out on one
5:55:30opportunity they know the next one is
5:55:31coming up I would always encourage you
5:55:33to try out different things so go for
5:55:35mix been experimenting over the last few
5:55:37years I've tried out a mix of kind of
5:55:40like seminars demos design gems all
5:55:44these kinds of things um and interest uh
5:55:48in these things also shifts I don't
5:55:50think uh that you have to stick to one
5:55:53of those you can offer different kinds
5:55:56of these opportunities to for others in
5:55:59the community to engage with you I think
5:56:01you should aim to grow from
5:56:03within um so obviously it's not nice to
5:56:06look for collaborators and like-minded
5:56:09people outside your groups and your
5:56:12teams but um bringing in Knowledge from
5:56:15the outside is not as effective I would
5:56:18say and so don't just look for other
5:56:21people working in the exr space but um
5:56:23be strategic also here bringing
5:56:26enthusiasts uh from the outside so and
5:56:30by enthusiasts I mean people who are
5:56:32passionate potentially about XR but uh
5:56:35they are Prim arily domain experts in
5:56:37their specific
5:56:38domains again I hope this was uh kind of
5:56:42useful and I look forward to uh some of
5:56:45the feedback and hearing uh some of your
5:56:47thoughts about this lecture I'm sure I
5:56:49can improve uh some aspects also with
5:56:52your help and over the next few years
5:56:54but um this is the knowledge that I
5:56:57thought I could share at this stage and
5:56:59maybe it's the starting point um for an
5:57:01interesting discussion Among Us uh and
5:57:04also learning about other strategies
5:57:05that others have employed so um well
5:57:09thanks for watching
5:57:29this welcome everybody um I'm looking
5:57:32forward to having a discussion with a
5:57:34few invited experts faculty managers
5:57:37Deans and um all the people involved at
5:57:40the University of Michigan where I want
5:57:42to talk about XR and strategy like
5:57:47Michigan as a university um has a lot of
5:57:50things going into XR at the moment and
5:57:52in this segment I want to unpack what
5:57:55that means um for different people
5:57:57working at different levels in the
5:58:01University I want to structure this
5:58:02discussion around two major themes the
5:58:06First theme is really about the current
5:58:09the now what are we doing right now what
5:58:11kinds of initiatives are going on right
5:58:14now and now I would like to start with a
5:58:17brief introduction to who we have today
5:58:21joining
5:58:22us okay so let's get started uh why
5:58:25don't you say who you are and what
5:58:26you're doing at Michigan my name is
5:58:28Michelle abersold I am a clinical
5:58:31professor in the School of Nursing and I
5:58:33also have an appointment at the school
5:58:35of
5:58:36information and I'm delighted to be here
5:58:39today uh most of my scholarship is
5:58:42really in the area of simulation based
5:58:45education including a lot of work in XR
5:58:48hi I'm Joanna Mari milenic I am uh the
5:58:51associate de for undergraduate education
5:58:53in the College of Engineering I'm also a
5:58:55professor of material science and
5:58:56engineering and um my role has been both
5:59:00to do XR research uh in our lab we're
5:59:03interested in trying to use XR to teach
5:59:06Concepts and Material Science but also
5:59:09as associate Dean to develop uh ways and
5:59:12the infrastructure for faculty to be
5:59:14able to take these kinds of Technologies
5:59:16and apply them in their own classrooms
5:59:18my name is Dan fesan I'm associate
5:59:20director for one of our centers on the
5:59:22north campus the D Center and I lead the
5:59:25emerging technologist group I'm also a
5:59:28member of the College of engineering's
5:59:30IT department called cane and my rule is
5:59:33to uh help evolve technology so we've
5:59:37been involved in XR for a long time but
5:59:39we uh my team develops spaces and we
5:59:43actually have a team of developers who
5:59:45actually build applications for
5:59:47different uses on campus my name is
5:59:50Jeremy Nelson I am the director of the
5:59:52XR initiative at the University of
5:59:53Michigan we've been tasked with three
5:59:56major goals uh to bring XR Technologies
5:59:59for teaching and learning to our 19
6:00:02schools and colleges in the residential
6:00:03curricula at the University of Michigan
6:00:05an arbor campus
6:00:06uh bring these Technologies to our
6:00:08online curriculum which is evolving
6:00:10quickly as well as creating Innovative
6:00:13public private
6:00:15Partnerships well this is great and such
6:00:18a nice set of participants uh that we we
6:00:20can learn from today so I wanted to
6:00:24structure this discussion first around
6:00:27um scale now scale is really when it
6:00:31comes to XR we often start small and but
6:00:35at the University level we really want
6:00:37to scale very quickly so I want to learn
6:00:39from you what that means in each of your
6:00:41projects and then as we go uh across the
6:00:44university what that means up to the
6:00:46scale of the University so Michelle I
6:00:49wanted to start with you and learn about
6:00:51some of your XR projects and initiatives
6:00:54and how you deal with the question of
6:00:57scale there both in terms of like how
6:01:00many students may be involved or how
6:01:03many subjects may be involved and tell
6:01:04us a little bit about
6:01:06the kind you can pick a project and tell
6:01:08us a little bit about um participants
6:01:10and Technologies and how you deal with
6:01:13scale so the whole question of scale I
6:01:16think is really an important one when
6:01:18you're thinking about XR Technologies
6:01:21and sometimes I also think that that
6:01:23might drive the choice of technologies
6:01:26that you're using
6:01:28so two of the projects that I'm working
6:01:31on right now I think about um both of
6:01:34them have a little diff scales so the
6:01:38first project that I actually got
6:01:39started with is a virtual
6:01:43reality program that will focus on
6:01:46helping practicing nurses Physicians and
6:01:50pharmacists really learn about teamwork
6:01:52and
6:01:53communication and this environment is a
6:01:57pediatric emergency room and so it will
6:02:00be multiplayer and small teams will come
6:02:04together using
6:02:06um an Oculus or a HTC but it will use
6:02:11the high-end devices and so they'll come
6:02:14together they'll practice their skills
6:02:18they'll practice communicating and
6:02:19they'll practice their teamwork so this
6:02:22particular project although we would
6:02:24like to get many teams to participate in
6:02:28this the scale is going to be somewhat
6:02:31limited by the number of high-end
6:02:34devices that we have have the riffs and
6:02:37The Vibes and being able to bring people
6:02:40together to do that so I look at this
6:02:42project as a little smaller scale now
6:02:46another project that we're involved in
6:02:49that was just uh funded through AI who
6:02:53is going to partner with us is a project
6:02:57where we're really going to help nurses
6:03:01and pharmacists understand some of the
6:03:04devastating effects of chemotherapy
6:03:07drugs when those drugs are
6:03:10administered intravenously and something
6:03:13goes wrong and those chemotherapy very
6:03:17potent hazardous drugs leak outside the
6:03:20blood vessel or leak outside the port
6:03:24and destroy the skin and the the
6:03:28internal skin and we don't see that
6:03:30right away so it's although we can
6:03:33explain that and we can show people the
6:03:36devastating effects they don't really
6:03:38see what's happening inside the body so
6:03:39we're kind of calling this project Under
6:03:41the Skin now that project we want to
6:03:44scale up and scale up big so we want to
6:03:47be able to bring this to 60 participants
6:03:50who come to our workshops we want to be
6:03:53able to take this to maybe a hundred
6:03:56graduate nursing students who are in our
6:03:59assessment
6:04:00Labs pharmacists so we're using the
6:04:03Oculus go because that's something where
6:04:06we can get many headsets and distribute
6:04:09them and put this out widely so I think
6:04:12for us sometimes it's driven by need and
6:04:16then how sophisticated the device is we
6:04:19use yeah this is great and it also
6:04:22Echoes like one of my observations In
6:04:24from from working with with many of you
6:04:26and also from my own work that um large
6:04:29scale often means low Tech because
6:04:31that's a way to scale and and and work
6:04:34with the budget constraints that we have
6:04:36so this lowest common denominator
6:04:38approach is one way to go about it so uh
6:04:41Joanna I was wondering if we are now to
6:04:43talk about the college level the College
6:04:46of Engineering specifically in the
6:04:48University of Michigan I know that one
6:04:50of the initiatives you have going on is
6:04:52the avmr clinic and I wanted to learn uh
6:04:57how you deal with scale at that level
6:05:00thank you that's a great question you
6:05:02know so just as we heard before you know
6:05:03there are different kinds of
6:05:04technologies that you can employ you can
6:05:06do a rift or you can do a an Oculus go
6:05:09and so forth you know one of the ideas
6:05:10that we had early on is to use mobile
6:05:13devices to do augmented reality for
6:05:15instance um and so that was the idea
6:05:18that I first started working on as an
6:05:19individual researcher when I got into
6:05:22the the dean position one of the things
6:05:24that I began to realize is that as you
6:05:26go to different kinds of classrooms and
6:05:29different kinds of contexts the students
6:05:31need to understand how to interact with
6:05:34the uh with the
6:05:36a mixed reality itself and we don't
6:05:38actually have a very good way of doing
6:05:40that yet there is no normative context
6:05:44you know like in a lecture hall people
6:05:45know you know you sit you know there's a
6:05:47there's a a a screen or a blackboard at
6:05:51the front and everyone faces the front
6:05:53and and listens to a lecture and that is
6:05:55the normative context but there is no
6:05:57normative context for XR kind of
6:06:00applications now I don't think that I
6:06:03can figure out what the best Norm
6:06:05context is myself instead what it has to
6:06:08do is we have to have a lot of different
6:06:09people playing um in these spaces to
6:06:13kind of you know figure out what the
6:06:15best normative context is and so our
6:06:17approach is to get these tools in as
6:06:21many people's hands as possible and to
6:06:24get different faculty members to think
6:06:27about what makes sense in their context
6:06:30and um by having a lot of smaller
6:06:32experiments and and like really lowering
6:06:35the barriers to adopt these kinds of
6:06:38Technologies in the classrooms I think
6:06:40that we'll be able to get to a normative
6:06:42context uh where XR learning kind of
6:06:46makes sense um so that's been our
6:06:49approach so rather than you know having
6:06:52you know one person think about what is
6:06:54the best way to do it in this particular
6:06:56context is to just lower barriers get
6:07:00people to uh work with um developers
6:07:03like in uh Dan fesan group get people
6:07:06working with uh phds who know about
6:07:10pedagogy and know about educational uh
6:07:13research in ways that um engineering
6:07:16professors don't necessarily know
6:07:17getting those people working together to
6:07:20really understand what are the best ways
6:07:22to um to deploy these kinds of
6:07:25Technologies in in different kinds of
6:07:27context so it's it's a it's similar but
6:07:29different we're trying to get the
6:07:31Technologies in as many people's hands
6:07:32as possible and that's how we're dealing
6:07:34with the scale
6:07:36thank you Janna and and I know that one
6:07:38of one um potentially small but yet
6:07:42significant thing that you've been doing
6:07:43is running a series of workshops uh
6:07:45bringing people together who want to
6:07:47learn about how to actually teach uh or
6:07:50how to actually engage with a avmr or XR
6:07:53Technologies and so I was wondering um
6:07:56these workshops I've participated in one
6:07:58of them I think um what what would you
6:08:00say you have learned from running
6:08:02workshops with faculty and students who
6:08:04want to learn about
6:08:06XR the thing that I've learned is that
6:08:09we have so much more to learn you know
6:08:12it's it's the thing that really strikes
6:08:14me is that this is such a wide openen
6:08:17space that no one really has the answer
6:08:22and that you know there's a lot of
6:08:24different ideas out there and there's a
6:08:25lot of like contradictory kind of
6:08:27findings that people are are are
6:08:29reporting on and there's there's no
6:08:32single way which means that we need more
6:08:34people in the space we need more people
6:08:36generating more data generating more
6:08:40experiences and and you know we need
6:08:42more interaction between all of us to
6:08:44figure out what are the best ways and
6:08:46what are the the worst ways so that we
6:08:47can get rid of those things that don't
6:08:49seem to work really well and move on um
6:08:53and start doing different kinds of work
6:08:55that is really impactful for the student
6:08:58so now Jeremy um in some sense uh you
6:09:01probably have the birds eye view at
6:09:03Michigan right now um um you have been
6:09:06with us for some time now and I know as
6:09:08one of your first steps you've actually
6:09:10reached out and talked to a lot of
6:09:12people so it seems like one of the big
6:09:15questions you had for yourself first was
6:09:17like what is actually the current scale
6:09:19at Michigan uh and then how do we maybe
6:09:22extended and accelerated so can you tell
6:09:24us a little bit about your steps first
6:09:27steps and and how do you how do you make
6:09:29a sense of the scale of XR work at
6:09:32Michigan yeah no thank you all yeah Ian
6:09:35the first thing I really wanted to
6:09:37understand is is where were those
6:09:38pockets of expertise who was building in
6:09:41space who was teaching with the
6:09:43Technologies and I think the folks we
6:09:44have today today together on the panel
6:09:46of leading that space um so I really
6:09:50tried to meet with you know as many
6:09:52faculty staff students uh other
6:09:56non-academic units that are in the space
6:09:59to really understand where are the you
6:10:01know where are the folks that can build
6:10:03the technology who are the people
6:10:04consuming what do they want to explore
6:10:07what types of equipment do we have and
6:10:09so really try to build that sense of
6:10:10community and as well as meet with
6:10:12faculty and in other schools or colleges
6:10:15that aren't even thinking about it yet
6:10:16to help understand where where they are
6:10:19in the adoption phase or contemplative
6:10:22steps uh and then share with them what
6:10:24we've seen in other places to help build
6:10:26that uh excitement interest and and so
6:10:30you know there's been a lot happening
6:10:32obviously the the work here at Michigan
6:10:34didn't start with the XR itive there's
6:10:35been a lot of work happening for years
6:10:38and really tried to to bring that
6:10:40together and and see where there's ma
6:10:41critical mass that we can support um so
6:10:44a few things you know I'm thinking about
6:10:46ways that we can create Frameworks or
6:10:49processes or tool sets to help uh enable
6:10:52scale for you know faculty to to begin
6:10:55to bring their assets or to bring their
6:10:57um Concepts to a platform or to the
6:11:00technology that they can bring to the
6:11:02class so particularly um there's a lot
6:11:06of work that can be done with 360 video
6:11:08there's some platforms that we're
6:11:09exploring there where you can use
6:11:11traditional video methods and Inter and
6:11:15insert digital elements and digital
6:11:17interactions uh to to scale out
6:11:19development on that phase as well as we
6:11:22launched a seed fund to Spur Innovation
6:11:26and the firstr of projects so we just
6:11:29announced six projects that we'll be
6:11:30working on Michelle's project is one of
6:11:32those six um we'll be helping build a
6:11:35virtual nuclear reactor for nuclear
6:11:37engineering working with folks in our
6:11:39school of music theater and dance to
6:11:41create some toolkits for immersive audio
6:11:44be working with physics to create a
6:11:45physics lab so lots of pretty exciting
6:11:47projects to get going and get students
6:11:50using that building on what Joanna was
6:11:52saying about getting
6:11:54more experiences in the hands of the
6:11:56faculty and the students so we can
6:11:58continue to learn and and develop okay
6:12:02this is great uh thanks for sharing uh
6:12:04that now Dan you have been patiently
6:12:06sitting with us and I know that you have
6:12:08also been uh someone who has been very
6:12:10active for a long time in this space um
6:12:14one thing I wanted to ask you directly
6:12:16and specifically is I know that you have
6:12:18been running uh the XR community of
6:12:22practice um for some time now and so I
6:12:26want to make a transition from um scale
6:12:29to also approach like I want to learn
6:12:31more about how can we Implement some of
6:12:33that scale how how can other people
6:12:35learn from this like what we are doing
6:12:36and also how can maybe other people
6:12:38improve on what we're doing so um this
6:12:40extra community of practice I was
6:12:42wondering what was your idea in terms of
6:12:45like who did you want to bring together
6:12:47who do you bring in from the outside and
6:12:50how would you characterize the learning
6:12:52that is going on and the exchange that
6:12:54is going on in that community of
6:12:56practice so my uh goal to start the cop
6:13:00was as as we're all familiar University
6:13:02of Michigan is a big diverse Institution
6:13:05that uh sort of um encourages people to
6:13:10uh think locally develop ideas and you
6:13:13know so a
6:13:14thousand seats can Sprout but in a lot
6:13:17of ways we do have a lot of duplication
6:13:20of effort that happens in lots of places
6:13:23so the idea was that we uh for a long
6:13:27time the D was seen as being in a
6:13:29leadership position in in XR development
6:13:32and we I wanted to open a platform for
6:13:33us to be able to share share all the
6:13:36work we're doing all the work we've done
6:13:38but at the same time also to invite
6:13:40others who were doing amazing work in
6:13:42their labs and for class classroom use
6:13:45to say share what you know what are
6:13:48where are you where are things going and
6:13:50sort of expand on on the ideas and what
6:13:53you would think you might be doing in
6:13:54the next two three years so that we can
6:13:56bring the community together so that we
6:13:59can actually um build a scaffolding that
6:14:02everybody can leverage as opposed to uh
6:14:05building a lot of silos where we we're
6:14:07Reinventing the wheel lots of metaphors
6:14:10but uh so that was the concept so it's
6:14:13been it's been great to see it evolve
6:14:16and I we just made it open so we got we
6:14:18have faculty who show up at these
6:14:20meetings we have students who show up
6:14:22meetings we have staff who are working
6:14:24in Labs supporting faculty work in these
6:14:27meetings and usually we give them a
6:14:30platform that to be able to present on
6:14:32their ideas and we intentionally
6:14:35designed so that meetings can actually
6:14:37uh rotate around campus so we go to your
6:14:39lab your space your environment so you
6:14:42can share uh what you're working on and
6:14:44what you're uh experimenting with and
6:14:47it's been a great way for us to sort of
6:14:49an email group that sometimes gener
6:14:51traffic for people saying have you seen
6:14:53this and it's been great to have it
6:14:55evolv uh over time and so that was the
6:14:58sort of the intention I think it's been
6:15:00uh it's been evolving to meet those
6:15:03needs yeah that's great and then I know
6:15:06that we also often get like requests
6:15:08from the outside of people who want to
6:15:10like come in and present some of the
6:15:12technology and I think that's that's a
6:15:14really great way to disseminate it to a
6:15:16lot the knowledge to a lot of people so
6:15:19the uh I wanted to ask you a more
6:15:22specific question about approach this is
6:15:24a really difficult question but I
6:15:26thought Dan you are the man for this one
6:15:29so um you have in your in Your Capacity
6:15:33uh managing a team working on on a
6:15:35variety of XR projects for some time now
6:15:38I know when you ask me this question I
6:15:40would tell you well each project is
6:15:42specific and needs a tailored strategy
6:15:45and a specific approach so now here's my
6:15:47question for you do you think if you
6:15:50reflect on the work is there like a a
6:15:52common approach that you take when you
6:15:55to taking on a new XR project um are
6:15:59there some common metrics or some common
6:16:01key components that you look for or is
6:16:04or is really every project so different
6:16:06that there is no common
6:16:09strategy that kind of touches on your
6:16:12question of scale as well because in a
6:16:14way uh I always go back to sort of
6:16:17mission of the DOR stat Center which is
6:16:20what I try to drive our our identity
6:16:23which is we are supposed to be in a
6:16:25space where we are experimenting and
6:16:27thinking about where things are going as
6:16:29opposed to where things are in a lot of
6:16:31ways we're supposed to be um uh as a Dr
6:16:35fou has been quoting grety which is
6:16:37we're supposed to be where the Huck
6:16:39where the puck is going to be as opposed
6:16:40to where the puck is now you know and so
6:16:43that drives a lot of the stuff we do we
6:16:45have Criterium we use to sort of
6:16:48evaluate uh projects as they come in
6:16:50because we don't have capacity to really
6:16:52take on all the projects that they would
6:16:54like us to take on so some of the key
6:16:56criterias are um is this project going
6:17:00to sort of explore unique uses of this
6:17:03technology that we haven't tried or
6:17:05other people on campus haven't seen so
6:17:07that it will allow us to open
6:17:08conversations if we are working with
6:17:11Michelle at nursing but we see that this
6:17:14idea actually has applications to lots
6:17:16of other disciplines so we take on that
6:17:19project to say the a collab
6:17:21collaborative work environment like
6:17:23Michelle's project which we developed
6:17:25for VR heart is about a group of people
6:17:28looking at a specific uh a patient and
6:17:31then how can they communicate with each
6:17:32other there's lots of applications of
6:17:34that kind of tool and lots of
6:17:35disciplines so we when we take it on it
6:17:38says let's think through how we do this
6:17:40kind of uh collaborative VR environment
6:17:44and then what we learn actually will
6:17:46help us apply to other disciplines so
6:17:48those are the kind of key things which
6:17:50is if we develop this is not just the
6:17:52one off thing we're going to do and it's
6:17:53going to be done it's going to actually
6:17:55have uh potential to grow in other
6:17:57spaces well I was just going to um you
6:18:00know Echo what you just said you know
6:18:01when we're doing our clinic for instance
6:18:03when we pick the
6:18:05um the the proposals that we have that
6:18:08we're going to be supporting the College
6:18:09of Engineering is the same kind of thing
6:18:11we're like you know is is the project
6:18:13going to be benefited by being in XR or
6:18:17can it be just as easily done using a
6:18:19different technology in XR is really
6:18:20besides the point and then how portable
6:18:22is it to other contexts right because we
6:18:26really that is the most important thing
6:18:28especially in this phase now is that we
6:18:30want to make sure that we develop these
6:18:32tools in a way that are as applicable
6:18:36across the board as possible and not
6:18:38just going to create a a cool app but
6:18:42it's only relevant in in one specific
6:18:45and very limited context yeah a project
6:18:48that was funded by College of
6:18:50Engineering as part of the clinic was
6:18:52how do you integrate 36 video in a VR
6:18:55environment with interactivity so Joanna
6:18:58funded our effort to be able to say okay
6:19:01we have a toolkit that does a lot of
6:19:03cool great things that we can manipulate
6:19:05but we never really got a chance to
6:19:06implement a 360 interactive content so
6:19:09we developed that because we think that
6:19:11360 is going to be one of those lwh
6:19:14hanging fruit where people are going to
6:19:15be able to sort of like give me some I
6:19:17have some content I have some camera I
6:19:19captured I want to bring it into an XR
6:19:22VR environment how can I use it so
6:19:24building that platform would allow us to
6:19:27move uh to move it into a bigger use
6:19:30case U and the other piece of criteria
6:19:32and which is based on our Capac
6:19:35D say like we have some unique resources
6:19:37we have a motion capture rig we have a
6:19:40photogrametry rig and we have all these
6:19:42kind of elements that are not really
6:19:45available in lots of other spaces and so
6:19:47if a project comes to and says there's a
6:19:51a a movie class a class that's going to
6:19:55teach how to U make a movie like citizen
6:19:58King and they want to use a motion
6:20:01capture to capture an actor rep playing
6:20:03a scene and embed in a VR environment
6:20:06nobody else can really do that because
6:20:08we don't have a Vicon Vicon motion
6:20:11captures rig sitting in a lot of places
6:20:14so those kind of drive a lot of the
6:20:17projects we take on and ultimately is
6:20:20also capacity and do we have the
6:20:22expertise to actually execute on the
6:20:24plan so those were sort of the sort of
6:20:26high overview of things we think through
6:20:29as as we get uh contacted by faculty and
6:20:32researchers to say is this is this maybe
6:20:35a potential pathway for them to uh to
6:20:37pursue using XR okay so this is great
6:20:40and um it it also highlighted some of
6:20:43the synergies just between some of our
6:20:45teams and where we are and I think we we
6:20:48would even have more stories to share
6:20:50there um I think we we we nicely talked
6:20:53about scale in this segment we the
6:20:55observation that scale ultimately means
6:20:58potentially making low Tech choices at
6:21:01least at this stage making sure that we
6:21:03can reach more people
6:21:05uh we talked about approach and how to
6:21:07define what is an interesting project
6:21:10and it can sometimes be disappointing
6:21:12for somebody starting out in XR uh
6:21:14thinking that this is interesting isn't
6:21:16my app interesting and the choices that
6:21:18we as researchers or Dan as managers and
6:21:22Joanna as Deans make when it comes to
6:21:23funding projects I think these were Val
6:21:26insights uh that I wanted us to share
6:21:28with our Learners and so thank you for
6:21:30this first part of our discussion
6:21:51okay welcome back to our Learners um we
6:21:53are going to continue our discussion
6:21:55with invited experts uh from Michigan in
6:21:58this uh second part of our discussion I
6:22:01want to focus the discussion on around
6:22:04success how do we define success uh what
6:22:07are potentially metrics that we look for
6:22:09and then also I want to look into the
6:22:11future um together with you obviously so
6:22:15I'm going to take some turns here and uh
6:22:18and in the end we'll open it up
6:22:20completely and see how well that goes
6:22:21okay but I wanted to start uh with
6:22:24Michelle first which seems to be
6:22:26recurring always picking on Michelle so
6:22:29Michelle um for some of the specific
6:22:31projects that you have described earlier
6:22:33or maybe even others you're working on
6:22:35um I was wondering how do you go about
6:22:37defining success for these projects uh
6:22:41both in terms of research and maybe
6:22:43integration with teaching um are there
6:22:46any uh specific metrics is there
6:22:48anything you aim to show to demonstrate
6:22:52success that's a great question Michael
6:22:54just looking at this whole idea of
6:22:57success and and how you figure out
6:22:59whether or not your XR project was
6:23:02successful
6:23:04some of the ways that I try to approach
6:23:06this is obviously you know I think
6:23:08everybody knows each project is unique
6:23:11and what you're trying to achieve is
6:23:12going to look
6:23:14different and I think it just kind of
6:23:16goes back to really some of the basics
6:23:19so what makes sense
6:23:21for Learning and research whether
6:23:25it's you know case study or you know for
6:23:29me uh mannequin based simulation are
6:23:32going to look in some ways similar to XR
6:23:36so you know I think about you know
6:23:38things like learning outcomes what is it
6:23:39that I'm trying to achieve with this so
6:23:43when I looked at the heart VR one of the
6:23:46most important things that we were
6:23:48trying to accomplish was to give um our
6:23:51Learners an opportunity to practice
6:23:54communication skills and practice their
6:23:56teamwork skills those were the two
6:23:58things that were the most important and
6:24:01so we created the environment to to
6:24:04support that a sort of secondary
6:24:08objective was the opportunity to
6:24:11practice the Pediatric Advanced life
6:24:14support
6:24:15algorithms so they were able to go
6:24:17through and practice the algorithms we
6:24:20knew that we were not going to be
6:24:22dealing with any type of task-based
6:24:25learning so they weren't going to
6:24:27practice CPR skills they weren't going
6:24:29to practice you know any kind of
6:24:32Hands-On skills so that guided our
6:24:35development so then at the um end of
6:24:39that what we looked at is so how would
6:24:40we know whether or not it's successful
6:24:42now we haven't fully implemented it yet
6:24:45however things that we've built into
6:24:47that is first of all did they were they
6:24:51able to demonstrate effective
6:24:54communication skills if we ran them
6:24:56through this program several times did
6:24:59the communication skill improve um did
6:25:02we see back in their home units you know
6:25:04could we follow up with them and ask
6:25:06them you know did this help it's hard to
6:25:10measure transference into the clinical
6:25:12environment so we often don't try to do
6:25:16that um but we do try to ask them you
6:25:20know did this help do do you feel more
6:25:23confident in your skills and maybe
6:25:26sometimes look at knowledge things like
6:25:28that so what can we easily measure for
6:25:30us sometimes the research questions are
6:25:33really the same
6:25:34as the um learning questions what we're
6:25:37thinking about for education so another
6:25:40area that we can use XR very effectively
6:25:44is cavia came and talked a lot about
6:25:46this whole idea of perspective taking
6:25:49and so I have another project where we
6:25:51are going to help folks that are in
6:25:55healthcare provider roles understand
6:25:59what it's like when your de or heart of
6:26:02hearing go to a healthare visit and so
6:26:06we're going to use the idea of as Dan
6:26:08talked about 360 video and creating
6:26:13these 3 360 videos where you can sort of
6:26:16step in the shoes of somebody who's deaf
6:26:18or heart of hearing and the whole goal
6:26:21for that is just to learn what it's like
6:26:26to understand what it's like understand
6:26:28the
6:26:29perspective so if they understand it
6:26:33they can then
6:26:35work towards creating a better
6:26:37experience so our goal is not to teach
6:26:41them what to do better but to just give
6:26:44them that experience and improve and
6:26:47increase their understanding so those
6:26:49are really veryy two different goals and
6:26:52I think with everything you really look
6:26:54at what am I trying to achieve and then
6:26:57I always say how do you match the
6:26:59technology with what you're trying to
6:27:02achieve and for our purposes we thought
6:27:04360 video could do that for us and we
6:27:09didn't need to try to create a much more
6:27:13expensive virtual reality animated
6:27:16environment we could do that with 360
6:27:18video so we'll see how it works out so
6:27:22that was great uh I think it showed uh
6:27:25obviously that we are all also still
6:27:27learning and um even if a project may
6:27:31not be successful in some traditional
6:27:33sense uh
6:27:34for example I had some other questions
6:27:36in mind as I was listening to you like
6:27:39did did a project ever fail in the sense
6:27:41that you you matched it with the wrong
6:27:42technology but I'm gonna ask Joanna um
6:27:46because Joanna I know in your role um um
6:27:49talking again about the avmr clinic but
6:27:51you can take this question more broadly
6:27:54when you um accept a project for funding
6:27:57are there a specific criteria you apply
6:28:00in terms of how we should be thinking
6:28:02about success in each of the these
6:28:06projects yeah so the way to determine
6:28:08whether or not a project is successful I
6:28:10mean there's there's two ways that I
6:28:12kind of think about it you know so one
6:28:13way is you know specific to that project
6:28:16you know and one of the criteria is that
6:28:19every project does have some kind of a
6:28:22mechanism or some kind of a way to
6:28:24determine whether or not a project is
6:28:26successful if it's actually achieving
6:28:28the goals that it it is planning on
6:28:30achieving and the second thing the
6:28:32second metric of success I would say is
6:28:35whether or not that technology is then
6:28:38ported to another context so that to me
6:28:41is the ultimate um success right so
6:28:46maybe you know the 360 um situation that
6:28:50we just heard about from Michelle you
6:28:52know so that's a scenario that that
6:28:54she's applying in a particular context
6:28:57but then if we were able to see that
6:28:58that context that that application or
6:29:01what was learned from that particular
6:29:03thing was then applied to another course
6:29:06say in like a design course around um
6:29:10you know consumer Centric or socially
6:29:13enabled design for instance that to me
6:29:16would be an enormous success now we
6:29:18don't have successes like that quite yet
6:29:21because you know we're still it's still
6:29:22very early days uh in the projects and
6:29:25we're still trying to determine whether
6:29:27or not the individual projects
6:29:29themselves are successful with regard to
6:29:30their own uh criteria
6:29:34so I don't know if if this is
6:29:36appropriate or not but I kind of wanted
6:29:37to to show um talk a little bit about my
6:29:40personal resour research where there was
6:29:43there was a failure if you will that
6:29:46just opens up more questions so the
6:29:48research that we're U trying to do is to
6:29:50try to help students understand Crystal
6:29:52structures which is an important Concept
6:29:53in Material Science and a lot of that uh
6:29:57go that concept requires spatial
6:30:00reasoning and so what we were thinking
6:30:02about is perhaps we can use virtual
6:30:04reality and just mixed reality kinds of
6:30:07um representations of of Crystal
6:30:09structures to help students develop like
6:30:11an understanding of what these crystals
6:30:13really look like in space and the idea
6:30:15was is that they would do better and and
6:30:17get better spatial reasoning out of this
6:30:20where spatial reasoning of course is the
6:30:22ability to take a three-dimensional
6:30:23object and turn it in your mind rather
6:30:26than doing it in your mind it would be
6:30:27so much better if you could just do it
6:30:29with the object in virtual reality or
6:30:32mixed reality and so we did these these
6:30:34experiments and we did them over a
6:30:36course of two years and we saw
6:30:39absolutely that students who did a
6:30:42particular task using virtual reality to
6:30:45manipulate these objects on that task
6:30:48did better than those that didn't but
6:30:51here's the you know so that was like a
6:30:53huge win we're like who yay fun this is
6:30:55exciting but then uh we we did a pre and
6:31:00post to see whether or not those gains
6:31:02were retained and we actually saw that
6:31:04those gains were not retained and in
6:31:06fact the students without who didn't use
6:31:09a virtual reality um did better
6:31:12afterwards than the ones who had virtual
6:31:15reality and so that now raises so many
6:31:18fascinating questions like you know is
6:31:20there something about doing this mental
6:31:22manipulation that actually cements the
6:31:25learning or not and so how can we now
6:31:27change the experiment how can we
6:31:29actually change the learning activity so
6:31:32that those gains aren't lost maybe
6:31:34there's something else that we're
6:31:35missing so it's this fascinating you
6:31:37know again it's going back to you know
6:31:40there are things that we just don't know
6:31:42like that is a
6:31:43counterintuitive thing that we just
6:31:45found um so how do we actually apply
6:31:48that so that that's a failure in an
6:31:51indiv in an individual case but I think
6:31:54we'll have um ramifications in the long
6:31:56run that will be much more positive this
6:31:59is great both I mean I was feeding with
6:32:01you as a researcher for a second because
6:32:02oh damn negative results but I mean on
6:32:05the other hand I think it applies I
6:32:07always say that to my students in my own
6:32:09courses as well at least we've learned
6:32:11something and I think that applies in in
6:32:13this entire space and it's a good
6:32:15attitude uh to make progress towards
6:32:17learning about XR I think so Jeremy and
6:32:21Dan um you can figure it out between the
6:32:24two of you who wants to take on this one
6:32:26I want us to think even a little bit
6:32:29more forward um so Dan probably has some
6:32:31new projects coming in I know that
6:32:33Jeremy as part of the XR initiative you
6:32:35have just funded a few um I know that uh
6:32:39one thing so I want to talk about next
6:32:42steps um we talked a lot about research
6:32:44we did mention uh a lot of our efforts
6:32:47with students so one thing that I think
6:32:50um is a little problematic at the moment
6:32:52coming back to scale is we do have a
6:32:55student oriented XR uh spaces so for
6:32:59example in durad Center um I can also as
6:33:01an instructor I can go there sometimes
6:33:04and and teach so I feel like with a lot
6:33:06of these things that we're doing and the
6:33:09facilities we have we're kind of like at
6:33:11maximum capacity and I I feel like
6:33:13that's also what I hear when I talk to
6:33:15you so um how can we improve on that how
6:33:20can we extend our capabilities to do
6:33:24more and better XR in the future how do
6:33:26you think about this so I I think we've
6:33:29talked about this maybe I'll I'll
6:33:30briefly capture sort of uh the rational
6:33:34of how I I thought about the
6:33:36visualization Studio at the you know
6:33:38when we developed the visualization
6:33:40studio uh open it in Fall 2018 we
6:33:44literally looked at the environment and
6:33:46said there's a lot of people who are
6:33:48being locked out of using and accessing
6:33:50these
6:33:52tools even not being able to go you know
6:33:54you can go buy some faculty member may
6:33:56buy a headset but there was not a
6:33:58holistically developed environment that
6:34:00can allow a group of 20 30 people to
6:34:02join and collaborate and work and
6:34:04develop environment so we basically put
6:34:07a marker on the floor and on the ground
6:34:08and said we need something to help us
6:34:11think through the process my hidden but
6:34:15long-term Mo uh plan uh strategy was
6:34:18that I want to build something that can
6:34:20demonstrate to the campus what we can do
6:34:24and the intention was that this is one
6:34:26way you can build X that is designed to
6:34:29support as many disciplines as possible
6:34:31so it's not micr focused on specific
6:34:34discipline and what I wanted to do and
6:34:37we're having some really great
6:34:39conversations with various schools who
6:34:42are coming in and saying your formula
6:34:44looks like this I'm going to build a lab
6:34:45in my building that's going to be
6:34:47applying specifically to architecture
6:34:50students because I need other things
6:34:52around this environment and we're
6:34:54talking with folks in lsna saying you
6:34:56guys have done X but we're going to need
6:34:58Y and then we're going to use your
6:35:00formula but we're going to evolve it
6:35:02differently and so The Hope was that we
6:35:04will build something not necessarily to
6:35:08fully um uh satisfy the demand that we
6:35:12saw coming on campus it was a way for us
6:35:14to say this is one way of doing it we
6:35:18want to keep involving it because we
6:35:19started with a SW set of Hardware but
6:35:23within a year we
6:35:27flipped Rift SES and all that kind of
6:35:30stuff we Chang this model of how we try
6:35:32to operate the center which is change
6:35:34quickly and identify the things that
6:35:36work and others can learn from it and
6:35:38evolve it so that the scale is going to
6:35:40pop out at at different areas on campus
6:35:43it's going to pop out at different
6:35:44schools and different un departments are
6:35:46going to evolve the space to fit their
6:35:49specific needs and that you know the cop
6:35:51and other tools are going to help us
6:35:53collaborating into those environments
6:35:55so right we are a uh over subscribed uh
6:35:59I'm trying to solve the short-term
6:36:01problem so we are adding six more
6:36:03Workstation in the was supposed to be
6:36:06winter semester but it's going to happen
6:36:08in the fall we're going to add six more
6:36:10workstations add more capacity so that
6:36:12at least the size of a class that you
6:36:14have Michael can come and be able to use
6:36:16it even if they have you know a couple
6:36:19of work stations to double up but that's
6:36:21never going to feel like it's going to
6:36:23satisfy all the things we need so the
6:36:25the strategy the long-term plan is we
6:36:28have the space used on the third can we
6:36:31you know I have ideas for how we can
6:36:33expand a capacity for a room that can
6:36:35hold 50 people that has XR enabled space
6:36:38but still that's not going to be enough
6:36:40but the other pieces are the other
6:36:43pieces of the Machinery are moving where
6:36:45other units are thinking and mapping and
6:36:47adding to their budget for what they
6:36:48want to do in the next cycle so that
6:36:51they could have a space that's capable
6:36:52of supporting faculty and students in
6:36:54their in their individual units um so
6:36:58that that that's those were the two sort
6:37:00of Pathways that we're trying to follow
6:37:02okay this is this is great and this is
6:37:04uh this already actually helps a lot of
6:37:06people across the university because a
6:37:08lot of people including myself make use
6:37:11of that those spaces so Jeremy killer
6:37:14question uh how do we what are the next
6:37:17steps to even increase the scale at the
6:37:20larger University level and I know you
6:37:22think a lot about teaching and
6:37:24presential learning and online learning
6:37:26and how do you bring these Technologies
6:37:28to like a mass so I'm wondering what
6:37:31what do you think are your next steps as
6:37:33part of the XR initiative sure yeah no I
6:37:36think I think the blueprint that Dan put
6:37:38down with the visualization studio is
6:37:39something you know that we can build on
6:37:41top of and in other parts of the
6:37:43university to allow even more folks
6:37:46outside of the the North Campus
6:37:48Community to participate in experience
6:37:51there's there's one smaller lab on
6:37:52central campus now that six or eight
6:37:54students can participate in that's just
6:37:56not to the same scale so I think there
6:37:59blueprint there of how to build out of
6:38:00space and how to replicate that um so we
6:38:03we're talking with other units around
6:38:05campus on how to do that I know the
6:38:07school of information is looking at
6:38:08building a new building and they want to
6:38:10include space in there our College of
6:38:12architecture and urban planning have
6:38:13already been talking about building out
6:38:15an XR lab so I think there's kind of
6:38:18taking that model and building out you
6:38:19know large spaces where folks can come
6:38:21together and as well as smaller more
6:38:22iterative spaces also I think with some
6:38:25of these new devices like the Oculus
6:38:28Quest and more of the portable um you
6:38:32know PCO devices that you can have
6:38:34anywhere really anywhere where there's a
6:38:36wireless connection and enough room I
6:38:38think looking at ways to to make that
6:38:39available I know Michelle as well as
6:38:42another faculty industry have required
6:38:46dozens of and hundreds of oculus go
6:38:49headsets and are able to deploy those so
6:38:50I think that's a method as well but we
6:38:52have to be thinking about scale in terms
6:38:54of maintaining these devices the
6:38:56security the patching the update content
6:38:59distribution so those are all similar
6:39:01types of challenges that have been faced
6:39:02with it terms of desktops and laptops
6:39:05you know as well as you know in the AR
6:39:07space mobile phones almost many folks
6:39:10many Learners have a smartphone that has
6:39:12AR capability in it and so how can we
6:39:15leverage what's available or at least
6:39:17you know that minimum set to to scale
6:39:20this out and create experiences that can
6:39:23be consumed by more folks so I think as
6:39:25we as we generate more content as we
6:39:27create more content uh it'll help us in
6:39:30in making larger purchases of equipment
6:39:33spaces that hopefully aren't out of date
6:39:36before you know you get to use them for
6:39:38too many semesters I I can add one more
6:39:40thing Michael is uh something we've been
6:39:43talking about internally within cane is
6:39:46that and this is stuff we we mapped out
6:39:48early but we haven't executor on which
6:39:50is um just in the DOR St Center we have
6:39:53250 workstations that are you know
6:39:57really good machines that have the
6:39:59software but there's hundreds of
6:40:00machines that are distributed throughout
6:40:02campus and so making specialized XR gear
6:40:06available for checkout like somebody can
6:40:09come and say grab a piece of Hardware
6:40:11that we maintain and support and then
6:40:14all of a sudden you have lots of places
6:40:16that can become a group work environment
6:40:18for people being able to explore XR
6:40:21technology and so that may be you may
6:40:23not need to equip every machine with the
6:40:26hardware but if we have 20 headsets that
6:40:28people can check out when they need them
6:40:30which we're uh we have some limited of
6:40:32those tools available but we would like
6:40:34to expand it so when you ask what the
6:40:37pain point is that if we can expand that
6:40:39capacity both on Central on North Campus
6:40:41and students can go out and check you
6:40:43know a Pelican case with a a Vive or
6:40:46Rift s or you know other gear and if we
6:40:51build the platform so that the software
6:40:53is built into of our load sets that are
6:40:56in the labs and all they need to do is
6:40:57plug and go um especially with the
6:41:00hardware that's inside out tracking so
6:41:02you're not worrying about sensors having
6:41:04to mount on the desktop and stuff like
6:41:06that then that feels like it's it's a
6:41:09step that will allow us to engage and
6:41:12lots of students actually are coming to
6:41:14campus with really powerful laptops so
6:41:16that they can actually take it to their
6:41:17dorm and be able to use use these tools
6:41:20so that's another of uh strategy uh that
6:41:25I would like to pursue we're having some
6:41:27internal conversations but we'll be
6:41:29happy to engage for more wider approach
6:41:32to C campus yeah so one of the things
6:41:35that I also wanted to make sure that we
6:41:36pointed out is you know XR is coming
6:41:39right you know the the technology is
6:41:41getting better all the time you know the
6:41:42devices are getting less and less
6:41:44expensive you know the our ability to
6:41:47you know detect motion you know it it
6:41:50just it just changes by leaps and bound
6:41:53every single year and so really you know
6:41:56the the message I think that that we
6:41:58really need to say is that we need to do
6:42:00Investments now in Hardware not because
6:42:03you know not because of the hardware so
6:42:05much is because we need to develop the
6:42:07content we need to understand what XR is
6:42:10good for so that by the time XR is in
6:42:15everyone's hands which it will be um
6:42:18we'll have a very good sense of how to
6:42:20deploy XR in the best possible way in
6:42:22the learning context in the training
6:42:25context you know culturally um think
6:42:27about security issues and so forth so
6:42:29the fact that we're doing all this
6:42:31investment now you know some people
6:42:33might be well you know as soon as you
6:42:34build it it's obsolete yes that's true I
6:42:37suppose but but that's not the point the
6:42:40point is developing the normative
6:42:42context the point is understanding under
6:42:46what circumstances XR is best used for
6:42:50the point is you know trying to figure
6:42:51out all of these other issues around
6:42:54security and accessibility before it's
6:42:58in everyone's hands that's the point
6:43:01that's why we do the investment now
6:43:21okay so this was really great to hear
6:43:22about current efforts and then also
6:43:25steps in each of these efforts uh at
6:43:27Michigan and now I want us uh I want to
6:43:30think together with you about the future
6:43:33uh future let's say 5 to 10 years where
6:43:36do you think we will be as a society and
6:43:40then also obviously um us at Michigan
6:43:43and our roles as faculty Deans and
6:43:47managers what's that future going to
6:43:49look like is is X I think we all think
6:43:52XR is still going to be there maybe I
6:43:53should check in with you first but then
6:43:56if you could each make a wish where we
6:43:59would be in terms of XR adoption
6:44:01Technologies and just like to go around
6:44:04and hear a little about your
6:44:06thoughts I guess I'll start um when I
6:44:09think
6:44:10about XR and where I believe that will
6:44:15be in the future and where I'd like to
6:44:17see us in the future I have a few uh a
6:44:20few ideas um first of all um I want to
6:44:24speak a little bit about health care and
6:44:29looking around and interacting with some
6:44:33my
6:44:33colleagues in dealing with the pandemic
6:44:38the covid pandemic that we're having
6:44:40right now and trying
6:44:44to upskill
6:44:46nurses so that they can move from a
6:44:50general care to an ICU
6:44:53environment would
6:44:55be so much easier if we had some Virtual
6:44:59Technology available to do that
6:45:03so although this pandemic has been
6:45:06extremely challenging I think it is also
6:45:09for those of us in the simulation
6:45:11Community we've been communicating um on
6:45:15a regular basis sharing best practices
6:45:18one of the things that we are coming to
6:45:21understand is if we had more portable
6:45:26Technologies like virtual reality
6:45:29augmented reality and even some
6:45:32app-based stuff
6:45:33we would be in a much better position to
6:45:36provide
6:45:37training in what we are dealing with
6:45:40right now that also fits in with much of
6:45:45what myself and some of my colleagues
6:45:48are really excited about being able to
6:45:50use Virtual Reality 4 and I think of
6:45:54some of the work that we're doing with
6:45:55the heart VR and Dr Mahajan and I have
6:45:58talked about this idea of having sort of
6:46:01a toolkit for just in time training for
6:46:05um not just our students because it will
6:46:07be great for our students to be able to
6:46:09do that as they're practicing at home or
6:46:12off campus to get ready for maybe some
6:46:16competency evaluations but also for our
6:46:19practitioners when you're in the um
6:46:23environment and it doesn't matter
6:46:25whether it's the acute care environment
6:46:27whether you're paramedic in an ambulance
6:46:29whether you're working in a remote
6:46:30hospital or critical access Hospital
6:46:33if you're coming up to do a procedure or
6:46:37to do something that you haven't done in
6:46:38a while and you want to give you want to
6:46:43take the opportunity to quickly practice
6:46:46that the Sometimes the best way to be
6:46:49able to do that is to be able to step
6:46:50into a virtual environment and just do a
6:46:54dry run and you you can't first of all
6:46:58you can't do that you can't go run down
6:46:59to the Sim lab and do it and you can
6:47:02watch a video or read something it's
6:47:03just not the same but if we had these
6:47:06virtual assets deployed throughout the
6:47:08hospital then people could access this
6:47:11in a much more just in time way to
6:47:15practice a skill and do that my whole
6:47:19career has really focused on how can we
6:47:23provide better safer care to our
6:47:25patients and that's why I got into
6:47:27simulation but I see XR as really being
6:47:33One Step better one step more advanced
6:47:38as well as one piece to our whole
6:47:42simulation bucket or portfolio to really
6:47:45help our practitioners both as learning
6:47:49students and on the front lines to be
6:47:52able to
6:47:54deliver more effective and safer patient
6:47:57care so that's really my wish my desire
6:48:02and what I see in the future because I
6:48:06would love to see an entire classroom of
6:48:10students have their laptop and an Oculus
6:48:14go or some other device right there so
6:48:17as I'm talking to them about Anatomy I
6:48:19can say okay go take a look and they can
6:48:21put it on and they can see exactly what
6:48:24we're talking about they can bring it to
6:48:26life if I say let's do something on
6:48:29heart failure I can say okay let's jump
6:48:31in and do a virtual case study on that I
6:48:33just think that this is where the future
6:48:36is going to be and this is one of the
6:48:39ways we can provide some excellent
6:48:41learning for our students so that's my
6:48:43wish for the future and I don't think
6:48:46it's a uh wish that is crazy I think
6:48:50it's gonna happen yeah everything that
6:48:54Michelle said that it I mean she's
6:48:56Absolut I agree with her 100% you know
6:49:00it's it's kind of funny to think about
6:49:02you know 10 years in the future if you
6:49:03take a look at you know you take our
6:49:05cell phone you know here's our cell
6:49:07phone that's only been around for you
6:49:09know 10ish years and now can you even
6:49:11imagine living your life without it can
6:49:13you imagine going to you know a city
6:49:16you've never been to without being able
6:49:18to have Google Maps or to check yel to
6:49:21see you know where you should eat dinner
6:49:23or you know how do you get to this new
6:49:25place I I just can't even imagine what
6:49:27our lives would be like without our
6:49:29smartphones I would like to predict that
6:49:31in 10 years virtual reality XR is going
6:49:34to be very similar um it might take a
6:49:37little bit longer than 10 years because
6:49:38we don't quite have a commercial Pro
6:49:40product I don't think that is quite as
6:49:43um you know consumer hard as the the
6:49:46first smartphone was but I think in
6:49:49addition to that you know what it's
6:49:51really going to do it's going to create
6:49:53an entirely new way of thinking about
6:49:56things in ways that we we haven't really
6:49:59had a chance to think about before it's
6:50:00going to create a new a a new
6:50:03uh branch of epistemology if you will if
6:50:06you think about how knowledge is
6:50:08currently um organized Even in our
6:50:11computer screens we have file structures
6:50:14they're really one-dimensional only
6:50:17onedimensional they're lists if you
6:50:19think about a book A book is
6:50:21three-dimensional only because you can
6:50:23turn the page but the page itself is
6:50:25two-dimensional if you think about
6:50:27drawings and sketches and the way that
6:50:30we convey knowledge to one another
6:50:33it's all two-dimensional which is kind
6:50:35of artificial if you think about it
6:50:37because we're three-dimensional
6:50:39creatures right we exist in
6:50:41three-dimensional space imagine now that
6:50:45we can conceive of things in
6:50:48three-dimensions from the get-go where
6:50:51we don't have to take this
6:50:53three-dimensional reality that we're
6:50:55living in and project it onto
6:50:58two-dimensional space we can
6:51:01actually think in three-dimensional
6:51:03space and and sketch in threedimensional
6:51:05space and you know present information
6:51:08organize information in
6:51:10multi-dimensional space even I mean I I
6:51:12really do think that this is that XR is
6:51:17just the very beginning of a completely
6:51:20new way of us thinking about things
6:51:23Michael may have heard me talk about you
6:51:25know calculus and what calculus has done
6:51:27for us so calculus really one way to
6:51:31think about calc calculus is that
6:51:33calculus gave us a way to think about
6:51:35multi-dimensional space in a very
6:51:37rigorous way you take a
6:51:40derivative um you know a rate that is
6:51:44just really how much how much does you
6:51:47know distance change with time two
6:51:49different dimensions an integral is
6:51:52really the area under the curve two
6:51:54different dimensions and what we've done
6:51:56is we've taken multi-dimensional space
6:51:58and we've turned that into a single
6:52:00dimension of code an equation is just a
6:52:03way of representing multi-dimensional
6:52:05space in one dimension say we don't have
6:52:07to do that say we don't have to take
6:52:10that cognitive step with um converting
6:52:14the multi-dimensional into a single
6:52:16Dimension and we can just do this
6:52:19directly what that even has to do with
6:52:21accessibility now because one of the
6:52:24biggest things I think that prevents
6:52:26people from going into stem Fields is
6:52:30this abstract thinking we have to be
6:52:33very abstract in the way we talk about
6:52:35the physical world XR if we can
6:52:38manipulate physical space directly I
6:52:41think that means that we can open up
6:52:43stem fields and other different kinds of
6:52:46fields to to people that have never been
6:52:49there before so I'm very excited about
6:52:52the prospect of XR not only from from
6:52:54the kinds of things that Michelle was
6:52:56talking about but also you know all of
6:52:58these other uh kinds of things and and
6:53:01different ways of thinking about our
6:53:03physical world so I'm so excited I agree
6:53:07with both Michelle and Joanna
6:53:09wholeheartedly and it's it's the the
6:53:13concept of 20 2007 when iPhone came out
6:53:16and you know things
6:53:18changed drastically I just see AR
6:53:22especially but XR in general over five
6:53:26to eight years sort of changing how we
6:53:28do a lot of stuff and this is related to
6:53:31the fact that um AI the the other AI
6:53:35artificial intelligence is is uh really
6:53:38accelerating machine learning and in the
6:53:40fact that Vision systems are becoming so
6:53:43sophisticated that uh these tools are
6:53:46going to be ubiquitous uh you know I
6:53:49just ordered uh uh to experiment with
6:53:52the new iPad that was released that has
6:53:54an a liar built into it which means next
6:53:57iPhones are going to have Liars built
6:53:59into them that will actually be sort of
6:54:02uh studying the space and the software
6:54:05is going to be a lot more sophisticated
6:54:07so these tools are going to change How
6:54:09We Do tele medicine we've tried to work
6:54:11on projects with Dr Mahajan and how do
6:54:13you help uh a surgeon at a remote site
6:54:17that you can virtually sort of guide
6:54:19them what they need to do that stuff is
6:54:21going to change uh ubiquitous access to
6:54:23high-speed networks uh machine learning
6:54:27and you know sort of massive uh
6:54:29processing uh gpus sitting in the cloud
6:54:31is just going to
6:54:33really revolutionize the things we do
6:54:35both on campus off-campus life and so
6:54:38the things that I see um for us to pay
6:54:41attention to is that the questions about
6:54:44privacy then Auto automatically is on a
6:54:47on on steroids we have to really pay
6:54:49attention to it because the things that
6:54:51we were check in a box without reading
6:54:53giving it away all of a sudden we have
6:54:55this devices that have uh dozens of
6:54:59sensors that are capturing a ton of
6:55:01information that is getting shared and
6:55:04so those are the things that internally
6:55:06as a University of Michigan other
6:55:08learning institutions have to pay
6:55:09attention and how do we train the future
6:55:12work Workforce to pay attention to this
6:55:14stuff um but I see V are being sort of
6:55:17this amazing empathy machine that could
6:55:20actually put you in somebody's shoes to
6:55:23understand what it is to be in that
6:55:25position uh anecdotally we just we we
6:55:28did a a fast project on Uncle Tom's
6:55:31Cabin where uh Eliza is
6:55:34escaping uh uh as the slaveholders are
6:55:38chasing and the person the students
6:55:40could then inhabit the Eliza who's being
6:55:42chased or the person who's chasing her
6:55:45and then the reactions we saw from the
6:55:46students was just like you can read that
6:55:48book but now all of a sudden you're in
6:55:50that role where you're running away dogs
6:55:52are barking after you and all that kind
6:55:54of stuff and so this these kind of
6:55:56things are going to be as the quality of
6:55:57the content the processing and all that
6:56:00stuff improves it's going to be sort of
6:56:02second nature for us to be able to
6:56:04acquire information with these tools but
6:56:07that's the the caveat is that we really
6:56:09have to pay attention what's happening
6:56:11with the data and how uh you know when
6:56:15the you know when uh when the
6:56:16information gets shared
6:56:18broadly so those are my my two sents
6:56:21okay that's great Michelle J Dan those
6:56:24those great so Jeremy um I'm going to
6:56:27ask you the same thing you can make a
6:56:28wish or you can make a prediction where
6:56:31do you think XR is going to be in 5 to
6:56:3410 years and all use of
6:56:35XR yeah I I think the acceleration of
6:56:39the hardware the ubiquity of the
6:56:41Computing I think we will have glasses
6:56:44that are able to transfer the world and
6:56:46and bring us uh together I think that
6:56:49the concept of presence in terms of
6:56:51imagine if we were all sitting together
6:56:53virtually in the same space or augmented
6:56:55in the same space and we could have this
6:56:57conversation differently than just
6:56:59looking at a screen I agree with what
6:57:02Joanna said about we've for thousands of
6:57:04years been developing for a rectangle
6:57:06basically whether it be a wall or a
6:57:09painting or a phone or a television or a
6:57:11computer monitor I think the the
6:57:13opportunities um that the technology
6:57:15holds both virtually and augmented I
6:57:18think we'll we'll see a little bit more
6:57:19sorting where a augmented reality will
6:57:22probably be more pervasive throughout
6:57:24the world and virtual reality will have
6:57:25more very specific use cases I think the
6:57:28the simulation the training the base the
6:57:31the soft those sorts of um trainings
6:57:34will be I think
6:57:36more refined and and honed where that
6:57:38use case is I think our role as a
6:57:40university as an as an institution will
6:57:43be to help figure out where best to sort
6:57:46and how to teach to that space and how
6:57:48students can learn in that space and
6:57:50become co-creators I think much like
6:57:52we've seen with social media and other
6:57:54spaces I think students and the world
6:57:56will become co-creators and the tools
6:57:58will become more available for people to
6:58:01build and enhance on top of that it's a
6:58:03very exciting time now to help shape
6:58:05that future and I'm excited to be part
6:58:08of it so this is exciting to actually
6:58:11listen to you and uh hear your actually
6:58:14not so different perspectives I think we
6:58:16all enthusiastic uh there was a little
6:58:18bit of a warning we need to get this
6:58:20right there are lots of new issues that
6:58:22we're raising to me it's a little bit
6:58:24ironic that um when I teach this uh this
6:58:28content and um Joanna and also Jeremy
6:58:31were making this point about how we live
6:58:33in 3D space and how we understand it so
6:58:35well and how we have been designing for
6:58:372D rect angles for a while I would say
6:58:40that now we're starting to finally get
6:58:41good at designing for 2D we need to
6:58:43think about 3D and we know nothing uh in
6:58:46some sense we really have to learn from
6:58:48scratch our our Knowledge and Skills of
6:58:49living in 3D space every day don't
6:58:51necessarily translate well to design we
6:58:53make lots of mistakes there I always say
6:58:55we need new guidelines and better
6:58:56guidelines in any case um but um that's
6:59:00probably also what we're exploring
6:59:02together we each learning in our own
6:59:04fields and it was nice to hear your
6:59:07wishes and your predictions uh about the
6:59:09future of XR which in general is
6:59:12positive we agree it's not going to go
6:59:14away we actually agree that it's
6:59:16probably going to be very pervasive and
6:59:19affordable
6:59:21accessible um even to people with
6:59:23special needs may even enhance some of
6:59:26our abilities in many ways so there's
6:59:29potentially a bright future so thank you
6:59:32for um joining uh us today and sharing
6:59:35your uh knowledge and expertise and
6:59:37insights with our online Learners I
6:59:39thought this was a great discussion and
6:59:41we'll hopefully have some interesting
6:59:43follow-up comments and discussions in in
6:59:46in forums that come with this uh would
6:59:48like to hear uh issues about issues and
6:59:52concerns that our Learners maybe have or
6:59:54also dreams and wishes you want to make
6:59:56for the future um and so we can discuss
6:59:59those and reflect on those together as a
7:00:02community of Learners and earlier
7:00:05adopters and practitioners so thanks
7:00:08again for your time and for joining in
7:00:23today hey everybody so in this segment I
7:00:26would like to talk about initiatives and
7:00:29specifically about one student
7:00:32initiative here on campus at the
7:00:33University of Michigan the alternate
7:00:34reality initiative and I've invited a
7:00:37few students who I've met a few years
7:00:38ago and have been very active in this
7:00:40space we going to talk about challenges
7:00:42they have had and also how it's going
7:00:45overall they've been actually quite
7:00:47successful so why don't we introduce um
7:00:51well ourselves to our viewers um so
7:00:54let's get started with you guys say who
7:00:57you are and maybe your role in this
7:01:00alternate reality initiative and and
7:01:02then we can kick it off really okay I
7:01:05guess I'll start off uh everyone my name
7:01:07is Matthew Cova I'm the current
7:01:10co-president of the alternate ra
7:01:11initiative and I'm one of the
7:01:13co-founders uh hello I'll go next uh my
7:01:16name is besta Iden and I'm the current
7:01:18co-president of Ari hey everyone my name
7:01:21is Michael Zang I'm the co-founder and
7:01:24former president of Ari it's cool so I'm
7:01:26talking to all the presidents here so
7:01:28that is great uh all right so well
7:01:32it is a student initiative one thing
7:01:34that I was particularly impressed um
7:01:36when I attended your very first meeting
7:01:39the kick off several years ago was uh
7:01:42the professionalism that you put into
7:01:44this initiative introducing Michael was
7:01:46the president at the time and you had a
7:01:47few other folks around so tell me
7:01:50organizationally how how was it for you
7:01:53how is it going how many other
7:01:56presidents are there or how many other
7:01:57members do you have on your teams what
7:01:59kinds of roles how are you organized
7:02:02internality let's get started this way
7:02:04so when we started we were kind of like
7:02:06a rag tag bunch of different students
7:02:08where we posted a Facebook announcement
7:02:11basically trying to find other students
7:02:12interested in the VR and AR space and so
7:02:16that's how besta Matthew and I all came
7:02:18together and from there we then started
7:02:20forming like as a formal student
7:02:22organization and having these different
7:02:24roles and our leadership structure and
7:02:25the people that have been a part of our
7:02:27team has changed over the last few years
7:02:29as well yes and then right now for
7:02:33different positions that we have uh of
7:02:35course we've always had your president
7:02:36vice president or right now two
7:02:38co-presidents then aside from that we've
7:02:41had uh someone usually be in charge of
7:02:43the workshops that we do during our
7:02:44meetings and right now with project
7:02:47teams we've had a project teams uh we
7:02:49also do different sorts of community
7:02:51events through our meetings and uh
7:02:53different types of Outreach so we have
7:02:56uh a bunch of different leadership
7:02:57members for those roles I want say right
7:03:01now we have eight different leadership
7:03:03members on the team so we're able to
7:03:06divy up the work amongst
7:03:09everyone so that includes like marketing
7:03:12I guess about your various initiatives
7:03:15and a lot of communication I am a you
7:03:18know proud receiver of your newsletter
7:03:20that comes in weekly so um what other
7:03:22things are going on uh beste what what
7:03:25are you responsible for right now uh so
7:03:27right now uh as a co-president I focus
7:03:30mainly on the worksh and development
7:03:32side so I help organize the workshops uh
7:03:35make sure we have new and different
7:03:38content uh each of our Workshop meetings
7:03:41um and we also balance those out with
7:03:43some community events so we have movie
7:03:45nights game nights just to build the
7:03:47community of our
7:03:49team so basically anybody uh I guess
7:03:52students primarily with interest in XR
7:03:55or like how were you recruiting what
7:03:57were the people that you wanted to be
7:03:59part of your community
7:04:01yeah so for the most part uh we'll take
7:04:04anyone uh we are open to any students on
7:04:07campus undergraduate or graduate
7:04:09students to come and join for the most
7:04:11part our students do tend to be more
7:04:14Technology based more stem based a lot
7:04:16of our students come from CS computer
7:04:18science um or other Majors like that but
7:04:22then for how we actually get the
7:04:24students we recruit students at
7:04:26different oncampus events so for example
7:04:29at University of Michigan there's uh
7:04:30Northwest there festiv fall uh rackam
7:04:34has their own event for graduate
7:04:35students and through those events we're
7:04:37able to get a large Outreach so usually
7:04:40between Northwest and fesal we're able
7:04:42to get around 250 to 300 undergraduate
7:04:45students to sign up for our newsletter
7:04:48and then through rackam we're able to
7:04:50get around 50 or so graduate students to
7:04:52also sign up for that and that's really
7:04:54just how can we build this community
7:04:56whether or not they actually come to our
7:04:57meetings we're able to at least get our
7:05:00information out to them whether that's
7:05:02just information about our weekly
7:05:03meetings about any job opportunities
7:05:07that we see any events that we're
7:05:10holding we also don't really
7:05:12differentiate uh experience level we
7:05:15have some people who are more
7:05:16experienced that come to our meetings
7:05:18and people who are just starting out and
7:05:20we make sure to make our Workshop
7:05:23content uh applicable to both of these
7:05:25groups because we want the new people to
7:05:28be able to learn and we want the
7:05:30experienced people to have some of a
7:05:32challenge and to add on to that we
7:05:35essentially uh created Ari with the idea
7:05:37of the being four different types of
7:05:39areas that students will be going
7:05:41through so starting off with exploring
7:05:43XR technology before they learn through
7:05:45our workshops they build different
7:05:47projects and then connect with other
7:05:49students or other people in the EXO
7:05:51industry so that way we can appeal to a
7:05:53large variety of students in different
7:05:55parts of their XR
7:05:57Journey well that's great there's a lot
7:05:59of useful tips out there for people who
7:06:01want to start off initiatives so some of
7:06:03the ingredients that I I hear include
7:06:06workshops so and I think Ari always
7:06:08strike me as this I was always struck me
7:06:09as this initiative where people come
7:06:11together to learn about XR so can you
7:06:14give some examples of workshops you have
7:06:16done so some of our workshops uh we have
7:06:19some very technical workshops so we have
7:06:22for example a um Oculus RI Workshop
7:06:26through Unity where we teach basic uh
7:06:29programming and controls for the OCC
7:06:31Swift we have some augmented reality
7:06:34workshops such as creating Snapchat
7:06:36filters uh which is a more of a fun
7:06:38workshop and then we also have those
7:06:40that are not um directly XR like we have
7:06:44blending blender and modeling tutorials
7:06:47so that people can make their own assets
7:06:49and import these to their VR or ar
7:06:52projects so we try to cover a lot of
7:06:54bases with our workshops and the the
7:06:57workshops are organized by you guys
7:06:59right it's not like you're um I I I
7:07:02remember attending some of the workshops
7:07:04at least the ones that I attended there
7:07:05were uh other students giving those
7:07:07workshops um is that is that the norm or
7:07:09was that uh special that day so uh for
7:07:13the most part we are the ones our
7:07:15leadership members are the ones who put
7:07:16on the workshops uh it usually takes our
7:07:19leadership members you a few weeks maybe
7:07:21even a few months to get the workshop
7:07:23materials together to practice uh to
7:07:26create an online blog for those
7:07:28workshops that other students could
7:07:29follow along with on our website um so
7:07:32for the most part it is us who are
7:07:34putting them on although like you said
7:07:36uh we do sometimes invite guests from
7:07:38other student organizations or from
7:07:41companies to come in and give workshops
7:07:43of their own so we had a Hollens
7:07:45Workshop uh last year that we did not
7:07:48put on a different student organization
7:07:50did that for us oh that's cool so the
7:07:53partnering up I hear the piggybacking on
7:07:54other events that was with Michael
7:07:56pointed out and and you Matthew to
7:07:58really get the numbers the numbers were
7:07:59really impressive I didn't even know we
7:08:01had so many people interested because I
7:08:03see just like 40 students in my class
7:08:05right because we cap it at the moment I
7:08:07agree uh the weight lists have been
7:08:09longer and and they're in the Master's
7:08:10level but I think at the undergraduate
7:08:13level um which I think you have a lot of
7:08:16students that are undergraduates and I
7:08:18think all of you are actually
7:08:19undergraduates um yourself so you really
7:08:23you are the home that that they have at
7:08:25the moment and I think you're doing a
7:08:26really good job so we have workshops I
7:08:29know that at some stage we really have
7:08:30to talk about one of the most exciting
7:08:32Parts which is your own conference but
7:08:34um so apart from the workshops I also
7:08:37noticed a lot of fun activities so beat
7:08:40saer tournaments um is one thing what
7:08:43else have you been doing to try to you
7:08:45know keep the engagement high in your
7:08:49community uh so some of the things we've
7:08:51done uh so beatsaber tournaments are one
7:08:54of the more fun and we try to get a lot
7:08:56of people from outside our club to join
7:08:59these uh fun events as well we also do
7:09:02some movie nights or TV show nights
7:09:04where we watch something that's related
7:09:06to VR or ar uh I remember once we did a
7:09:10Black Mirror episode which was a lot of
7:09:13fun um
7:09:16and those are the ones that come to my
7:09:18mind but we try to get students outside
7:09:20our or to come to these events because
7:09:22it's more of a fun experience than a
7:09:26workshop yeah then aside from just doing
7:09:29community events like that to engage our
7:09:31Community uh We've also started to do
7:09:33project teams which has helped us have a
7:09:35core group of students who in a way are
7:09:37kind of forced to come to all of our
7:09:39meetings which helps our retention up a
7:09:41bit um but through the project teams uh
7:09:44it's usually groups of five to seven
7:09:46students working on a project that they
7:09:48come up with together and they come to
7:09:51each our meetings to meet with our
7:09:52leadership members to discuss the
7:09:55progress to go over maybe different
7:09:57project management management methods
7:09:59for them uh as well as you know this
7:10:02past year uh they had opportunity to
7:10:04meet with you Michael uh to have to get
7:10:07some guidance from you and to be able to
7:10:08get some other advice as they continue
7:10:10on their work yeah but I think one thing
7:10:13that we were worried about is really
7:10:15setting the right incentives for
7:10:16students to to keep doing an entire
7:10:18project that is essentially just like
7:10:20for their own learning and one of the
7:10:23reasons we thought I should maybe get
7:10:25more engaged is maybe I can get them
7:10:27some credits but in the end they just
7:10:29all seem to be doing it just for fun
7:10:30anyway I don't remember anybody asking
7:10:32me whether they could do this as as an
7:10:34independent study so the the incentive
7:10:38seems to be primarily I just want to
7:10:40know about this stuff but what are the
7:10:42do the students do they have specific
7:10:44goals do they want to land jobs in the
7:10:45EXA industry or is it how do they
7:10:48approach it like what's their motivation
7:10:49to come other than it being fun
7:10:51obviously are there professional
7:10:53motivations and goals behind it I think
7:10:56right now it's a bit of a mix so I know
7:10:59some our students uh are currently
7:11:01working within this industry you know
7:11:03whether it's be through internships or
7:11:04through consulting jobs uh but I mean
7:11:07for a large proportion of our students
7:11:10and including myself a lot of this is
7:11:12just for fun it's because uh they like
7:11:14this technology they think uh you know
7:11:17it may have a lot of different
7:11:19applications in the future and so they
7:11:21want to learn more about it now and of
7:11:23course you on campus uh we are one of
7:11:26the leading resources for students
7:11:28especially undergraduate students to
7:11:30come and learn uh about
7:11:32XR but yeah I mean I do think some of
7:11:35them would like to go into this
7:11:36professionally um but as for the project
7:11:39teams right now their incentives are
7:11:41just to create a project and then uh you
7:11:44mentioned our conference before we also
7:11:46let our Stu our student project teams
7:11:49come and present at our conference to
7:11:50present their work to show it off to the
7:11:52other companies that are there the other
7:11:56attendees yeah but I must say for you
7:11:58know self-driven learning and project I
7:12:00mean the quality of the projects was
7:12:02pretty high it was actually essentially
7:12:04self-taught uh through your workshops
7:12:06and their own attendance obviously I
7:12:09think I noticed a lot of technical
7:12:11students so you know they had a lot of
7:12:13technical skills and you can do a lot in
7:12:14XR if you can program um I think there
7:12:18something you want to change in the
7:12:19future maybe to include more people but
7:12:22um the you know the quality was really
7:12:25good so I want to uh move on to one of
7:12:29the potential highlights over the last
7:12:31two years I've attended your conference
7:12:33the XR Midwest conference remember I
7:12:37remember Michael and Matthew you kicked
7:12:39it off um last year and then we had to
7:12:42repeat this year so can you tell us a
7:12:44little bit more about the goals of that
7:12:46EXA Midwest conference who came what
7:12:48were your
7:12:49takeaways yes so the initial thought for
7:12:52putting on the conference was uh we're
7:12:55in the midwest when we were starting up
7:12:58uh Ari one of the struggles that we had
7:13:00consist Rec ly was just that we couldn't
7:13:02find that many resources outside of our
7:13:04school uh to kind of support us you know
7:13:08being in the midwest there aren't that
7:13:09many major you know companies that have
7:13:11headquarters here most of them are on
7:13:13the west coast maybe East Coast some you
7:13:15know down south in Texas uh but here in
7:13:18the Midwest there really weren't too
7:13:20many opportunities uh for both company
7:13:22relations and for different events so we
7:13:26ended up having additional funding from
7:13:28one of the programs that we applied for
7:13:30and we decided to put on a conference
7:13:33with that additional funding so for the
7:13:34first year we didn't have any sponsors
7:13:37we did pretty much everything ourselves
7:13:39and then uh we were able to get you know
7:13:42eight nine different Midwestern
7:13:43companies to come in present the
7:13:45students uh talk about how they worked
7:13:48in the midwest talk about their products
7:13:51to help Inspire our students and our
7:13:53other attendees kind learn more about
7:13:55the industry see what opportunities are
7:13:57out there and I think for our first year
7:14:00in 20 19 we ended up having around 80
7:14:03attendees we had uh 10
7:14:06speakers and overall we got a lot of
7:14:09really good feedback from that event
7:14:11another motivation for doing XR Midwest
7:14:14was essentially trying to bring all the
7:14:16different resources and knowledge that
7:14:18we've gotten from other people in the
7:14:19industry who aren't in the Midwest and
7:14:21trying to increase the accessibility of
7:14:23that so for many of our team members
7:14:26we've had the opportunity to attend uh
7:14:28XR industry conferences or XR hackathons
7:14:31across the us but for many of those
7:14:34large events where you have like key
7:14:36opportunities to network with people in
7:14:37the industry or work on really exciting
7:14:40projects at a hackathon those are
7:14:41prohibitively expensive for many
7:14:44students often costing the hundreds of
7:14:46dollars to attend such an event and so
7:14:48since there wasn't any other type of
7:14:50large event like that in the midwest yet
7:14:52we also knew that there was a growing
7:14:54Community a Grassroots movement of
7:14:56students as well as Professionals in the
7:14:58midwest interested in XR uh holding
7:15:02hosting XR Midwest is essentially our
7:15:04way of also giving back to the Greater
7:15:05Community Beyond just the University of
7:15:08Michigan well that is nicely put really
7:15:12cool and uh really successful this year
7:15:15it had to be online and the numbers were
7:15:18extremely high um in terms of attendance
7:15:21um any major takeaways from from this
7:15:24year's um version of the
7:15:27conference yes I mean this year kind of
7:15:29like you said we had to transition it
7:15:31from in person to online uh so that gave
7:15:33us about two and a half weeks to do that
7:15:36um one of
7:15:38my yeah not not too much time to restart
7:15:41an entire marketing campaign to try to
7:15:43get attendees but it end up working out
7:15:45like you said I think we had like 200
7:15:47unique visitors on the day of the event
7:15:50around 300
7:15:51registrations uh but my major takeaway
7:15:54was just
7:15:55that no matter what happens people are
7:15:58always going to be interested in this
7:16:00topic so whether we're having it in
7:16:02person or we're having it you online
7:16:04through Zoom no matter what we're going
7:16:06to get people and no matter what we're
7:16:08going to be able to you know spread the
7:16:10word or you know or like Michael said to
7:16:12be able to give back to our community uh
7:16:14you know so for the online V we were
7:16:16really focusing on not just you know
7:16:19students at University of Michigan we're
7:16:21also focusing on getting attendees from
7:16:24you know the greater State in Michigan
7:16:26from around the Midwest you know is C
7:16:28we're targeting in our marketing
7:16:31uh so pretty much no matter what we're
7:16:34always be able to get people to be
7:16:35interested and as long as we're able to
7:16:38come get our word out there and explain
7:16:40our reasoning for why we're holding the
7:16:42conference uh I I think that we'll be
7:16:45able to be successful with it okay so
7:16:47we've Ari has been really one of the
7:16:50driving forces at the undergraded space
7:16:52here on campus but outside of Ari what
7:16:54do you think are other kinds of
7:16:56opportunities for students on campus or
7:16:58even Beyond um to learn about XR can you
7:17:01give some examples there so something
7:17:04we've been doing is inviting speakers
7:17:06who are professors or are doing work in
7:17:09this field to come to our meetings and
7:17:11talk about opportunities Beyond just re
7:17:14to explore XR as this could be through
7:17:17research opportunities on campus uh work
7:17:20opportunities uh Beyond which we post to
7:17:23some of our job opportunities on our
7:17:26newsletters and we also encourage
7:17:28students to do projects outside of of
7:17:31our AR such as through attending
7:17:33heatons I remember the first year these
7:17:37heatons uh were announced we only had
7:17:39two students me and Michael attend the
7:17:42first one and then the second year the
7:17:43number goo we had four students
7:17:45attending the hecaton so we've been
7:17:47pushing our students to explore XR
7:17:50Beyond just our club at the University
7:17:53or outside the campus
7:18:06so I have basically two more things I
7:18:08really want to discuss with you before
7:18:10we talk about challenges and current
7:18:12issues and things like that uh I wanted
7:18:14to ask you guys um what do you think
7:18:17your own involvement and leading in this
7:18:20initiatives um did it pay out for you
7:18:23like I know that Michael has at least
7:18:25Michael and and Matthew have seen you've
7:18:27been speakers in events um so like what
7:18:31what kind of trajectory are you on are
7:18:33you going to be the Future Leaders in in
7:18:34the XR space is that is that even an
7:18:36ambition or is it just a side effect of
7:18:39being so active what are the benefits of
7:18:41you for you as students and in your own
7:18:44careers um leading such an initiatives
7:18:47um an OP like a career in XR as well as
7:18:53the yeah so Rew that um we found that
7:18:57one of the challenges essentially was
7:18:59trying to create resources for students
7:19:01who were interested in pursuing a career
7:19:03in this space and so we had formed this
7:19:05inter this um organization called ixr
7:19:09the inter Colgate xar community and
7:19:11essentially we Tred to bring together
7:19:12all these different universities and at
7:19:15our height we had 23 universities
7:19:17actually across the us trying to build
7:19:19up this network of sharing resources and
7:19:22advice to help get more students in this
7:19:23space and I think in terms of my career
7:19:27as well as in what I've tried to bring
7:19:29to many of the different uh
7:19:31organizations and communities both in
7:19:33person like the alternate reality
7:19:35initiative as well as communities online
7:19:37in Social VR uh the core tenant has
7:19:39always been trying to get more people
7:19:41into the space because that leads to
7:19:44more developers as well as leads to more
7:19:46users in order to help bring xar to the
7:19:49main oh that is cool and it sounds like
7:19:51a really good strategy as well so you
7:19:54guys were talking about challenges and I
7:19:55want to dedicate the rest of our time
7:19:58we've been talking about really quite a
7:20:00success uccessful Story the alternate
7:20:02reality initiative but there have been
7:20:03many challenges along the way can you
7:20:05give some examples of challenges you've
7:20:07had and then also maybe how you at least
7:20:10address them if not
7:20:12overcome well one of the first
7:20:14challenges we had starting out was the
7:20:17availability of Technology because this
7:20:19technology was still fairly new and
7:20:22still being explored so I remember uh on
7:20:26the first day we went to the
7:20:27University's visualization studio and
7:20:30there were two three headsets on
7:20:32computers and those were the resources
7:20:35we had so I remember we we bought our
7:20:38own headset uh we were struggling with
7:20:41the technology side and we had some more
7:20:43of a focus on the augmented reality side
7:20:45because of this but then as virtual
7:20:48reality grew and became more popular and
7:20:50people became more invested in this
7:20:52community uh we the university supported
7:20:56this technology as well and now the
7:20:58visualization studio is an amazing area
7:21:01to work at there are many types of
7:21:03headsets several headsets and it's very
7:21:06supported with the staff who help you
7:21:08when you have trouble and it's also
7:21:10accessible uh for our members to work
7:21:13there during our meetings so this
7:21:15availability of Technology was a
7:21:18challenge to begin with but now it's
7:21:20been one of the reasons we've been able
7:21:21to grow and push ourselves with what we
7:21:24what we can
7:21:25teach all right that's a good example I
7:21:28remember also having discussions with
7:21:30Michael and Matthew about you know
7:21:32engagement through I mean it's a student
7:21:34initiative and it lives and dies with
7:21:37the semester Cycles right exams and all
7:21:39those kinds of things so uh engagement
7:21:42and uh you know keeping keeping those
7:21:44workshops alive and filling them with
7:21:45people um there were challenges how did
7:21:48you address those
7:21:50challenges yes so uh like you mentioned
7:21:52now for us especially in our first two
7:21:55years of running this retention rates
7:21:57were our biggest challenge we would
7:21:59start off with 30 maybe even 40 people
7:22:02at our very first kickoff meeting right
7:22:04after that we'd be down to 20 maybe you
7:22:07say around 20 for one or two meetings
7:22:09and then we end off at 10 or five people
7:22:13there outside of our leadership members
7:22:15I remember our first year our very last
7:22:18meeting of the year we wanted to have a
7:22:20big goodbye thing uh goodbye meeting you
7:22:23know send everyone off the finals uh
7:22:26excited we plan have a game night going
7:22:28to have pizza it's going to be a good
7:22:30time
7:22:31and I don't think a single person showed
7:22:33up um so the something that we struggled
7:22:36with earlier on uh but for the most part
7:22:39how we've dealt with that is just by
7:22:41continuing to do Outreach so like I said
7:22:45earlier whenever we do Outreach through
7:22:47other recruitment events we're able to
7:22:49get you know 200 300 maybe even 400
7:22:52different students to sign up for our
7:22:54email list and of course uh we aren't
7:22:56getting 3 400 people at our meetings uh
7:23:00but as we continue to add people on uh
7:23:02with those recruiting events we're able
7:23:04to reach out to more people and for
7:23:06example this past year we had by far our
7:23:08best of attention I believe our first
7:23:10meeting started off with around 50 uh
7:23:13students at a kick up meeting which then
7:23:15went down to 40 for the next two
7:23:17meetings which then uh stayed around 30
7:23:20for a while and for the course of the
7:23:21year we ended up having probably
7:23:24averaging around 25 to 30 students at
7:23:26each of our meetings at our
7:23:28workshops uh so just just by keeping on
7:23:31recruiting and I think uh one more thing
7:23:33that we've done in this past year that I
7:23:35imagine would have helped uh for the
7:23:37retention is just that we stopped doing
7:23:40so many workshops so our first two years
7:23:43we had workshops almost every single
7:23:45meeting and then we threw in a community
7:23:47event or an open discussion uh every you
7:23:50know maybe once a month or so but for
7:23:52this past year we started to have
7:23:55community events or non- technical
7:23:57workshops uh almost every other week so
7:24:00we would alternate between technical
7:24:02workshop and then something else which I
7:24:04think helped to engage the community and
7:24:07uh you know helped keep the attention of
7:24:08maybe students that are already a little
7:24:10bit more advanced that may feel that you
7:24:12know the basic workshops that we're
7:24:14giving don't really apply to them or we
7:24:16also have students that aren't really
7:24:18there for the development side they just
7:24:20want to be part of the community they
7:24:21want to learn more so not having as many
7:24:23workshops for them helps uh because
7:24:25they're able to come to meetings where
7:24:27they just play games or watch a movie or
7:24:29hang out e
7:24:32well these are all good examples of what
7:24:34we should do with student initiatives
7:24:36damn I wish I were a student again now
7:24:38that's cool uh these are actually great
7:24:40examples and and the struggles um that
7:24:42you talked about and how you addressed
7:24:44them I think this very useful okay so
7:24:46these are some of the current challenges
7:24:47you talked about but let's go a little
7:24:49bit back in in in time I remember when
7:24:51you started out I think Michael coming
7:24:53to my office was very exciting uh to get
7:24:55to know you guys um so what were some of
7:24:59the challenges you really had to to
7:25:01start this initially one of the first
7:25:03things that we did um especially Michael
7:25:05when we were just starting out and when
7:25:07we were starting to form our first
7:25:08leadership team was that we ended up
7:25:10reaching out to any and all professors
7:25:13and other faculty members at the
7:25:14University who were working with XR and
7:25:17we reach out to try to get more
7:25:19information from them on what the
7:25:21ecosystem at the University of Michigan
7:25:23looks like already looked like we
7:25:25reached out to them to try to get any
7:25:27guidance to try to get any feedback on
7:25:29what we were doing
7:25:31uh but again as freshman uh it was
7:25:35difficult to really get their attention
7:25:37and for most of the faculty members that
7:25:39we reached out to we were able to meet
7:25:41with them you know through a video call
7:25:42over the summer but then after that uh
7:25:45it didn't really lead anywhere uh we
7:25:47were able to kind of make some
7:25:49connections and some relationships for
7:25:51example with a few of the vice provosts
7:25:53of course you know we met you and we've
7:25:55been able to talk with you Michael uh
7:25:57but for the most part uh that didn't end
7:25:59up really going any where uh so what we
7:26:02did was uh we focused on other students
7:26:05and so we spoke with around 50 different
7:26:07students from other universities that
7:26:09were running their own XR organizations
7:26:12and we spoke with them about what they
7:26:14were doing that was successful what
7:26:15events they were holding and of course
7:26:17what they were doing that was
7:26:18unsuccessful and I think a lot of how we
7:26:21shaped Arie to be what it is today was
7:26:24shaping it after that feedback that we
7:26:26had gotten that advice that we had
7:26:27gotten from other students who were
7:26:29active in this Community uh so a lot a
7:26:32large part of our success is thanks to
7:26:34them and is thanks to the that community
7:26:36that we were able to serve that's cool
7:26:39and it's fair I mean to be honest uh
7:26:41when we at the professor level started
7:26:43to to you know get people interested it
7:26:46was almost similar story you very
7:26:48quickly get 50 people in the room right
7:26:51I I've heard about
7:26:53XR so I feel like I want to do this but
7:26:56the retention rate I mean you know there
7:26:58were actually not a lot of people at the
7:27:00time you wanted to really kick this off
7:27:01I think there were not a lot of people
7:27:03on campus that were ambitious and brave
7:27:05enough faculty I mean now I really hope
7:27:08it it it would be different now that if
7:27:10you if you started re right now with the
7:27:13resources that we probably have now and
7:27:15the growing interest on campus which you
7:27:17know is it would be a different timeline
7:27:19because that success is also due to you
7:27:22so Star Trek timelines don't really work
7:27:24that way um so anyway I I feel like um
7:27:29it is a it is a issue and I guess
7:27:31persistent and just doing it by
7:27:34yourselves really um you you got a long
7:27:37way that way and I feel like other
7:27:39students should not be discouraged I
7:27:41think if you find one or two faculty
7:27:43members that Express support it is going
7:27:46to be helpful but even if not you you
7:27:49students are very talented so um I think
7:27:52Ari is really just you guys um I'm going
7:27:56to throw in one last challenging one
7:27:58which is so I guess we now have an
7:28:01understanding of uh the alternate
7:28:02reality initiative your student
7:28:04initiative and what it is right now and
7:28:06I'm wondering whether you you want it to
7:28:09be just that or do you have um new
7:28:12bigger plans for it even and whe and
7:28:15even if not is there anything that is
7:28:18missing is there any any kind of support
7:28:20that you that you wished for that I mean
7:28:24if you could make a wish what would that
7:28:25be and I'm not promising that I'll you
7:28:28know that I'll be in a position to
7:28:31fulfill it but I would like to learn
7:28:33from you guys what you see as a really a
7:28:35major obstacle that that you've overcome
7:28:38so many so is there anything else that's
7:28:40keeping you holding you back from
7:28:42from the AR
7:28:44dreams one of the biggest challenges
7:28:46that I see both with the alternate
7:28:48reality initiative as well as with all
7:28:51the other XR University organizations is
7:28:54essentially trying to overcome the
7:28:56uncertainty of the future of XR I from
7:29:00what we've seen is that many students
7:29:01are very passionate at least initially
7:29:03to learn more about the technology and
7:29:05try to work on as many projects as
7:29:08possible but I think one of the
7:29:09challenges of taking that initial
7:29:11interest to the next level of possibly
7:29:13pursuing a career in this space is who
7:29:16has actually done that so far for so
7:29:18many of the people that we've met these
7:29:21are essentially like the first group of
7:29:22students who are coming into college and
7:29:24university actually starting off with
7:29:26their interest in XR and then possibly
7:29:29pursuing a career in x are versus most
7:29:31of the other people in the industry Who
7:29:33currently come from often times a gaming
7:29:35industry or from software consumer
7:29:38Technologies so in this case we're kind
7:29:39of this initial class of students who
7:29:42are trying to create a pathway showing
7:29:45other students how they could Poss to
7:29:46pursue a career in this space when so
7:29:49much is still unknown about like whether
7:29:52it will grow tremendously as well as
7:29:54what are the opportunities that are
7:29:55available because for pretty much every
7:29:57single University of Michigan student
7:29:59we're all really good at finding
7:30:01different opportunities different ways
7:30:02to make our
7:30:05Mark yeah thanks this is this is really
7:30:08helpful and this is a good summary
7:30:09actually I think of our of our entire
7:30:12discussion so I really enjoyed actually
7:30:13talking to you and hearing it obviously
7:30:15it's not the first time I meet you but I
7:30:17always I'm always really excited when I
7:30:19listen to your experience and I you know
7:30:22the credit is really just with you guys
7:30:24you have done this uh almost entirely
7:30:26just by yourselves and you're going
7:30:28through different cycles of the
7:30:29leadership happy to meet the new
7:30:31presidents and the former presidents and
7:30:33we're still friends and everything so
7:30:35that's great uh it is actually in many
7:30:37ways a a great example of a student
7:30:39initiative and I think Michael put it
7:30:42really nicely we we are still in this
7:30:45space where there will be the first
7:30:46cohorts of students coming out it would
7:30:48be interesting when people have have had
7:30:51XR when they were babies right like I've
7:30:54been doing XR all my life uh it'll it'll
7:30:56be a different audience and probably new
7:30:58challenges to address this then uh you
7:31:01know smartphones are not interesting
7:31:03anymore everybody has them I wonder
7:31:05whether we have similar issues at some
7:31:07stage with XR and um it was really great
7:31:10to to hear from your initiative and I
7:31:14think our Learners which I hope many of
7:31:16of them are actually students like
7:31:18yourself get excited about starting
7:31:20something like this um so thanks for
7:31:22talking to me about it
7:31:37congratulations you've made it through
7:31:39this first course as part of the X amuk
7:31:42this course was really focused on the
7:31:43terminology learning about the concepts
7:31:46and Technologies and then learning about
7:31:48the trends and then discussing the
7:31:51issues finally also a little bit on
7:31:54strategy um and I think um these are all
7:31:58just um initial insights into XR this
7:32:01space is really evolving very quickly
7:32:04which is probably one of the main
7:32:06takeaways uh that you will hopefully
7:32:08have it's really hard to actually have
7:32:10an overview like uh this so like um I
7:32:15really enjoyed putting this uh content
7:32:17together if you made it through this
7:32:20course that means you actually have a
7:32:22really good overview of the space and
7:32:25and actually that would be really great
7:32:26for having critical discussions and also
7:32:29then thinking maybe about ways of doing
7:32:32XR which is really the focus of some of
7:32:34our other courses focused on design and
7:32:37development so now that you have made it
7:32:39through this course let us celebrate a
7:32:41little bit together and look at well the
7:32:44things that we should know about at this
7:32:46stage so this was the road through the
7:32:49course right I hope it wasn't too Rocky
7:32:52and uh if you haven't had the
7:32:54opportunity to look at some of the
7:32:55honors track contents and also some of
7:32:58the exercises
7:33:00uh I would still encourage you to do it
7:33:02I think it's a an even better way to
7:33:04learn about these Technologies um but
7:33:07you know it's fine also just going
7:33:09through these and maybe making it
7:33:10through the quizzes already would make
7:33:12me very happy and hopefully provide you
7:33:15a little bit of at least uh some
7:33:18fundamental knowledge in this space so
7:33:20one of the things we've done actually
7:33:22quite fundamentally is really
7:33:23distinguishing virtual reality from
7:33:25augmented reality right and so by now
7:33:27you should know that virtual reality
7:33:29really replace reality and augmented
7:33:31reality enhances reality and you should
7:33:33really refer to each like using the
7:33:35right terminology you shouldn't confuse
7:33:37the two AR and VR it happens all the
7:33:40time in discussions with people that are
7:33:42not as engaged in this space so I really
7:33:44want you to to know those things and
7:33:46know the differences and then I always
7:33:48like to say that the r in virtual
7:33:51reality is not the same r as the r in
7:33:54augmented reality uh the r in augmented
7:33:56reality is actually the real world and
7:33:58in and and the R in virtual reality is
7:34:00really a computer generated somehow
7:34:03designed uh synthetic World we've also
7:34:06learned about the reality virtuality
7:34:07Continuum and we've actually done
7:34:09exercises around this uh classifying
7:34:12existing applications along this uh
7:34:14Continuum as part of the honors track um
7:34:17we did learn about some of the display
7:34:19types uh throughout uh from those that
7:34:22are a little uh well weaker on the AR
7:34:26side to more AR into the virtual space
7:34:29let's just say it this way we've G we've
7:34:32we have actually learned about each of
7:34:35these Technologies driving VR and AR so
7:34:40um I think that was uh quite cool that
7:34:42we now have that understanding and uh
7:34:45can put together this technology tree
7:34:49that we perhaps now can navigate and um
7:34:53understand some of the things that I put
7:34:55in here um but we've also learned about
7:34:59how to ask questions around these
7:35:01Technologies uh so I've introduced you
7:35:04to this idea of doing qoc based design
7:35:08space analysis really uh asking
7:35:11fundamental questions about some of the
7:35:13things you may be designing in the
7:35:16future and also thinking about available
7:35:19options for each of these questions and
7:35:22then criteria and this qoc approach is
7:35:25really more like a formalization of all
7:35:27the things that are going on in my head
7:35:30when people are asking me about how
7:35:31would you do this Michael what kind of
7:35:33Technologies do I need is this even
7:35:35possible could we even do something like
7:35:38this and so I think it's it's it's very
7:35:40good and approaching it from this
7:35:43questions options and citerior
7:35:45perspective may allow you to involve
7:35:49these stakeholders to involve maybe your
7:35:51potential users in some of the
7:35:52discussions rather than th just throwing
7:35:55the Technologies around their ears you
7:35:57can actually engage in meaningful
7:36:00questions so that was uh the goal of
7:36:02that one and uh I I hope that um you
7:36:05will adopt this somehow into some of
7:36:07your practices that would be really
7:36:09cool learned about the trends really
7:36:12going through uh it from an HCI a human
7:36:15computer interaction perspective
7:36:17thinking about people tasks technology
7:36:19and how each have evolved so we now know
7:36:22that obviously there's a lot of interest
7:36:24in the social arvr space multi-user
7:36:27experiences cross device arvr is very
7:36:30interesting at the same time while
7:36:32initially we were just enabling more or
7:36:35less Able Body users we really now have
7:36:37to think about inclusivity and enabling
7:36:40all kinds of uh users uh including those
7:36:43with special needs and we actually have
7:36:45to do an even better job there although
7:36:48we beginning to see some really good
7:36:49examples in that space so initially what
7:36:53we can see here in terms of context from
7:36:55uh more specific a little bit more
7:36:58constrained context uh like your work
7:37:00areas or living room um to really the
7:37:04world and to a new world where AR at
7:37:08least AR will be always on and always
7:37:11available uh which is really going to
7:37:13enable all kinds of contexts and um that
7:37:16is a future that we need to prepare for
7:37:19and uh knowing about this these Trends
7:37:22and the issues I think is a at least a
7:37:25good uh kind of
7:37:27preparation I've shared a few technology
7:37:31Trends obviously from tethered uh to
7:37:34Standalone devices to buil in we will
7:37:36actually see more and more arvr
7:37:40Technologies built into various devices
7:37:42more devices will have cameras and
7:37:44sensors and some of these AR techniques
7:37:47techniques that we would now associate
7:37:49with ar uh will actually be much more
7:37:51widespread the computation will be going
7:37:54around between requiring additional
7:37:56Hardware doing more or less everything
7:37:59in soft and then Outsourcing to the
7:38:01cloud I think uh we will see and maybe
7:38:04you agree with this or we will see um in
7:38:07the next few years and obviously uh we
7:38:10also can see a technological trend on
7:38:12the application side from dedicated arvr
7:38:15only Standalone apps to apps that are
7:38:17including arvr modes to maybe in the
7:38:20future even apps that are uh basically
7:38:23porting the content into both AR or VR
7:38:26depending on task context and user
7:38:29preference
7:38:30uh this idea of cross reality um
7:38:32designing interfaces that can actually
7:38:34span these different kinds of modalities
7:38:38and that is something that I also expand
7:38:40on in the design and development
7:38:43oriented course we talked about the
7:38:44issues so we should have at least a good
7:38:46overview of some of the well classes of
7:38:49issues in the space right social and
7:38:50ethical concerns um accessibility in
7:38:53equity and privacy and security there
7:38:56are additional issues obviously but
7:38:58these are some of the really big ones
7:39:00beyond the technological considerations
7:39:02or the design
7:39:04considerations uh that we really focus
7:39:06on in some of the other courses but at a
7:39:08higher level of thinking and really
7:39:10actually what I want us to think about
7:39:13more as a community as a society uh
7:39:17these are really some of the key issues
7:39:20so in the last part I was talking about
7:39:22strategy was talking about h building
7:39:25this growth mindset right around
7:39:28knowledge and and your team how to
7:39:30actually build both knowledge and
7:39:33actually diversify your team and what
7:39:36are good strategies there at least have
7:39:38shared some of the strategies that I
7:39:40have used or some of the things that I'm
7:39:41looking for when I'm filling new roles
7:39:44we talked about equipment and users uh
7:39:46this is just the beginning of a strategy
7:39:49discussion uh I do think that um XR is
7:39:53actually all about strategy um strategy
7:39:57in terms of like what projects you work
7:39:58on what what initiatives you contribute
7:40:00to really what you believe in where this
7:40:02space should be moving and then also how
7:40:05you continue to learn about this space
7:40:07so for example I can say for myself that
7:40:10while I have been active in this space
7:40:12for a few years now and actually have
7:40:13been teaching it for many years putting
7:40:16this moo uh together and also the other
7:40:18courses I feel like I've learned even
7:40:21more uh hopefully you also learned uh
7:40:24quite a bit by well going through this
7:40:27together uh with me
7:40:30and um well I hope uh by that time
7:40:33people have already had some really cool
7:40:36uh discussions in uh some of the
7:40:38interactive elements throughout this
7:40:40course uh I invite you to look at course
7:40:432 and course three where some of our
7:40:46Learners are actually working on real
7:40:48projects in this space designing uh a
7:40:52new AR of VR experience and then also
7:40:55taking it into the development stages as
7:40:58well so I think um even if you're not
7:41:02necessarily a designer or developer uh
7:41:05you could have a little bit of a peak in
7:41:07there and see what's going on I would it
7:41:09would be great to see you there as well
7:41:12so this course then was really on the
7:41:15knowledge and shaping of XR uh this is
7:41:18the first course in a series it is
7:41:21focused on really providing this
7:41:22fundamental
7:41:23introduction and then we have two more
7:41:25courses one focused really on design and
7:41:28design thinking and prototyping and then
7:41:31one focused on development really
7:41:33building this toolbx and working with
7:41:36the tools ENT Technologies themselves
7:41:39overcoming some of the barriers to entry
7:41:41that are still uh there making this jump
7:41:44from 2D to 3D and then learning really
7:41:47about specific uh VR design and
7:41:50development aspects and the same for AR
7:41:53as well finally we're going to look at
7:41:54some of the more advanced Topics in the
7:41:56space we're also going to learn a little
7:41:57bit about XR research which I really
7:42:00care about as well so these are courses
7:42:04that I would uh obviously like to see
7:42:06you take if you're a little bit
7:42:08interested or passionate about some of
7:42:10these um but this is really our uh first
7:42:14course is part of the larger
7:42:16specialization so here I'll end with an
7:42:19overview of the specialization you have
7:42:21now made it through this
7:42:23course again the design one is really
7:42:26focused on methods and tools with
7:42:28additional exercises around design
7:42:30critiques storyboarding and prototyping
7:42:33both with physical tools and then also
7:42:36digital tools and even doing it
7:42:37immersively that means directly in AR or
7:42:40VR quite interesting also as part of
7:42:42this course and the second or actually
7:42:45in fact the third course here um you
7:42:47would uh actually engage in a project
7:42:50and then finally do a peer review of
7:42:53other projects actually quite exciting
7:42:55so the development oriented course uses
7:42:57slightly different language and has has
7:42:59a little bit of an expectation of you
7:43:01being eager to learn some of the
7:43:04technology behind some of this and
7:43:06working with the tools and so it takes
7:43:09more of a programmer approach if you
7:43:11will but it is still not too hard it
7:43:14doesn't require computer Graphics or
7:43:16whatever you may think backgrounds it is
7:43:19uh still well it's taught by me and I am
7:43:21somebody who works in the area of human
7:43:23computer interaction so the exercises
7:43:25there will be making this jump from 2D
7:43:27to 3D building a 3D scene building a VR
7:43:31and an AR scene and then as I said these
7:43:33actually add onto each other and you
7:43:36will actually create a project this way
7:43:38which will finally be peer reviewed I
7:43:41invite you to join me in this world
7:43:43around XR and really contributing to
7:43:46this discussion and also by being active
7:43:49in the space as a doer and shaper which
7:43:53I think would be a great accomplishment
7:43:55thanks again for taking this course
7:43:58again I had a lot of fun preparing this
7:44:01I hope um you could feel that a little
7:44:04bit uh it wasn't always easy to record
7:44:06all this content uh
7:44:08covid-19 uh really had quite an impact
7:44:12on uh actually all the courses uh so I'm
7:44:15glad we made it uh to this point and uh
7:44:17thanks again for taking this course with
7:44:20me and really don't hesitate to reach
7:44:22out I would like to hear from you um it
7:44:24was actually nice to work with you um so
7:44:28hopefully I'll see you in uh some of the
7:44:30other courses or in some versions of
7:44:32this course or other things I may do in
7:44:34the future you never know where it takes
7:44:37people so bye-bye

About this transcript

This page contains the full transcript of INTRO TO AR / VR / MR / XR : TECHNOLOGIES , APPLICATIONS & ISSUES | MODULE - 1 by LEARN FROM EXPERT, generated from the public captions YouTube serves with the video. The transcript has 80,197 words across 11,202 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.