YT YouTube2Text

YouTube2Text → Transcripts → USER EXPERIENCE INTERACTION DESIGN FOR AR / VR / MR / XR | MODULE -2

USER EXPERIENCE INTERACTION DESIGN FOR AR / VR / MR / XR | MODULE -2 — Transcript

by LEARN FROM EXPERT · 103,689 words · 14,653 segments · language en · Watch on YouTube

Open in transcriber Summarize with AI Download SRT/VTT

Full transcript

0:06hello there uh I am Mich nebling uh I'm
0:10your instructor I am a professor an
0:14assistant professor to be precise at the
0:16University of Michigan where I work in
0:18the school of information um I thought I
0:21should introduce myself and tell you
0:23maybe a few things about myself before
0:24we go really into this XR Muk
0:27specialization um so I I've been working
0:30in the space for about 5 years now I
0:32would say uh I've been doing research in
0:35this space specifically in the area of
0:37human computer interaction and I've been
0:39teaching uh in the residential program
0:41at the University of Michigan for three
0:44years now um two arvr courses one
0:47focused really on design and one focused
0:49on development this is in many ways an
0:51update of the teach out uh that I've
0:54have done a few years ago with a few of
0:55my colleagues it was the augmented
0:58reality virtual reality and mixed
0:59realties teach out from the University
1:01of Michigan maybe you've seen me there H
1:03it was great to meet some of the
1:05Learners in that uh moo and uh many of
1:09you have actually said we wanted to
1:11learn more about design and development
1:13so we've heard you and we've added a
1:15significant content in both of these
1:18areas right so I care about XR you guess
1:23that um it is really important for me
1:26that we all understand these
1:27Technologies they are making it into our
1:30everyday lives um they are there they're
1:32not going to go away uh we still have an
1:35influence on how to shape those
1:37Technologies so that we have a better
1:40control right how these Technologies
1:43make it into our everyday lives in any
1:45case it's time for me to pay back so and
1:48pay it forward so I hope what you learn
1:51here today uh is that is something that
1:53I had to learn uh myself not just today
1:56but over the next few weeks it's not
1:57going to be that fast almost trick do
2:00there right and um I hope that you share
2:03that knowledge and that you invite other
2:05people to also learn about these things
2:07not everybody cares about development
2:09maybe some of us care about design
2:10hopefully even more and then a lot of us
2:13should really care about where this
2:15space and these Technologies are going
2:17and we should have some critical
2:19discussions about this so thanks for uh
2:23well letting me meet you and joining me
2:25in this uh effort of spreading the
2:27knowledge around XR and um yeah I really
2:32look forward to seeing uh you in the
2:34course hope we get to interact with each
2:35other maybe perhaps even meet in person
2:37one day that would be exciting um hello
2:41and welcome to the XR
2:51Muk course overview of the user
2:54experience in interaction design for
2:56augmented reality virtual reality and
2:59mixed course as part of the xam Muk in
3:02this course we're going to talk about XR
3:05design and how to make that transition
3:07from more or less traditional user
3:09experience and interaction design
3:11principles methods and tools to
3:14XR and this is a transition I've I've
3:16made myself and um I've taught these
3:19topics for many years and I've done
3:22research in this space of XR design and
3:25coming up with new ways of enriching
3:27paper and making paper immersive so that
3:30we can use it to actually prototype aivr
3:32interfaces without writing code and
3:34without implementation so the topics
3:36I'll cover in this course are very very
3:38important to me and I hope you'll learn
3:40a
3:41lot so the way of structured this course
3:44is into four Big Blocks we're going to
3:48have an intro where we're going to talk
3:51about what doing XR involves finding
3:54your process and really how do you get
3:56started and then we're going to talk
3:58about design thinking design thinking is
4:01this six-step method coming out of the
4:03Stanford design school and it's been
4:06applied uh for many years in many
4:08different areas is often associated with
4:09product development um it is really
4:12about a mindset of how to impro how to
4:15approach uh design and in I want to make
4:18that transition to XR and and now there
4:21are interesting challenges with XR for
4:23example not a lot of users have a lot of
4:25experience with XR there are some early
4:28adopters you might be one of them but
4:30it's not like that everybody really
4:33experiences XR all day every day it's
4:36not like with Mobile on the web where we
4:38now have an established set of tools
4:41methods and principles so with XR it's
4:44actually quite different and also the
4:45process is still something that is
4:48constantly evolving so I think it's a
4:50very good thing to look at design
4:52thinking and how this may translate to
4:54XR and so then we're going to continue
4:57into prototyping both physically
5:00and digitally and I've divided this into
5:02two kinds of separate modules because I
5:06personally think there's a lot of value
5:07in physical prototyping it's a lot of
5:10fun it allows us to Think Through user
5:13experiences and prototype them out using
5:16physical materials like paper
5:17transparency cardboard and Play-Doh for
5:21the 3D modeling Parts which is actually
5:23quite fun and really allows us to think
5:26through an experience before we
5:27implement it in code or before we move
5:29digitally
5:30and then we also talk about doing things
5:32digitally and then when we do things
5:34digitally we have a number of options
5:36since now we're doing arvr we don't just
5:38have to do traditional desktop 2D design
5:41we can actually jump right into into the
5:43headsets and doing it in a way that is
5:46called immersive prototyping and so when
5:48we have covered each of these blocks
5:51I'll give you a quiz and we'll assess
5:53your learning and we're making sure that
5:55we're on track in addition to the
5:57quizzes you also have options to access
6:00additional content as part of the honors
6:02track the honors track really comes with
6:05a set of activities and exercises that
6:08allow you to practice and exercise the
6:10methods and
6:11skills so in this case I have created an
6:15exercise around design thinking it's
6:17actually a two-step exercise focused on
6:20critique and we're doing design
6:22critiques of existing interfaces first
6:23based on guidelines and then going also
6:25deeper on
6:28ethics and the second part of the honors
6:30track we're looking more at physical
6:32prototyping and we're coming out of
6:34storyboarding and wireframing and we're
6:36going to do paper
6:37prototypes paper prototypes to me is
6:39always one of the most fun activities in
6:42any kind of classes that I teach I think
6:45it's a way for the students to really
6:47Express and explore their creativity and
6:50I hope you'll find them valuable and
6:52maybe you learn a thing or two of how to
6:53make good use of physical materials um
6:57to really you know think about the our
7:00experience and play it through walk
7:02through it and walk your stakeholders
7:05through it before you sit down and
7:07really go into coding very very
7:09important a timesaver and avoids
7:11premature commitment to anything arvr
7:14just because you think you want to do
7:16arvr doesn't mean that it's the right
7:18solution and so that's something you may
7:20be able to eliminate and rule out early
7:22in the process so that's actually really
7:25quite important and then in the last
7:27block the Honest track associated with
7:29liter prototyping we're going to have
7:30two types of prototypes we're going to
7:32do a digital prototype that is basically
7:34just using desktop tools and then
7:36allowing us to preview on an arvr
7:40device and when we do the immersive
7:42prototype that's really something that
7:44I'm still exploring myself understanding
7:46really what's the value of it what are
7:48the kinds of things we should do so you
7:49can sketch in 3D and all kinds of do all
7:52kinds of interesting things and if it
7:54really good and fasted it it might feel
7:56very productive but I'm still trying to
7:59figure out what's the real value in the
8:01larger workflow in the larger process
8:04now none of these techniques are
8:05designed to replace each other they are
8:07all designed to complement each other
8:09let's take a look again as you're doing
8:11your storyboarding which revolves around
8:14biof frames and organizes wireframes
8:16basically into sequence and you're
8:17deciding on user flow then you're
8:19jumping into paper prototypes and then
8:21digital prototypes and then maybe
8:23optionally if you want an immersive
8:25prototype now each of these really tell
8:28you more about design so in each of
8:30these steps you learn more about their
8:33requirements and you're getting closer
8:36to what we then call the minimum viable
8:52product in this video we're going to
8:54talk about what doing XR involves we're
8:57not actually doing XR yet uh uh so this
9:00is not the video that goes through all
9:01the design and development activities
9:03that are out there now in this one I
9:05really want to provide an overview and I
9:07want to draw an analogy to film making
9:10because in many ways I think creating an
9:12XR experience is similar to making a
9:14movie why do I say that well think about
9:17it so creating a movie is really a
9:20creative complex task it usually
9:23involves lots of people in different
9:25roles so uh from the director uh and the
9:28person writing the script uh to the
9:31Director of Photography or the cameraman
9:33then you need actors bringing the
9:35characters to life you need somebody who
9:37takes care of Light Of Sound and then
9:40obviously special effects right we often
9:43go to the movies and we really talk
9:44about two things either great story or
9:47great special effects now the best
9:48movies get both right and uh with XR
9:52it's it's really the same yeah so we
9:54have a story that we somehow have to
9:57convey in VR or in AR and I'll think
10:00about story in in in in a variety of
10:02ways and I'll explo explain that a
10:04little bit more to you and the other
10:06reason why I choose film making as this
10:09analogy is because uh everybody can make
10:12movies these days we have we can just
10:15use our smartphones to record things and
10:17then share and uh you don't actually
10:20have to have formal training and with XR
10:22it's the same uh there's lots of tools
10:24out there so we will increasingly see
10:27hobby is creating XR experiences now
10:29that is both cool but also dangerous
10:32because it would be better if we really
10:34understood XR design if we were
10:37ethically responsible and would really
10:40pay attention to you know making our
10:43users feel comfortable and safe in these
10:45XR experiences and there lots of and
10:48there are lots of examples out there
10:49where this is not always the case
10:52so let's uh take a step back and uh
10:56Define a few key terms and you've heard
10:58about these terms before for so I'm
11:00going to draw out the reality virtual
11:02reality Continuum so I put reality on
11:04one side of the spectrum and then I put
11:06virtual reality on the other side of the
11:08spectrum so now if you were to add AR it
11:11would be roughly here because it is
11:12mostly augmenting the physical V you see
11:14mostly the physical world and then we'll
11:16add in some virtual content superimposed
11:19and so merging with a physical world now
11:21the more virtual content we add the more
11:23we are entering the space of augmented
11:26virtuality so this is mostly virtual
11:28content and incorporating elements from
11:30the physical world so now that is an
11:33interesting uh design Dimension that you
11:35have here and it's really a Continuum to
11:37refer to this this part of the spectrum
11:40we often use the term mixed reality
11:42mixed reality refers to the inner part
11:44of the spectrum doesn't necessarily
11:46include VR and to most people it's
11:48almost like a synonym to AR but as you
11:50can see in this visualization that would
11:52be ignoring all the things we can do in
11:54augmented
11:55virtuality so I often use the term XR uh
11:58and some people refer to XR as extended
12:01reality suggesting it would be something
12:03more than all the previous AR VR Mr
12:06things we've seen before but I just use
12:09it as a placeholder so the x is a wild
12:11card for a m and VR so augmented mixed
12:15in virtual reality so now that we have
12:18established these uh key terms what is
12:20an XR experience so I think of XR
12:22experiences as incorporating three main
12:26elements environment 3D characters and
12:29interactions so when I talk about
12:32environment we need to distinguish the
12:34physical and the viral environment so
12:36the physical environment is obviously
12:37the one in which the user is and then
12:39the virtual environment is the one in
12:42which we Port the user in VR the user is
12:45in this completely virtual environment
12:47they may not actually have any
12:49relationship really to the physical
12:51world in which they are and in augmented
12:53reality we really want to read the
12:56physical environment and meaningfully
12:57augment it with virtual content so these
13:01are some of the main differences and
13:02challenges also when it comes to AR and
13:04VR and I'm going to expand a little bit
13:06on this idea of environment uh as we
13:08move forward in this
13:09video the second main ingredient is 3D
13:12characters and here I have examples of
13:14avatars 3D models and data now 3D models
13:17is probably the first thing that came to
13:19your mind but let's talk about avatars
13:20quickly I could be an avatar so somebody
13:24may have uh carefully captured me uh
13:28using some pH of 3D
13:30reconstruction or maybe some kind of
13:32motion capture um so that to bring me to
13:35life so that I really look like a
13:37virtual copy of the real Michael so
13:40that's avatars so then I said 3D models
13:43and 3D models it could be that somebody
13:47with a 3D artist who has skills in 3D
13:49modeling sat down and carefully modeled
13:51me and then they rigged me which means
13:54they inserted a skeleton so that they
13:56can then start to animate me and then
13:58they can find a few key animations and
14:01which then look like gestures and that
14:02way bring me to life so if I were a 3D
14:05model that would be how it's done now I
14:08think of 3D modeling really as a
14:11separate set of skills that if you have
14:14them it obviously makes you a more
14:15complete XR designer but I do think that
14:18XR design there's so many other issues
14:20we need to take care of if we think
14:22about the user experience more than any
14:24of the specific content elements and so
14:27this is really not a key set of skills
14:30that every XR designer has to have
14:33there's also lots of free 3D models out
14:35there so that's great and lots of
14:38platforms on which we can share uh 3D
14:40models so we'll introduce you to those
14:43and so you can basically get around uh
14:45doing XR without being a 3D artist
14:49finally you may want to visualize or
14:51somehow spatially represent data so for
14:54example through sound so data could be
14:57some kind of graph that you want to plot
15:00uh in an immersive way to communicate to
15:02users interesting complex Concepts and
15:05so I don't actually have a really great
15:07example here but I just wanted to tell
15:09you that these 3D characters are not
15:11just avatars and 3D models it could be
15:13really something a little bit more
15:15abstract so for example a graph or 3D
15:18sound 3D sound is more than Stereo so we
15:22could have a source somewhere in the
15:24scene that emits sound and that sound
15:26would come from that direction and we
15:29would adjust the pitch and the volume
15:30accordingly so these were three examples
15:33of 3D characters and there are many more
15:35let's talk about interactions the most
15:37important thing to distinguish with
15:38interactions is there are explicit
15:40interactions and implicit interactions
15:42so explicit interactions are those where
15:44the user is really aware of them for
15:46example gestures that they know the
15:48system will recognize or speech commands
15:51that the user is aware the system will
15:53understand implicit interactions are
15:55those that are not directly asked of the
15:58user so for for example in AR and VR we
16:00have a lot of implicit camera based
16:02interactions gaze head gaze or eye gaze
16:05based interactions H and these are
16:08examples of implicit interactions so in
16:10arvr we really have this interesting
16:12interplay of implicit and explicit
16:14interactions and we really need to
16:16design for that that has its own
16:18challenges and one of the main issues
16:20currently is that most design tools so
16:23those geared towards non-technical users
16:25are often very limited in the kinds of
16:28interactions that they support so there
16:30are toolkits and libraries out there
16:33more geared towards programmers that'll
16:35introduce you uh to in the course focus
16:38more on development to really uh expand
16:41on the vocabulary of interactions that
16:44we have available as XR
16:46designers so if we step back a little
16:49bit I mostly talk about content and
16:52interactions when I say content I really
16:55think about both the 3D models and all
16:57the 3D characters in the scene and also
17:00the environment so basically the
17:02physical environment in which the scene
17:04takes place and then the virtual
17:06environment these are more like the
17:07background elements and the 3D
17:09characters are really more like the
17:10foreground elements but both are content
17:13elements okay so that is how I think
17:16about XR experiences the main
17:18ingredients in terms of content and
17:20interactions so now I want to think a
17:22little bit more about where the
17:25experience takes place
17:27context and what is really the story of
17:30the experience to come back to this
17:32analogy from film making so when it
17:34comes to context we really need to
17:36distinguish AR and VR so context the
17:40physical environment around the user
17:42really impacts the virtual reality
17:44experience now even though you don't
17:46really see it you design this VR
17:48environment and uh that actually needs
17:50to fit in the real world let me
17:53illustrate this quickly so you are
17:55designing the inner bubble here the VR
17:58world and it is actually somehow
18:00enclosed in reality now the virtual
18:03reality the virtual world may be much
18:06larger physically speaking than the
18:09actual physical room in which you are
18:11and this actually introduces interesting
18:13challenges I'll introduce you to
18:15Locomotion and ideas of redirected
18:18walking and then common metaphors for
18:21travel in VR for example teleporting so
18:24we learn about all these things but it
18:26is important to really think about the
18:28fact that the virtual environment does
18:31normally not perfectly match the
18:33physical environment in which the user
18:35is um because we are trying to Port them
18:38into a different world um that is often
18:41the idea behind virtual reality and so
18:43this introduces interesting issues and
18:45so context is really important so when
18:48we now think about AR there's a more
18:51direct relationship between context and
18:53augmented reality so the physical world
18:55in which you are really drives the AR
18:58experience you design this AR
19:01environment and it actually has to blend
19:03with the reality so it has to merge it
19:06has to not just fit in but it actually
19:08has to complement add or remove
19:11something from this physical world and
19:12in order to do this meaningfully you
19:14really have to understand it so on the
19:16VR side we really talk a lot about
19:18environmental design and on the AR side
19:21we really talk a lot about environmental
19:23understanding and here's a key thing
19:25that I really want you to consider so if
19:27you're designing for AR it's not just
19:30designing this inner bubble you're
19:32actually designing the experience that
19:33includes also the real world around it
19:36so it's not just a matter of putting a
19:383D model in front of your user and and
19:41then having no relationship to the real
19:43world around you no a good AR experience
19:46really takes care of this uh combined
19:49this merged experience and so that's why
19:52I actually highlight both aspects here
19:55now keeping with this let's talk about
19:57story
19:59so in VR the way I think about it is you
20:02are actually telling the story at the
20:04intersection of the real and the virtual
20:06so what I highlighted here in red this
20:08boundary is really what you're working
20:11with think about it this way the best VR
20:14experiences that I've had somehow got
20:16this mapping between the physical and
20:19the virtual world really right so for
20:20example I was in some kind of game with
20:23a headset and it was collaborative and
20:25then I had to solve a puzzle and that
20:27puzzle involved pushing virtual buttons
20:30now as I was pushing these buttons in
20:31most VR experiences you don't actually
20:33have the physical the haptic feedback um
20:37the tangible haptic feedback but in this
20:40case the physical environment uh really
20:43matched the virtual reality experience
20:45there were actually physical buttons
20:47that I was able to press and so this
20:49really was a really cool story a
20:51negative example would be if you had
20:53obstacles in the way that are not
20:55communicated to the user so they're like
20:58something is like a chair is in the way
20:59or something like that and you don't see
21:01it because you're fully immersed in this
21:03VR experience that would be really
21:05really disruptive so you're really
21:08telling the story at this intersection
21:10now in AR that's different because
21:12you're actually telling the story at the
21:14union of the reality and the virtual
21:18content so you're actually telling that
21:22story that I highlighted here not just
21:24your virtual content and with no
21:26relationship to the real world
21:27unfortunately there are a lot of AR
21:29experiences out there that seem to think
21:31it's enough to just bring some kind of
21:33virtual ele element into the physical
21:36world and that's AR but it's not really
21:39AR right this augmented reality so I I
21:43just thought it would be a good way to
21:45think about storytelling that way so now
21:47that we've established this let's take a
21:50a look at an example
22:06so in the following I'd like to share an
22:09example our example of The Lion King
22:122019 remake uh we were studying uh
22:16basically we were working on a research
22:18project in my lab at the same time as
22:20the lion king um was about to come out
22:24the 2019 remake and it was so
22:27interesting to to us I mean we also we
22:30we I was just awarded uh a faculty uh
22:34award from Disney and I promised I'll do
22:38something with that research funding and
22:41I was working on this uh project where
22:44uh the idea was to really allow multiple
22:48designers to collaborate be it in AR or
22:51in VR and collaboratively author
22:55actually using immersive authoring and
22:58XR experiences and so we we looked at a
23:01couple of examples and one of the most
23:03complex examples that we looked at was
23:04The Lion King and so it was very
23:08interesting uh for us to to look at the
23:11teaser the teaser had just come out and
23:13so we analyze the teaser and we were
23:16basically using that Lion King example
23:19as a case study to really learn more
23:21about tool support and what is involved
23:25in doing XR um and and why did we choose
23:28the Lion King well The Lion King don't
23:30know if you knew that but that remake
23:32was actually directed uh and shot in in
23:35virtual reality so basically uh the
23:38cameraman was like in there uh in the
23:41headset and chasing down the animals and
23:43and things like that um and and that was
23:46how it was recorded and so it was a very
23:48interesting case study and I'll share it
23:51with you because I do think it's it's
23:52good with uh with keeping with this
23:54lecture where we want to understand what
23:56doing XR actually involves
23:59so this is uh The Lion King case study
24:02and it's from our paper which I invite
24:04you to read on on AA director um and I
24:07want to want you to pay attention to a
24:09few things I'll actually start at the
24:11end not the beginning of this project
24:14and I'll show you an example of a scene
24:16that we we shot we were still
24:18experimenting with Rafiki climbing up PR
24:21Rock and it was a really a vast virtual
24:25environment that we were trying to
24:27prototype obviously with limited means
24:29we didn't have a $250 million budget um
24:32and we had two actors um here so we're
24:36going to do this scene where Rafiki
24:38climbs up Pride Rock and um the actor or
24:42actors in this case will be Jane in the
24:44VR headset and then Katie will be our
24:47Director of Photography operating the
24:49camera uh and this is an AR capable
24:52phone and I'm I'm going to be the
24:54director and I'm going to show you uh
24:56one of the many takes uh that we we did
24:59so that you get a sense for how we
25:02worked uh on this little project and
25:04again uh I mean we are not Disney and we
25:07had a much smaller budget but I think it
25:09still serves the purpose of this lecture
25:24action
25:26cut action
25:42okay J go back I want more movement from
25:44you so why did I share this with you
25:46well um this is my example of creating
25:50an XR experience and using techniques
25:53more or less from from film making so if
25:55you think about film making uh one more
25:57time so we have all these various roles
25:59that I talked about earlier and then how
26:02do they actually translate to XR
26:05designers and developers where the
26:07director that is usually uh the person
26:09in charge and if it's a one man project
26:12then would be just the exra designer or
26:14the XR developer in a larger more
26:16professional XR experience you might
26:18have a project lead so that might not be
26:20the same kind of designer or developer
26:22but many XR experiences are really just
26:24created by almost a single person and
26:27that person would then fun
26:28in all these different kinds of designer
26:31or developer roles that I'll spell out
26:34here a little bit now the most
26:35interesting part when it comes to
26:37differences between film making and XR
26:41is that the Director of Photography so
26:42the person in charge of the camera is
26:45actually the user and now that
26:48introduces a lot of interesting
26:50challenges design challenges it gives a
26:51lot of agency to the users and um
26:54autonomy and that's really what we want
26:56that's like one of the main afford of of
26:59VR and AR this idea of being able to
27:01explore virtual content in six degrees
27:04of freedom and really go go about and
27:08choose your own uh perspective and all
27:09that but it from a design perspective
27:12you really have to make sure that you
27:14draw the attention of the user
27:15accordingly that their the correct
27:17Visual and maybe audio cues and and all
27:20these kinds of things so that's
27:21interesting challenges there and then
27:23when it comes to the actors well on the
27:26XR side often it is just a designer who
27:29composes scenes of 3D models and if
27:31there are no freely available 3D models
27:33or the designers feel like they really
27:36need more but basic custom 3D models and
27:40usually you can do so and so much
27:42without 3D modeling but then usually you
27:44would have to work with a 3D artist uh
27:47who really can create 3D models and rig
27:50them and then animate them and and as I
27:52said this is really a a set of skills
27:55that not necessarily every XR designer
27:57has to have another way to actually
27:59bring characters to life would be
28:00through motion capture and that's
28:01actually a profession so you could do
28:04this basically these actors in a body
28:06tracked suit that's how uh most of the
28:10Marvel movies out there for example are
28:12being made that would be not the most
28:14cost effective but very realistic
28:17animations that you would actually get
28:18out that
28:19way and then thinking about the
28:22remaining uh roles and their
28:23responsibilities so the Gaffers are
28:25responsible for a light somebody
28:28responsible for for the audio mixing
28:30sound and then the special effect
28:32designer uh on the XR side that would be
28:35mostly probably the XR designer choosing
28:38appropriate lighting and materials so
28:40really making sure that the 3D models
28:42have the right materials and settings
28:45and and textures and that's something
28:47we'll we'll explore a little further and
28:49then also incorporating sound and again
28:51we're working with spatial sound here so
28:53placing sound sources in 3D and making
28:56sure that the audio is according to the
28:59physical and the virtual environment the
29:02geometry of of those um which is is a
29:05very interesting task to do this well so
29:09that um audio would also be like
29:11occluded in some sense if there is an
29:13obstacle in between that really adds to
29:16a more realistic experience and then
29:18finally special effects that's something
29:21that there's only so and so much you can
29:23do as a designer I mean you have
29:24particle effects and maybe shaders if if
29:26you understand that a little bit but
29:28most of the time you'll probably have to
29:29do scripting and so this is more the
29:31role of a technically a client person so
29:34I say exra developer here so now I'm
29:37going to walk you through the steps of
29:39basically going to the beginning of this
29:41project and um just this case study uh
29:45on focused on Lion King and how did we
29:47actually work so first we started with
29:48storyboarding and wireframing and this
29:50is something that I think is inherent to
29:53any exi project that I I do I usually
29:56start on paper and I plan plan things
29:58out I have an idea and I sketch those
30:00out so brainstorming for me doesn't
30:04happen in thin air it actually happens
30:07on on paper H in this case we obviously
30:10we were picking scenes from the teaser
30:13the teaser was available to us um so
30:16that would normally just be in your head
30:18and you would try to express it on on
30:20paper so the way we use storyboarding
30:22wire framing here is uh still relatively
30:25close to the traditional sense but we
30:27really like we really figured out like
30:29what are the camera movements what kinds
30:31of characters are involved and then how
30:33do they move and how do they relate to
30:35each other and what's the story um and
30:39then which sequences are we going to
30:40pick so that's how we use story boarding
30:42and biof framing here and I'll show you
30:45a little bit uh of what my students and
30:47I came up
30:49with so here's our example uh where we
30:53my students and I worked on this Lion
30:55King remake essentially the remake of
30:57the remake if you will and we had a
31:00$25 budget so we basically bought lots
31:04of physical prototyping materials
31:05Play-Doh we did have paper and
31:07transparency and a lot of the tools and
31:10we repurposed cardboard that became then
31:14our stage and what is really cool in in
31:18in this way is basically a a physical
31:20storyboarding activity so it goes beyond
31:23paper it's actually very close to
31:25physical prototyping in fact you may
31:27look at this as a physical prototype
31:29rather than a storyboard but I do think
31:32um when it comes to AR and VR things
31:36this idea of prototyping uh and and
31:39storyboarding physically like in 3D
31:41physical space is very very important so
31:45we had a lot of iterations here and you
31:46can see how uh my students uh Jane and
31:50Amy were filming and testing and there's
31:53another person filming Neil as filming
31:56them and then Katie was also in the
31:58background so a cool uh set of uh
32:01students a little bit of a film crew
32:03they had to make sure that the
32:05background is kind of right this was one
32:07of the harder scenes to film where uh a
32:11lot of character tiny little character
32:13movements and um pretty cool uh and
32:18so yeah that was that as we were
32:21storyboarding you could already see more
32:23or less uh the physical prototyping
32:25activity but I showed you the making of
32:28the physical prototypes and now I want
32:30to share with you the physical
32:32prototypes so this is really how a user
32:34would experience them and one thing I
32:37would invite you to do if you if you
32:39feel like it um due to copyright reasons
32:41we cannot play uh anything Disney here
32:45but I would like you to if you can
32:47access YouTube and could get access to
32:50uh the teaser that was basically the
32:52inspiration for our work and then you're
32:55going to the part where James O Jones
32:59says everything the light touches is our
33:03kingdom and so when you go there then
33:05after that it really starts with this
33:07really cool soundtrack and that's how I
33:10have experienced this prototype and I
33:12usually play it to my students in my
33:14class so I'm going to play the resulting
33:18physical prototype and this is now in
33:20some sense your experiencing uh it was
33:23created with uh physical only so reality
33:27only no arvr yet but it gives you a
33:31first sense of what it might feel like
33:32to be in this virtual world um that we
33:34are prototyping
33:36here all right and here we go this is
33:39the opening
33:40scene and we're looking at Pride
33:43Rock and then Rafiki is coming up this
33:46was the thing we had previously done
33:48with VR and we're going to go there
33:50again Simba moving thee
33:52head
33:55and uh raiki breaking that thing was one
33:59of the hardest scenes we had to shoot
34:02this in slow motion you can see this
34:04flickering which is just a
34:07light and then we're placing that on
34:10Simba's
34:12head and walking to the
34:15top and showing baby
34:19Simba and then we have this really cool
34:21scene behind where you can see all the
34:23animals I mean they're a little bit
34:25abstract here but all the animals down
34:27there there all right so this is uh that
34:30example and um these were just the
34:33physical prototypes and then we were
34:36also doing the digital prototypes so we
34:37took each of the scenes and we
34:40prototyped them digitally with a number
34:43of tools uh some of which I'll also
34:45cover in in some of the courses here so
34:47we did involve Unity we used A-frame and
34:51web XR we did some modeling in blender
34:54but mostly correction several of our
34:56models actually came from Google poly
34:58and we're done with other Google blocks
35:01or tiltbrush and so these are all kinds
35:04of immersive modeling and uh immersive
35:08authoring tools so 3D modeling and
35:10immersive authoring tools and so there
35:12are actually quite a range of little
35:14Technologies involved it does not look
35:16as polished somebody in our reviews
35:18called this crude programmer art and
35:21that's probably what it is compared to
35:23Disney but even this was quite a lot of
35:25effort so let's let's take a
35:28look so here I'll show our digital
35:31prototype with scripted
35:33animations uh so it's based on the
35:39teaser and Rafiki is going up and we see
35:42Simba you see some Reflections and you
35:45can see some differences in the quality
35:46of the models this was again one of the
35:48hardest scenes to do a 9h
35:51hour uh uh animation and scene um done
35:56by one of my
35:59students and then we're going in on the
36:03final uh look at the scene
36:07here and a little bit static this
36:10shot and this one I really like because
36:13you can see a few other animals down
36:16there there was a weird um transition
36:19just now but that is the the digital
36:22prototype that we
36:24created and you can see how the camera
36:28spins a little bit that's kind of nice
36:30yeah it is a prototype first it allows
36:33you to Envision um the final
36:51experience okay and so from these
36:53examples what have we just seen so we we
36:56basically created this ran King
36:58experience and I made the analogy to
37:00film making and we have done
37:03storyboarding we have done physical
37:04prototyping and we have done the digital
37:06prototypes and at this stage what you
37:08just saw we could call the minimum
37:11viable product the
37:13MVP and this you would now need to take
37:16further and do another few iterations on
37:19it and we entering this space of
37:21actually developing XR experiences and
37:23so when I think of developing XR
37:25experiences uh I want to distinguish two
37:27types of experiences so one is
37:30relatively basic but even that one
37:32involves already a few components so you
37:33have to decide between AR and VR you
37:36usually put some kind of 3D characters
37:38in there maybe an existing 3D model and
37:40you do have to have uh at least
37:42directional light and ambient light in
37:45order to see anything in the scene so
37:47this kind of basic experience that's
37:50like a single XR designer can do this
37:52and mostly compose the scene if you look
37:54at a more complex example here so let's
37:57say weing bring in a custom 3D model
37:59maybe it's rigged and then it comes with
38:00animations maybe a few additional light
38:02sources and then also sound and so this
38:06I would Now call an advanced XR
38:08experience if you wanted to create
38:10something like this you really need to
38:11jump into different roles as an XR
38:13designer so the custom model you would
38:15actually more or less need to be a 3D
38:17artist so creating a 3D model and maybe
38:20animations you can do this in some
38:22existing tools you can do this for
38:24example partially in unity and in unreal
38:26you can do some basic modeling and and
38:29model editing at least but you would
38:30normally really jump into a different
38:32kind of tool more geared towards 3D
38:34artists and it's really quite different
38:36in terms of how XR authoring works but
38:39then as an XR designer you could still
38:41do the lights and make sure that the
38:43materials of your 3D models are properly
38:46done and that's where I think most of
38:49the issues actually happen when you
38:50download and use free 3D models there's
38:53a couple of issues there they usually
38:56come at very different scales the origin
38:58in these models isn't always really set
39:00in a uniform way so it's not center it's
39:02not the vintage point or anything like
39:04that or the bottom center or all the
39:06things that would make sense no it's
39:08like really somewhere off so what we had
39:10to do for example when we composed our
39:12scenes um that you just saw from free 3D
39:15models we did Pride Rock ourselves but a
39:17lot of the animals were just like
39:19borrowed from Google poly and they are
39:21for free and you can use those models
39:24but then you still have to pay attention
39:25to scale and the other big thing the
39:27other big issue is really the material
39:29and that the models have the right
39:31settings so that they properly reflect
39:33light and so as an EXA designer you
39:35would have to go in and fix a lot of
39:36these things and then sound so I mean we
39:40had some issues here with sound so you
39:42would want to maybe have all these
39:44animals make some sounds a lot of
39:46background music and sound and maybe
39:49really the soundtrack would have really
39:51added a lot to that experience here we
39:53couldn't do this for copyright reasons
39:55but you could imagine getting a bunch of
39:58free sounds and you could experiment
40:00with that as well and then we did script
40:03the animations I mean there's different
40:05ways in different tools so the the the
40:07portion with the stick breaking that
40:08part we did that in in unity with the
40:12animation editors there but a lot of the
40:14animations you saw that looked very
40:16crude they were actually scripted in in
40:18A-frame so basically using a combination
40:20of JavaScript in HTML it's not the best
40:22way to do these kinds of animations so
40:24now that I have established the basic
40:27big and advanced XR experience what does
40:29it actually mean in terms of uh content
40:32and interactions so when you break it
40:35down an XR experience really into the
40:36content elements into the core
40:38ingredients we're talking about 3D
40:40models and animations lights and shadows
40:43environment so the the the terrain and
40:46all these things around physics uh
40:49spatial sound maybe menus and headset
40:52displays and as I said we distinguish
40:54explicit and implicit interactions so
40:56explicit interactions like the cursor so
40:59usually the head gaze um or could be the
41:01eye gaze as well some kind of retical
41:03that shows us according to the surface
41:06that we're pointing at it gives us that
41:07feedback a controller like a VR
41:10controller or any kind of hand movement
41:12speech or voice so these are the speech
41:14of voice commands so these are examples
41:16of explicit interactions and then
41:18implicit interactions a lot of camera
41:19based movement as I said but also could
41:21be things like the GPS location so where
41:23you are in the world that could tell us
41:25a time zone and that could actually
41:27impact how we render things for example
41:29we're going to bring in this idea of
41:31fiducial markers so that is something
41:33that happens in a lot of AR experience a
41:35fiducial means like it's visible to the
41:37camera some kind of QR code or some kind
41:40of version of a marker usually printed
41:42on paper that we want to track and that
41:45tells the camera basically its
41:47relationship to that 3D content
41:49establish this coordinate system and
41:51that is often times just an implicit
41:53interaction the marker in itself
41:55although I do think there's a lot of
41:56value in marker based prototyping
41:58normally in the final user experiences
42:01the marker is just something that you
42:03have to have there to establish that
42:04coordinate system it doesn't usually
42:06play a big role and then environmental
42:09features could also trigger implicitly
42:11some kind of interaction so basically
42:13you're bringing a physical object into
42:15view or you're looking at a sofa if
42:17you're doing a furniture placement
42:18application for example these are
42:21examples of content and interactions and
42:23now let's look at how uh for different
42:26types of x experiences we're going to
42:28distinguish between VR basic and VR
42:31Advanced or immersive really and then
42:34we're going to look at Mark based and
42:35Mar what does that mean how much of of
42:38this do we actually need to fill in so
42:40in terms of like a basic VR experience
42:42you can achieve and accomplish with a
42:44few 3D models you often it's just one H
42:48and maybe some light and shadows and
42:50cursor so cardboard based kind of uh
42:54interactions and then a more immersive
42:57VR experience would really you know
42:59check a few more elements here so
43:02definitely In addition to light and
43:03shadows you really have more of an
43:06environment a virtual environment than
43:08maybe physics in there and maybe sound
43:09and that sound is spatial maybe means of
43:12navigation through menus and some kind
43:14of heads up display you may have more
43:17controller based interaction or hand
43:18based interaction and maybe access to
43:20speech device commands and and this
43:22additional set of interactions that is
43:24supported if you go back to the basic
43:27and then compare to the more immersive
43:29experience on the interaction side we're
43:31really increasing a lot more on the
43:33explicit interaction
43:35side let's look at AR so in AR we need
43:38to distinguish between marker based AR
43:40and marker less AR and I'm not so much
43:43focused on handheld versus headworn
43:46although I must say that most headworn
43:48ones are maress and the handheld ones
43:52really depends on whether your
43:53smartphone supports things like AR core
43:55AR kit and then we would actually be
43:58able to support maress experiences some
44:01people think of marab based as being
44:03inferior to Mar less just because you
44:05have to have a marker and marker less
44:07really means that the system
44:09computationally Finds Its Own features
44:10so builds its own markers and that's
44:12something we've established in in in the
44:14previous course but here I'm just going
44:17to give you some examples so let's look
44:19at Mark based which really is almost
44:22like a basic VR experience you usually
44:25maybe have some kind of menu on a
44:29handheld at least we may use GPS
44:32location and we definitely need to use
44:33the marker so the fiducial here gets a
44:35big check mark and then a mar less
44:38experience well you don't have to have
44:40the fiducial but usually you're closer
44:42to a more immersive so these Mar less AR
44:44experiences are often a lot richer just
44:46because Mar less means the platform
44:48gives you access to at least a little
44:51bit of the environment through surfaces
44:54and their normals so the orientation of
44:56the surface and then if you're working
44:58on a more advanced device that has
45:00actually spatial mapping and depth
45:01sensing and all these kinds of things
45:03you may get a better read on the
45:04environment and then you can do a lot
45:05more things like physics so you can
45:07throw virtual objects into the
45:09environment and those kinds of things
45:11could actually be done in a mark less
45:13Ari experience if you have access to
45:15these Advanced
45:17sensors so this concludes our tour of
45:20what doing XR involves and at this stage
45:23I was looking a little bit both into the
45:26design aspect ects of it but then we
45:27were also so with the wireframing and
45:29the storyboarding and then doing the
45:31physical and the digital prototyping and
45:33then we were also talking a little bit
45:35about what goes on when it comes to the
45:38development side of things and breaking
45:41down basic and advanced AR VR
45:44experiences and then also connecting to
45:46some of these Technologies now this part
45:49was really designed to be an overview we
45:51didn't go too deep in each of these
45:53aspects but that what this course that's
45:55what this course is for so we're going
45:57to really explore each of the aspects uh
46:00going further on the design side and
46:02then we also have the course more
46:04focused on development that really looks
46:06at those aspects as well so I hope you
46:09got a good first sense of what doing XR
46:12involves just based on this example um I
46:14would invite you to take a look at the
46:17making of the real lion king it's
46:20actually going to be a really cool and
46:22inspiring example and maybe something
46:25that gets you into the mood and for the
46:27rest of this course it's a lot of
46:29exciting things uh that will happen and
46:31that we'll learn about and I'm I'm I'm
46:33so glad to be able to share this stuff
46:34with
46:49you in this video we're going to talk
46:51about how to get started and this is
46:53really like a question that I get from a
46:56lot of people that want to get started
46:58in XR and don't know really where uh
47:01where to get started and um what are
47:04good examples what are good tools um
47:07should I learn this programming language
47:09should I focus on AR or should I focus
47:11on VR and what is this extended reality
47:16that you're talking about so there's a
47:18lot of confusion uh especially for
47:20novices in the space uh it's really
47:22quite a a vast space to navigate and so
47:25the idea of this lecture is to walk you
47:28through some of the key challenges uh
47:30that you have and key barriers that I
47:32have observed with students and then
47:34ways of navigating around them including
47:37paths to becoming an XR Creator so I've
47:40collected a few key barriers to entry
47:43that I commonly observe in uh students
47:46and then also many of which actually
47:47applied to myself when I was getting
47:49started in this area so the first really
47:52big issue is finding the starting point
47:55it's really unclear how to get started
47:58and where to start should I pick up a
48:00tool like Unity is is that my way into
48:02XR what if I'm a web developer how do I
48:05transition which of the web XR
48:07Frameworks out there should I be using
48:09is A-frame the right tool should I go
48:11directly learn 3js so lots of issues
48:16there the second key issue is really how
48:18to make that transition or jump from 2D
48:21to 3D many of us are actually trained in
48:232D design we are very familiar with the
48:26methods and tools there and the question
48:28is really how do I transfer that
48:30knowledge and adapt it to 3D design and
48:33so this is a big issue and a challenge
48:35for a lot of us and there also a core
48:37Focus we actually have a lecture it's
48:39part of the xam Muk that really focuses
48:41on making that transition from 2D to 3D
48:43talking about light and materials and
48:46all the kinds of things you have to pay
48:48attention to that you normally don't
48:49really concern yourself with when you're
48:51doing 2D
48:54design the third issue and key barrier
48:57to entry I would say is finding good
48:59examples first of all it's actually not
49:01clear um where to find good content and
49:05functionality if you are experienced
49:08with web design or mobile development
49:10you probably have your favorite
49:11repositories you have Design Systems you
49:14understand material design and all kinds
49:16of things there you know what responsive
49:18web design is ETC and so there are lots
49:21of examples out there and it took years
49:23to build really good repositories of
49:25knowledge and content that they shared
49:27and in the XR space we are facing this
49:29issue at the moment that there's lots of
49:32examples scattered out there but it's
49:34really hard to find them when you go to
49:36resources that are more common uh to
49:38developers like stack Overflow it's
49:40actually quite hard to browse through
49:42there and identify whether something is
49:43about XR is it 3D or is it 2D and so
49:48really finding good examples uh is an
49:50issue and I'll expand on that a little
49:51bit because we've done some research in
49:53that area as well in later segments of
49:55this video
49:56the fourth key bear to your entry is the
50:00vast landscape of tools out there for a
50:03novice User it's really unclear which
50:05tool is the best to use for them and
50:08what is actually the use case that
50:09they're designed for there's actually a
50:10lot of differences between these tools
50:12it's not like every tool can do all
50:14kinds of use cases often they are
50:16actually optimized for specific use
50:18cases like in the construction industry
50:20or more focused on movies and gaming or
50:23entertainment and those focused on
50:25education and meeting scenar iios and
50:27there are tools that allow you to build
50:29for example these kind of VR social
50:32spaces but it's unclear which of the
50:35tools to use and it's often not an
50:37option to jump into this tool that can
50:39do it all like Unity or unreal that you
50:42may have heard of these are really big
50:45comprehensive tools and they are
50:47overwhelming in the beginning and many
50:50people actually fall into this STP of
50:51trying to do it all with this one tool
50:53that they know best and my Approach is
50:55usually to get a good overview of the
50:58tools out there and think in terms of
51:00which one is good for what purpose and
51:02getting that Clarity is also one of the
51:05main things we want to accomplish as
51:07part of the xam
51:10Muk the fifth key barer to entry is
51:13really the current problem of platform
51:15fragmentation this is really something
51:17that is not new we're just repeating the
51:19same issues if you look at mobile so
51:21there's a basically Android and then iOS
51:24so Apple devices and the are the the
51:27biggest kind of that we have in the
51:29market and we have similar fragmentation
51:33in the VR space so there's Oculus and
51:36Vive and then Microsoft is trying to go
51:39for this Windows mixed reality platform
51:42that then speaks to various VR devices
51:45and also their Flagship the hollowland
51:47AR headset but basically creating an app
51:52that reaches users across all of these
51:54platforms and devices is really really
51:56difficult and it's something that the
51:58web XR standard is trying to address and
52:02then obviously there are crossplatform
52:04development tools like Unity but it's
52:07still it's still an issue and it's
52:09really raises a lot of uncertainty among
52:12designers and developers which platforms
52:14to adopt and should I go for Apple's AR
52:16kit or should I do Google's AR core and
52:19what does this mean and how can I
52:20actually develop for both platforms for
52:23example un's a foundation is something
52:25we'll talk about so lots of Technologies
52:28and just like me throwing out all of
52:29these examples um if if you're not
52:31familiar with them yet that just shows
52:33we have more work to do and we'll learn
52:35about all these things in this
52:39Mook the sixth issue is really let's say
52:42you have made it through the design and
52:45development how do you actually assess
52:47whether you have now created a user
52:49experience that is actually good I mean
52:52there are common metrics around
52:54usability such as learnability
52:56memorability and whether something feels
52:59efficient and effective to use but how
53:02does that really concretely translate to
53:05an AR of your interface and so there are
53:08tools out there that allow you to
53:10collect data and produce analytics and
53:13then visualize it for example in the
53:14form of heat Maps so you get a sense of
53:17what are the objects that users targeted
53:19at with their gaze eye gaze or head gaze
53:21depending on the technology that you're
53:23using the XR device that you're using
53:26but then if you have these heat Maps
53:28it's really unclear to me yet what are
53:31the the right metrics to really assess
53:33and that's something we'll talk about as
53:36well so let's look at the first issue
53:38the starting point in a little bit more
53:40detail so um students often ask me
53:44whether they should learn Unity or
53:46unreal which one I prefer and which one
53:49I think is the best tool I don't think
53:51this is the the right question though
53:53first of all these tools were actually
53:54not really designed to author AR and VR
53:57experiences is more like an afterthought
53:59these are game engines and they're
54:00increasingly at support and frankly it's
54:03quite sophisticated in comprehensive
54:04support but when you open those uh tools
54:07for the first time you'll be overwhelmed
54:09by all the language and terminology you
54:11have to learn and you're making this
54:13jump into this complex tool and into 3D
54:16at the same time so I'm not sure whether
54:19they are the right starting point and uh
54:22I personally do a lot of my work in in
54:25web XR just because I really want to
54:28reach across all the platforms I want to
54:30make it very easy and simple to
54:32prototype and I'll show you examples of
54:34this throughout the Muk but it's not
54:37really the answer either so starting
54:41point I would say is really taking a
54:44step back and thinking about whether you
54:47are a designer or you think you are a
54:50developer depending on how you answer
54:52this question there are actually
54:53different things you care about or you
54:55care about the same things differently
54:58so as a designer you really think about
55:00process methods authoring tools design
55:03guidelines usability evaluation
55:05techniques and as a developer it's
55:07almost the same but a little bit more
55:09technical so you think about development
55:10process development tool kits what are
55:12the right tools programming tools and
55:14programming languages uh how do design
55:17guidelines translate to design patterns
55:20and design systems and what does it
55:23actually mean architecturally speaking
55:25so what's a good way of of making my
55:27Unity or unreal apps more performant how
55:30can I improve rendering so design
55:32patterns also in that sense and then
55:34while as a designer you think a lot
55:36about usability evalation techniques as
55:38a developer you think about all the
55:39things you can track and log to build
55:42good logs that you can then analyze and
55:45so this is kind of like the ying and
55:47yang in reality these things have to
55:49come together but depending on how you
55:51see yourself you may be looking for
55:55different as ects or as I said for the
55:57same things but with a slightly more
55:59technical or less technical perspective
56:02and that's just something to realize as
56:04you're entering and trying to navigate
56:06that XR space so here's my way of
56:09thinking about how to become an XR
56:11Creator in that space so if you have a
56:14web development background there is a
56:16direct way which is based on the web XR
56:19standard and then I'll show a few
56:21examples that support that standard so
56:23if you're really heavy on JavaScript
56:24side you could do it with 3js as if you
56:27if you like declarative HTML you might
56:29go for A-frame and if you want to have a
56:31quick start into AR for example ma based
56:34AR then arjs may be a library that you
56:37want to pick up and and you will quickly
56:39achieve some initial results and feel
56:41good about
56:42it as a game developer so let's say you
56:45have a previous background in game
56:46development and maybe you even
56:48understand 3D uh design and 3D modeling
56:52then you will really already be familiar
56:54with unity and maybe real and you may
56:57really like one or the other tool more
57:00and then the good thing is that most XR
57:03platforms and devices are actually
57:06supported in unity and unreal um often
57:10through the vendors but then this is
57:11increasingly pushed and becomes really
57:13part of the unity and the unreal engines
57:16so an example of this is AR Foundation
57:19which is a part of unity now which
57:20allows you to develop against both
57:22Google's AR core and Apple's AR kit so
57:24smartphone based AR I also like the
57:27examples of steam VR which really is
57:29crossplatform uh VR so Oculus Vive and
57:33windows makes reality and then on the
57:35Windows mixed reality side I really also
57:37like mixed reality 2 Kit by Microsoft
57:39and actually some of my friends from
57:41Microsoft who are really working on this
57:43and the mixed reality to Kit really
57:45makes it very simple to develop VR and
57:49AR headset based experiences but they
57:51also increasingly Provide support for
57:54some of the smartphone based platform so
57:56if you are a game developer you really
57:58have almost the best access to these
58:01tools and the easiest transfer in terms
58:04of your existing
58:06knowledge if you have more of a mobile
58:08development background and maybe already
58:10experien with developing for Android
58:13natively or for iOS devices natively
58:16look into AR core and AR kit and their s
58:20decays because you're already familiar
58:22with these development environments and
58:24that might be the way for you to go
58:26about it and if you're more of a mobile
58:30developer I also think you will have an
58:32easier time developing directly against
58:34the Oculus and the vi SDK so I put them
58:36here as native SDK
58:39examples so now that we've established
58:41uh some of these direct paths depending
58:43on your background let's look at
58:45different ways of transitioning into
58:47some of the tools so what I'll do here
58:50is I'll paint in a few additional paths
58:52and then I'll also talk about whether I
58:54think something is easy or hard so let's
58:56look at an example so I said as a web
58:58developer you'll probably have an easy
59:00time making into web XR and then I would
59:03recommend uh using A-frame if you're
59:06really good with JavaScript maybe you
59:08can jump into 3js and learn that and if
59:11you want to have a very quick AR
59:14experience with markers you can just use
59:17arjs so if you're more like a game
59:19developer and you already familiar with
59:21unity unreal you should be able to pick
59:24up Ste VR and mrtk and also AR
59:29Foundation but for these you do require
59:32some programming experience with C or
59:35C++ depending on whether we're talking
59:37Unity or unreal uh I would not
59:40necessarily recommend jumping directly
59:42into these native sdks I just think it's
59:45not a good choice unless you want to
59:48really directly program against the
59:50platforms the one benefit that I see
59:53there is that when AR kid or a COR comes
59:56out with new features these will be
59:58initially uh only supported directly in
1:00:00the native sdks and they'll take a while
1:00:03before they actually being picked up as
1:00:04part of AR foundation for example so if
1:00:07you're more like mobile developer you
1:00:08probably already have significant
1:00:10experience with developing natively
1:00:12against Android or iOS or maybe you're
1:00:14using some kind of crossplatform
1:00:16approach like Apache Cordova and
1:00:19derivatives it should be relatively easy
1:00:21for you to pick up AR core and AR kit
1:00:24and develop on those platforms natively
1:00:26or jump directly into the Oculus SDK and
1:00:28the Vive SDK and not do this via unity
1:00:31in unreal however I do think if you have
1:00:35the mobile app development experience I
1:00:38think you are more on the advanced side
1:00:41as a programmer and I would make this
1:00:43path actually let's say doable or
1:00:47possible maybe not easy but it's
1:00:48definitely a valid path uh you would
1:00:51probably spend the majority of your time
1:00:52still trying to figure out unity and
1:00:54unreal uh and maybe you'll feel this
1:00:56slow you down before you actually get
1:00:58into using some of these sdks down here
1:01:01but it is a possibility and you should
1:01:04consider it so these were the different
1:01:07paths to being or becoming an XR Creator
1:01:10and I really wanted to cover those but I
1:01:13also talked about this significant issue
1:01:15and you will face this if you're more of
1:01:16a novice to the uh XR space which is
1:01:20making this jump from 2D to 3D so if you
1:01:23have a web design or mobile development
1:01:26background you'll be very familiar with
1:01:282D coordinate spaces and you know that
1:01:31this element is a little bit down on the
1:01:33Y and a little bit to the right on the A
1:01:35and you know how to do this but now if
1:01:38you're entering the 3D space okay we get
1:01:40it right so we have three dimensions so
1:01:42we have X Y and Z but the problem is a
1:01:46little bit of the origin and also scale
1:01:48like what does it mean we moving from
1:01:50pixels to new units that in VR and AR
1:01:54usually translate to meters
1:01:56and the origin of the coordinate system
1:01:58is really not clear and then we also
1:02:00bringing in the idea of texturing models
1:02:04and really making sure that the
1:02:05materials are right and we have light uh
1:02:09without light you see nothing in your 3D
1:02:10scene so not only do you have to make
1:02:13that transition to a three-dimensional
1:02:15space in terms of coordinate systems and
1:02:17also understanding origin and how that
1:02:19varies because usually the 0000 is where
1:02:22the user was at the start of the
1:02:24application there is no absolute
1:02:27coordinate space like we have in screen
1:02:29space so you have to make that
1:02:30transition but then you also have to
1:02:32think about the 3D models and making
1:02:35sure that they come with the right scale
1:02:36and that they have the appropriate
1:02:38materials so that they look good because
1:02:40you also need light and light is
1:02:42probably something you've never really
1:02:44thought about when you were doing 2D
1:02:45design but here light really matters
1:02:48both the ambient light and the
1:02:49directional light and really placing
1:02:51that light has significant impact on the
1:02:53kinds of elements that we actually see
1:02:55in this scene so this is quite a jump
1:02:58and we'll spend more time making that
1:03:00jump
1:03:14together so I also talked about good
1:03:16examples and finding good examples and
1:03:18uh in my research I really looked at
1:03:20what are good examples especially when
1:03:22it comes to learning about AR so you
1:03:25find some examples out there many of
1:03:27them are shared by the community the
1:03:30problem that I see there is that these
1:03:31are usually examples that are crafted
1:03:34for a very specific problem that they
1:03:37had to solve and so if that is not the
1:03:39problem you're trying to solve it's hard
1:03:41for you as a novice developer to really
1:03:43extrapolate from that example and then
1:03:45apply it to your own problem so if you
1:03:47find something more generic out there so
1:03:50applies to more problems you have this
1:03:52additional challenge that usually it
1:03:54feels quite abstract ract and maybe
1:03:57almost too complex and so it's hard for
1:03:59you even from this more General example
1:04:02to tailor it to your specific use case
1:04:04so it's really a loose loose scenario
1:04:06and so what would be a better example
1:04:08maybe somewhere in the middle U between
1:04:10the minimal that is too simple and the
1:04:15complete that is not too complex so we
1:04:17really aiming for something here we're
1:04:19looking this is like the sweet spot
1:04:22obviously something very specific has it
1:04:24advantages
1:04:26something very generic Works probably
1:04:28well for experienced developers
1:04:30something minimal works well if it's
1:04:32actually The Right Use case or it's like
1:04:36really modular and you can assemble Your
1:04:38solution from many small examples so I
1:04:41think that's also good and then
1:04:43something complete but not too complex
1:04:46is usually also a really good example to
1:04:48learn from but as a creator of an
1:04:50example it's really hard for you to find
1:04:52that sweet spot because as I said for
1:04:55the more experienced developers they're
1:04:57looking for something like this because
1:04:59it'll help them more and for the less
1:05:01experienced designers and developers
1:05:03it's probably better to have these
1:05:04smaller portions but then the struggle
1:05:06with making that extrapolation and
1:05:08imagining from this little isolated toy
1:05:10example to really imagine concrete
1:05:13applications and how it might solve
1:05:14their problems so it's really an
1:05:16interesting issue and we'll get back to
1:05:19that in the course focused more on
1:05:24development so I said there are too many
1:05:26tools out there and in fact I've written
1:05:28about these troubles with current AR and
1:05:31VR tools and I've classified them into
1:05:34five classes and what you see here is
1:05:37I'm drawing them along the level of
1:05:39fidelity that you can produce with these
1:05:41classes of tools and the skill and
1:05:43resources that are required and
1:05:45obviously there are tools that are
1:05:47relatively easy to use but you also
1:05:49can't do too much with them and then we
1:05:51have an increase in terms of both the
1:05:54complexity or Fidelity at least uh of
1:05:57examples that you can create U but then
1:06:00you also do need more skills and
1:06:02resources so that is actually makes it
1:06:05harder for you to access those I put the
1:06:07class 4 which is really about 3D
1:06:10modeling tools here just I wanted to
1:06:12have them in this landscape but I think
1:06:14first of all they do require a very
1:06:16different set of skills and resources
1:06:17and they're really not easy to use if
1:06:20you're more like a web designer or web
1:06:22developer background then you really
1:06:24have a hard time operating these
1:06:26tools and then there are the tools that
1:06:29I talked about A-frame unity and unreal
1:06:32and I would say they are up there
1:06:35because you can actually really achieve
1:06:38good Fidelity in these Tools H but you
1:06:40also do need uh quite a number of skills
1:06:44and you need time to build those skills
1:06:46and time is money so there's also a set
1:06:50of immersive authoring tools out there
1:06:52for example tiltbrush and Google blocks
1:06:55um that I would put down here so they're
1:06:57relatively High Fidelity because you're
1:06:59working directly in VR with these
1:07:01examples and the skills and resources
1:07:03required relative to the other things I
1:07:05talked about are probably not as much
1:07:08but I would say this is very generous of
1:07:10me to say that we can actually reach
1:07:12that level of fality here it's a little
1:07:14bit uh stretching it obviously in my
1:07:17research I really want to be here I want
1:07:20to be able to create the highest
1:07:21Fidelity of solutions with the least
1:07:24skills required so that would really be
1:07:26cool and yes I have worked on some of
1:07:30these tools here and I share some of
1:07:31them with you throughout the Muk but
1:07:34it's not always clear whether I've
1:07:36actually really reached that goal of
1:07:38being in that quadrant probably some of
1:07:40the tools are still a little bit Limited
1:07:43in terms of the Fidelity that I can
1:07:44reach but I do think that I succeeded at
1:07:47really limiting the skills and the
1:07:49resources required but the trade-off is
1:07:52that you cannot Prototype all kinds of
1:07:54experiences so the problem with all of
1:07:57these tools is that they only allow you
1:07:59to do so and so much some of the tools
1:08:03might be overkill for smaller projects
1:08:05where you could just use args to do
1:08:07something here for example uh and that's
1:08:10just like building this awareness of
1:08:13this landscape and being able to do this
1:08:15classification and then picking tools
1:08:17within all these different kinds of
1:08:20quadrants I think is really a key skill
1:08:22that we need to develop as part of this
1:08:24course
1:08:26so in summary the trouble with existing
1:08:28EXA tools that I see is this massive
1:08:31tool landscape it's hard to get started
1:08:33for new designers it's even hard to keep
1:08:35track for experien developers then the
1:08:38problem is that the design process are
1:08:40really unique patchworks every kind of
1:08:42project that I've done required a very
1:08:44specific set of tools there was not a
1:08:46lot of commonality between projects so
1:08:48every ARR app requires a unique tool
1:08:51chain prototyping tools are too limited
1:08:53for most apps that's what I found and
1:08:56then development tools are Out Of Reach
1:08:58for novice non-technical
1:09:00designers there is this significant Gap
1:09:03both within and between the tools so the
1:09:06tool chain is really optimized in the
1:09:08upwards direction as I was just
1:09:10illustrating in the previous graph so it
1:09:13makes it really hard to design
1:09:16iterations especially when you have to
1:09:17go back to a previous tool so let's
1:09:19imagine you prototype something quickly
1:09:21in an immersive authoring tool for AR
1:09:23and for VR and then you've taken it
1:09:25further in unity or in unreal and that's
1:09:29an okay thing to do but if you then feel
1:09:31like oh you want to address some
1:09:33customer feedback you have to really go
1:09:35back to the immersive prototype and
1:09:38there is no easy transition between some
1:09:42of these tools in reality though I think
1:09:45we will be able to work around a lot of
1:09:47these issues just over time we will get
1:09:50more experience by doing more XI
1:09:53projects develop a flavor and a sense
1:09:56for what's the right kind of tool and
1:09:57when to use which tool and I really now
1:10:01feel like in my own work and what I see
1:10:03with my students and that makes me very
1:10:04happy is that they're really building
1:10:05this tool box and they're not trying to
1:10:08do everything with this one tool that
1:10:09I've learned but they're really quite
1:10:11creative and exploring different kinds
1:10:13of tools throughout every project and
1:10:15that's also something I'll invite you to
1:10:17do and you will see more opportunities
1:10:18for this in later parts of this course
1:10:21and also in the other
1:10:23courses and finally I talked about this
1:10:25issue of platform fragmentation before
1:10:27but let me illustrate this slightly
1:10:29differently so when you're doing VR or
1:10:32ar you really think in terms of low end
1:10:36to highend so phone based VR I would put
1:10:39on the lower end because and actually
1:10:42that more or less has died Daydream and
1:10:44cardboard and uh even Oculus go the
1:10:47three Dove the three degrees of freedom
1:10:50head mod displays that's something
1:10:52that's been discontinued in favor of six
1:10:54degrees of freedom mod at displ and what
1:10:57I haven't even drawn out here is like
1:10:59this idea of building these kinds of big
1:11:02caves these virtual reality rooms that
1:11:04usually is not really a target of an app
1:11:08so if you're building a new VR app you
1:11:10really want to think about okay what are
1:11:12the requirements and it used to be
1:11:14thinking about three do versus six do
1:11:17how many controllers do they even have a
1:11:19controller cardboard for example doesn't
1:11:21have a controller and so addressing
1:11:23these issues between these best forms
1:11:25was and is still an issue now it seems
1:11:29to be the trend that phone based and
1:11:32three do will go away and six Dove so
1:11:35the more advanced headsets they will
1:11:37remain and this solves part of the issue
1:11:40at least on the VR side so when it comes
1:11:43to AR and on the lower end of AR I see
1:11:46smartphone based AR so not the headsets
1:11:49and when it comes to smartphone based AR
1:11:50the question is really should we require
1:11:53the user to use any kind of marker and
1:11:56could we package this nicely with the
1:11:58app for example I've seen cool brochures
1:12:01or business cards that function as
1:12:03markers and then it actually seems like
1:12:05a cool user experience you could also uh
1:12:08design them quite differently and
1:12:09actually place them nicely in the
1:12:11environment but I think there's this
1:12:13trend towards maress AR just because we
1:12:15have the more sophisticated smartphones
1:12:18and it will become a standard in the
1:12:20next few years and so when we look at
1:12:22the more higher end of the spectrum so
1:12:25we thinking about headsets so AR
1:12:27headsets for example like the Hollens
1:12:29which I've shown you in earlier segments
1:12:31of the course and you know that is
1:12:33really something that we will probably
1:12:35design more for in the future I think
1:12:37ultimately this is where most of the AR
1:12:40takes place but right now when you're
1:12:42creating a new AR app you really have to
1:12:44decide whether you're going to support
1:12:45smartphone based AR and then marker
1:12:47based or Mar or whether you're targeting
1:12:50the headsets and these headsets are
1:12:51still quite expensive so you actually
1:12:53really don't reach too many users but
1:12:56you can do the really cool examples and
1:12:58so this platform fragmentation really is
1:13:01an issue at the moment and it's quite
1:13:04difficult to navigate around it and the
1:13:06course that is more focused on
1:13:07development will really expand on all
1:13:09these issues and cover them in more
1:13:11detail so now I've talked through all
1:13:15the key barriers and I've provided uh
1:13:18quite a few examples and maybe new ways
1:13:20for you to think about it um I wanted to
1:13:23end with a few concrete tips so here's
1:13:26my advice for you to get
1:13:29started first get your hands dirty uh
1:13:32the dirtier the better I say here so
1:13:34really get your hands on an XL device
1:13:37doesn't have to be the most expensive
1:13:39could also be repurposing your
1:13:41smartphone as an AR or VR device with
1:13:44some of the smartphone reppers like
1:13:46cardboard or if you still have access to
1:13:49Google Daydream or something close to it
1:13:52like Samsung Gear VR so you could try
1:13:54out those things and it's really
1:13:56important even if they're used and old
1:13:58it's really important for you to
1:13:59experience it for
1:14:01yourself and then associated with this
1:14:03you should also try out some of the apps
1:14:05and then really think about tools as
1:14:07you're trying to enter this space of
1:14:09creating XR
1:14:12experiences then you should really start
1:14:14making your first XR app you should set
1:14:16yourself a small reachable goal though
1:14:20it's okay if you want to do a Todo or
1:14:22some kind of reminder app or AR some
1:14:26kind of measurement app although we have
1:14:28a thousand of those apps out there I do
1:14:30think that it's a good way for us to get
1:14:33started and the measure app for example
1:14:36is a good way for you to learn about AR
1:14:38and some of the principles there and so
1:14:41it's important to really do your own
1:14:43first app not just do tutorials or read
1:14:46blog articles really make something I
1:14:48think that's the best way to get started
1:14:50but remember set yourself a small
1:14:53goal third when you're there and you
1:14:56feel like wow I'm really getting
1:14:57passionate about AR and VR then really
1:15:00think about um how do you connect to
1:15:02other developers learn about the exra
1:15:04communities out there that's really
1:15:06where a lot of the knowhow is so start
1:15:09following arvr blogs I usually follow a
1:15:12lot of people on Twitter that's for me
1:15:14it's a good way to learn about what's
1:15:15currently being done uh all the vendors
1:15:18and associate with those vendors are
1:15:20actually people working at those vendors
1:15:22and many of them are actually really
1:15:23quite open about what they're working on
1:15:25and so social media could be a very good
1:15:27way for you to keep up to date as
1:15:29well and then number four would be start
1:15:32reading more about XR research and
1:15:35design so when I say start reading about
1:15:38it there's a lot of practitioners
1:15:40sharing their knowledge and experience
1:15:42again in blogs there are people out
1:15:44there that write more like white papers
1:15:46doing some kind of market analysis that
1:15:48might be helpful for you when you want
1:15:50to figure out what kinds of platforms
1:15:52you want to Target but I really like the
1:15:55articles created by practitioners
1:15:56there's a lot of good blog articles out
1:15:58there which I also share with you
1:16:00throughout the course and then if you
1:16:03are inclined to reading more academic
1:16:05papers and now the scientific Community
1:16:07is not always the most accessible but I
1:16:10personally obviously get a lot out of
1:16:12the latest The Cutting Edge the bleeding
1:16:14edge uh research so I really learn a lot
1:16:16from academic papers so this is my how
1:16:19to get started get your hands dirty make
1:16:21your first XR app make sure it's your
1:16:23own thing join XR communities and then
1:16:26start reading about some of the XR
1:16:28research and design yes so this was my
1:16:31how to get started and it's obviously
1:16:33just one way to get started um but in my
1:16:35experience this works very well I've
1:16:37worked with many students over the years
1:16:39and there is a little bit of Truth in
1:16:40some of the things I said um so maybe
1:16:43try it out for yourself and we continue
1:16:47working in this course and learning that
1:16:49way because I think we'll have a very
1:16:51good way for you to get started and then
1:16:53take it further in your own time
1:16:55after this if you turn out to be an XR
1:17:12Creator H when I say Exile design so I
1:17:14really think of three big things we need
1:17:16to touch on we need to talk about the
1:17:18design process the methods and tools and
1:17:20then also guidelines and evaluation and
1:17:23this is how I've started structure the
1:17:25course so design process is really the
1:17:27steps that are important uh to make it
1:17:30uh through a project and then come up
1:17:33with a good design uh so along the way
1:17:37I'm going to introduce you to a variety
1:17:39of methods and tools we going to build a
1:17:40toolbx and so the idea is that you have
1:17:43access to a number of techniques and can
1:17:45choose the ones that you are most
1:17:47comfortable with or that you think best
1:17:48fit the project or based on my
1:17:50experience are very good in early stages
1:17:53of design and then they kind of Fade Out
1:17:55and transition out uh and then we have
1:17:58access to other techniques uh targeted
1:18:01at digital prototyping and then later
1:18:03development uh as well so we learn about
1:18:06these techniques and we build our
1:18:07toolbox and then we also learn about
1:18:09design guidelines and best practices and
1:18:12design patterns and so we're building
1:18:14this understanding of what is good and
1:18:17what is maybe not so good in XR design
1:18:19but although I must say that this is
1:18:21still evolving and we are still learning
1:18:23a lot along the way as well so well I
1:18:26think that the the fundamental uh
1:18:29aspects are covered in this course uh I
1:18:32might already be working on some new
1:18:33techniques in my research and then I
1:18:35hope to share them with you as well as
1:18:38we go so um let's talk about process for
1:18:41a second when I say design process
1:18:44really we're going to start from a
1:18:47problem statement usually a competitive
1:18:49analysis and Designs based analysis so
1:18:52these are really the key things help us
1:18:55frame the problem really understand it
1:18:57better and then understand the
1:18:59competition so therefore basically the
1:19:02state-ofthe-art so what's been done and
1:19:04how well is it done what are the factors
1:19:07that contribute to good Solutions and
1:19:08what are the gaps there and really
1:19:11trying to find a way into providing a
1:19:13meaningful solution to the problem so
1:19:15both understanding the problem but and
1:19:16also really identifying gaps and finding
1:19:19ways of uh fitting those gaps you will
1:19:22have a variety of options to consider so
1:19:24we going to talk about design space
1:19:25analysis asking the right questions
1:19:28weighing the options that you have
1:19:30coming up with criteria that allow you
1:19:32to make good decisions and so we're
1:19:33going to talk about that and then we're
1:19:35going to enter the space of prototyping
1:19:38uh and there I already emphasized the
1:19:40physical um so I really I feel like
1:19:43paper prototyping is really where I
1:19:46usually learn a lot and my students have
1:19:48a lot of fun so maybe that is something
1:19:50that you'll also enjoy I introduce these
1:19:51two techniques using paper transparency
1:19:54playdoh and really making use of
1:19:56physical space kind of like tweeting it
1:19:58like a stage to Think Through the XR
1:20:01experience way before we start uh
1:20:03developing anything and using any
1:20:05digital tools we can really do a lot
1:20:07physically including testing with users
1:20:09and that I think is a very powerful set
1:20:11of techniques that I'll introduce you uh
1:20:14to there and then we're going to talk
1:20:16about digital prototyping as well so
1:20:19here I said we're going to do both
1:20:20digital and immersive so digital on a
1:20:22desktop with some more traditional tools
1:20:24tools that you may be familiar with as a
1:20:26graphic or web or mobile designer and
1:20:29there are some new versions of these
1:20:30tools available to us that are focused
1:20:33more on arvr design and this is still a
1:20:35space that is rapidly evolving lots of
1:20:37issues with current tools as well and
1:20:39then we're going to explore immersive
1:20:41prototyping really this idea of jumping
1:20:44directly into the headset or using an AR
1:20:46mobile device and prototyping um and and
1:20:50experiencing it as at the same time so
1:20:52this idea of concurrent design and
1:20:54testing which is really the definition
1:20:56of immersive prototyping and so I think
1:20:59that's going to be fun and so with all
1:21:01these prototypes you're going to arrive
1:21:03at what is called the minimum viable
1:21:05product think of it as a proof of
1:21:08concept like the minimum set of features
1:21:10put together in an early version of our
1:21:13let's call it product and uh that allows
1:21:16us to learn and try it out with users
1:21:19and learn about initial user experiences
1:21:22and how we can improve it and so this
1:21:24MVP this proof of concept implementation
1:21:26of our well of our XI experience that is
1:21:30really uh what this whole course is uh
1:21:33targeted at the course kind of ends um
1:21:37at the end of this MVP um which this MVP
1:21:40could really function as a design
1:21:41specification that you can then hand off
1:21:43to developers or if you're more
1:21:45technically inclined could start
1:21:46implementing with current programming
1:21:49tools and that's something we'll cover
1:21:51in a separate course so this is the
1:21:54process oriented way of looking at Exile
1:21:56design but if you think more about the
1:21:58concerns and what are the key questions
1:22:00so here I'm going to give you an intro
1:22:01to Exile design in a slightly different
1:22:03way we're going to think about ways of
1:22:06building empathy with your users so this
1:22:08is really at the core of design thinking
1:22:11we're going to talk about ethical and
1:22:13responsible design now that's going to
1:22:15be quite a tough lecture to give and one
1:22:18that I'm never really happy with but I
1:22:20really wanted to give you the right
1:22:22mindset to think about good design but
1:22:25then also ethically responsible design
1:22:29and we're going to talk about design
1:22:30guidelines so best practices and
1:22:31patterns of design that's something that
1:22:34I think is always evolving but it's
1:22:36really something that as a designer you
1:22:37really want to be aware of these Design
1:22:40Systems out there and really what are
1:22:42the right ways what's the current
1:22:44state-ofthe-art when it comes to XL
1:22:46design you want to be modern and you
1:22:47want to be consistent with other kinds
1:22:49of XR experiences that users may have
1:22:51tried and then we're going to talk about
1:22:54design critiques now design critiques
1:22:55that's something that is heavily
1:22:57practiced in some disciplines like
1:22:59architecture and architectural designs
1:23:01and I also do this in my uh design uh
1:23:04courses um it's really important to
1:23:07learn how to give good feedback and to
1:23:11you know critically analyze a product
1:23:13and then make constructive suggestions
1:23:16for improvement so we're going to
1:23:17practice this we're going to do design
1:23:19critiques in two different ways one that
1:23:21is more oriented towards uh the guidelin
1:23:25that I talked about and then one that is
1:23:27really focused on ethics because I do
1:23:29think that when it comes to XR we really
1:23:31need to take care of both of these uh
1:23:33big concerns so making sure that the
1:23:36current design kind of fits with common
1:23:38design practice and it's also really
1:23:41ethically responsible and finally I'm
1:23:44going to introduce you to uh something
1:23:45that I really like to do in my research
1:23:48and that I think is actually a very good
1:23:50way to bring together uh people that
1:23:53care about the same problem and we're
1:23:55going to talk about design gems and the
1:23:57way I think about design gems is like a
1:23:59three-hour block with a design prompt
1:24:01and guidance and we're bringing together
1:24:04basically Target users or stakeholders
1:24:08if you will that really care about the
1:24:10problem and we're going to try to solve
1:24:12it collaboratively and so you can bring
1:24:14in I often like to bring in students
1:24:16with varying level of experience in
1:24:19terms of like where they are and their
1:24:20degree but also then a variety of
1:24:22different backgrounds and so that's one
1:24:24of the things I really like working in
1:24:25the school of information because that's
1:24:27where we have access to this diverse
1:24:29pool of students and then we're also
1:24:32going to talk about design ethics and
1:24:34with design ethics we really want to get
1:24:36away from this basic usability driven
1:24:39question what's the most effective and
1:24:42efficient design of my solution and we
1:24:45want to think about what harm could it
1:24:47do how could it be misused so kind of
1:24:51like a little bit the Dark Side of
1:24:53design and did I accidentally create a
1:24:56weapon that is something that you should
1:24:58really be thinking about so with the
1:25:00design options that we have available
1:25:02and the choices that we make are they
1:25:05meaningful and are they ethical and are
1:25:08they responsible so that's something uh
1:25:11that I want you to think about so we I'm
1:25:12going to introduce you to key ways of
1:25:15thinking about it this idea of you know
1:25:16giving control to users autonomy
1:25:19transparency making them feel in control
1:25:21and really being aware of all the
1:25:23options that they have and then safety
1:25:26and I mean both the physical and the
1:25:28mental uh safety of our users so that is
1:25:31something we have to consider and design
1:25:34ethics is going to be a key topic we're
1:25:36going to talk more about design
1:25:38guidelines we're going to learn about
1:25:39those uh that are platform driven by
1:25:41vendors and then those that are done by
1:25:44designers and they are user oriented so
1:25:46usually done in some kind of user
1:25:48studies maybe not as thorough as some
1:25:52researchers might do it but uh still
1:25:54very good way to learn about design I
1:25:57really like those coming from experience
1:25:59so we've tried this and we maybe we
1:26:00didn't do it super rigorously but we
1:26:03have made this effort and we have
1:26:06learned from it so here are the Lessons
1:26:08Learned experience-based uh by
1:26:11practitioners and then obviously as a
1:26:13researcher you kind of like want to do
1:26:15this with scientific rigor and so there
1:26:17are different kinds of concerns for
1:26:20example the number of users that you
1:26:21involve in your studies is a bit
1:26:23different when it comes to research when
1:26:24it comes to like more product oriented
1:26:26or practition oriented user studies I
1:26:28think you can really work with just a
1:26:31few users and you will already find a
1:26:33lot of issues as a researcher however
1:26:36you're interested in statistical power
1:26:37and can we actually really learn can we
1:26:39make meaningful conclusions from what we
1:26:42have observed and so that's a little bit
1:26:44of a different
1:26:45approach and then we're going to learn
1:26:47about design critiques uh design
1:26:49critiques I'm going to introduce you to
1:26:51this method which I really like uh also
1:26:53coming out of of Stanford it's the I
1:26:55like I wish what if method and so in the
1:26:59I like is really this idea of starting
1:27:01with something positive you you're
1:27:03looking at a design and there must be
1:27:05something that you like you just have to
1:27:07say it and you that's a good way of also
1:27:09building empathy with designers which is
1:27:12very important and then you can start
1:27:15not attacking but critiquing and do this
1:27:17constructively so we're going to turn
1:27:18into this I wish something were
1:27:20different and you need to be well
1:27:23constructive and you need to be concrete
1:27:25and the the way I'd like you to think
1:27:27about it is you really want to make
1:27:29helpful suggestions so what if you
1:27:31consider uh this as opposed to placing
1:27:34this menu into the world what if it were
1:27:37just on device oh that's not cool that's
1:27:40not how we do things in a okay fine but
1:27:43it might be more usable or it might be
1:27:45easier for novice users and so you're
1:27:48going to have interesting debates and I
1:27:50think this idea of uh critiquing
1:27:53existing interfaces and learning from
1:27:55them also helping your friend or your
1:27:57partner or your design team I think is
1:27:59very good so one thing that I like about
1:28:01being a professor for example is that I
1:28:03teach others and while I teach others I
1:28:05also actually learn a lot more about the
1:28:08things I actually don't know or don't
1:28:10understand so well and as I try to
1:28:12explain them and sometimes I have to
1:28:14look up things just like everyone else
1:28:16and then as I explain them um I actually
1:28:19really learn more about it myself and um
1:28:22it's the same with design critiques okay
1:28:25so you can look at an existing design
1:28:27and for me it's very easy to find issues
1:28:29in existing designs and and and and
1:28:32that's a first skill but then the hard
1:28:34part is really to to make constructive
1:28:37uh suggestions for improvement and
1:28:39that's something we have to practice uh
1:28:41and that's something that I usually do
1:28:42in my courses throughout so like every
1:28:44week we're going to look at existing
1:28:46interfaces and critique them and uh
1:28:49learn about like common practices of
1:28:50design and what's good and what's bad
1:28:53and how this and how users might find it
1:28:55so this is a really important skill
1:28:58we're also going to talk about
1:28:59evaluation and so when it comes to
1:29:02evaluation a lot of at least researchers
1:29:04immediately think about usability and
1:29:06somehow proving that our design is a
1:29:09good design and that users found it
1:29:11usable and they're still very efficient
1:29:14uh it was an effective way of using it
1:29:15that maybe they had satisfaction as they
1:29:18using it and maybe even some fun but I
1:29:20also wanted to think about usefulness I
1:29:23can design the
1:29:24best to-do list in AR it's the most
1:29:27usable but is it useful do our users
1:29:31find it helpful do they see a purpose in
1:29:33it does it improve their daily lives and
1:29:36I feel like for a lot of the XR
1:29:38experiences out there this question of
1:29:41usefulness really raises the bar and we
1:29:44don't need a million measure apps in AR
1:29:47right I mean it's it's a cool little
1:29:49thing to implement if you're getting
1:29:51started with XR so might be that you
1:29:53want want to try this for yourself but I
1:29:55don't think we need a million Solutions
1:29:57in that space because the problem is
1:29:59really not that big so in addition to
1:30:02usability and usefulness we also want to
1:30:05really consider utility does our
1:30:08solution really solve the users problems
1:30:11and uh that is something that is really
1:30:13hard to assess often it has to do with
1:30:16long-term usage and that brings me to
1:30:18the fourth aspect of evaluation user
1:30:20acceptance will our users actually
1:30:22continue to use it or they just using it
1:30:24for the purposes of our study for
1:30:26example in research this is really one
1:30:29of the hardest questions to answer when
1:30:30you have a product and it rolls out and
1:30:32people continue to use it you might
1:30:34argue that users accept it and that when
1:30:37they accept it it's like the ultimate
1:30:39gril because they may see great utility
1:30:42and usefulness in it and it should
1:30:44probably also be usable now there have
1:30:46been Solutions in the history of
1:30:48interface design where people just
1:30:50didn't have a better options so it was
1:30:52great utility great useful but just not
1:30:54really usable so overall we're going to
1:30:57cover a right variety of methods and
1:30:59tools in this course uh we're going to
1:31:01learn more about ways of need finding
1:31:03and brainstorming so this is really in
1:31:05the design thinking category we're going
1:31:07to learn about low fality and high
1:31:09fality prototyping so rapid prototyping
1:31:12techniques and how to transition from
1:31:13the physical to the digital and then
1:31:16we're going to do digital and immersive
1:31:17authoring and really learning all the
1:31:19methods and tools there and really also
1:31:21understanding like between each of these
1:31:24main steps what do we actually learn
1:31:26about our design as we go obviously we
1:31:29are increasing the Fidelity or the
1:31:31quality if you will and each of the
1:31:34methods that I introduced you to they
1:31:36target different kind of feedback and
1:31:38that's something we will also think
1:31:39about throughout and then normally you
1:31:42would also go and develop with let's
1:31:45call them programming tools and that are
1:31:47introduced you to in a separate course
1:31:50and we would also look at building
1:31:52techniques into the inter faes so that
1:31:54we can learn about how users use our
1:31:56interfaces and so building a form of
1:31:58analytics and continuous kind of like
1:32:01user feedback and that's something we
1:32:03also talk about but that is not in the
1:32:06scope of this course it's just like
1:32:07important to keep that in mind and will
1:32:09be the subject of the third course in
1:32:12this exam Muk Series so I just said low
1:32:14Fidelity and High Fidelity prototyping
1:32:16and we're really going to learn about a
1:32:18variety of methods we're going to start
1:32:19with storyboarding and wireframing we're
1:32:22going to go and enter the physical
1:32:24prototyping space really this is where I
1:32:26usually have a lot of fun as I said so
1:32:28we're going to use paper transparency
1:32:30cardboard Play-Doh and in fact you're
1:32:33going to use these materials to think
1:32:35through an XR experience without
1:32:36implementing it and then we're going to
1:32:38take those things and we're going to
1:32:39iterate over and over again and then
1:32:41we're also changing the medium and going
1:32:43digital and I think there's a lot of
1:32:46value in taking the designs further this
1:32:48way and while I often as a researcher
1:32:51feel like hey there must be a cool way
1:32:53to quickly transition from physical to
1:32:56digital and in fact I've worked on this
1:32:57in some of my research I do think
1:32:59there's also a lot of value uh in taking
1:33:02a design that we have created physically
1:33:05and then reimplementing it and improving
1:33:07as we go with digital tools and we're
1:33:10going to also talk about immersive
1:33:12prototyping I just wanted to say that
1:33:13again because I'm very excited about
1:33:15that and the idea of this course is
1:33:18really to emphasize prototyping and tell
1:33:21you more about how significant this is
1:33:23so don't just jump into the programming
1:33:25tools out there I know that's also very
1:33:27important and I know everybody wants to
1:33:29be good with unity and unreal or WebEx
1:33:31are and I'll really also talk about
1:33:34those platforms but I've dedicated the
1:33:36third course to that this course is
1:33:39really a rapid way of thinking through
1:33:42XR experience and and if you do this the
1:33:44right way you may actually save a lot of
1:33:46time and be much more efficient when it
1:33:49comes to working digitally and
1:33:51programming so what is the significance
1:33:53of protot in well I think first of all
1:33:55it allows you to avoid premature
1:33:57commitment a lot of us really like AR or
1:33:59VR and often it's like one or the other
1:34:02although I'm relatively agnostic maybe
1:34:04you have a preference for VR because you
1:34:06that's the first thing you saw and you
1:34:07think it's so much cooler and now you
1:34:10you're you're looking at VR as a hammer
1:34:12and you're trying to find all the nails
1:34:14that you can hit with it and we want to
1:34:16avoid that because we really want to
1:34:18look at VR and critically understand
1:34:20whether it actually is a solution to our
1:34:22problem and in fact there are many
1:34:24problems out there where neither AR or
1:34:26VR is actually a solution and so we want
1:34:29to avoid that and we can figure that out
1:34:31without having to implement everything
1:34:33we can just prototype it with physical
1:34:36means as I said and then we can uh come
1:34:39to that conclusion much faster and
1:34:41cheaper prototyping also tells you a lot
1:34:44more about the context so figuring out
1:34:46the environment in which an experience
1:34:47takes space and how much you have to
1:34:50design it for VR and how you could
1:34:51change it in AR and this story really
1:34:54what's the story uh what's a good way to
1:34:56communicate that story how do users
1:34:59understand how they can operate the
1:35:00interface without you telling them every
1:35:03step along the way what's good
1:35:04environmental design what are user needs
1:35:07and uh what are the required skills and
1:35:09abilities of our users and all these
1:35:11things you will actually find out more
1:35:14or less implicitly by prototyping things
1:35:17and trying them out with users and then
1:35:20prototyping also gives a lot of answers
1:35:22so along the way you'll find out about
1:35:23interactions and input modalities and
1:35:26what kinds of sensing requirements you
1:35:28will have in terms of like do we need
1:35:30hand tracking for example ey tracking or
1:35:32can we go without it display
1:35:35requirements so should we think about
1:35:37something mounted on the user's head or
1:35:39is it fine if it's handheld or should it
1:35:41be placed somewhere else in the
1:35:43environment as a large display we're
1:35:45going to actually create a lot of assets
1:35:47along the way so that's why I really
1:35:49like the physical and the digital
1:35:51prototyping because you're producing a
1:35:53lot lot of knowledge and in addition to
1:35:55that knowledge also initial assets and
1:35:57in fact I've really seen some really
1:35:58cool prototypes where we were kind of
1:36:01like holding on to these paper sketches
1:36:04and basically creating VR experiences
1:36:06around paper assets so one of my
1:36:08favorite examples here is the Star Trek
1:36:10that I'll show you later that I've
1:36:12created with one of my students Katie
1:36:13and she's a really good designer and and
1:36:15and she can really sketch very well and
1:36:17so we actually produced an entire VR or
1:36:21ar experience just using paper and a
1:36:24specific template so creating a lot of
1:36:26assets there and then finally again the
1:36:28prototypes allow us to ask about the
1:36:31user needs and then also what are the
1:36:33skills and the prototypes then also give
1:36:35us answers to these questions so that's
1:36:37why I think prototyping is very very
1:36:39important cannot emphasize it enough and
1:36:42that is really one of the key messages
1:36:44of this course and hopefully something
1:36:47that you'll take away with you and
1:36:49incorporate into your own practice so I
1:36:53really look forward to uh this course
1:36:55and introducing to all these kinds of
1:36:57things I hope I haven't missed anything
1:36:59you know I've been thinking a lot about
1:37:01what are the key things that I need to
1:37:03convey and how much time do I have and
1:37:06um I've been playing around with topics
1:37:09and things I want to you know
1:37:10incorporate and so if you feel like
1:37:12something is missing H or something uh I
1:37:15should have talked more about I'm always
1:37:17also happy to hear your feedback on that
1:37:20and I could imagine future versions of
1:37:22this course um as as well so um thanks
1:37:25for taking the course and I really hope
1:37:27that you'll learn a lot and that you
1:37:29actually also have some fun so I'll see
1:37:32you in later segments of this
1:37:47course so we've covered a lot of the dos
1:37:49and don'ts uh based on my experience and
1:37:52that experience is coming from doing a
1:37:54lot of research in that space and then
1:37:56also teaching a lot of students about XR
1:37:59design and obviously operating a lot of
1:38:01arvr applications myself but um while I
1:38:05strongly believe that these are good
1:38:06guidelines um it's just a starting point
1:38:09uh just based on my experience and maybe
1:38:12there are people out there that disagree
1:38:14but overall as a starting point I think
1:38:16you can apply those guidelines look at
1:38:19your designs and then look at these
1:38:21guidelines as a checklist and see how
1:38:23many dos you have done and how many
1:38:25don'ts you have done and then think
1:38:27about ways of avoiding the don'ts um
1:38:30which I think might be a good way to
1:38:32improve
1:38:33design so still thinking about design
1:38:35guidelines I wanted to look at an
1:38:37example and I chose a recent experience
1:38:40that I had when I was attending the i e
1:38:43virtual reality conference in
1:38:452020 H which was supposed to happen in
1:38:48Atlanta but due to covid-19 had to move
1:38:51to remote and online we're going to look
1:38:53at some of their spaces that they have
1:38:56created some of their virtual
1:38:57environments that they have created and
1:39:00I have invited my uh student here scha
1:39:02who is going to join me in some of these
1:39:05discussions and Explorations as well and
1:39:07a couple of the things we're going to do
1:39:08is we're going to climb some stairs and
1:39:11which I think leads to interesting
1:39:13issues um we're going to explore the
1:39:16world going for a walk together and then
1:39:19you will also see me picking avatars and
1:39:22uh looking at a lot of text and then
1:39:24we're going to review some of the design
1:39:25decisions and I'm going to compile a few
1:39:28more uh tips and maybe guidelines if you
1:39:31will to uh also avoid those things in
1:39:34the
1:39:35future so here I'm looking down the
1:39:37venue this is the stage that they have
1:39:40and I'm waiting for my student sh ah
1:39:42there she is and I see her and I can see
1:39:46how she is essentially trying to make
1:39:49her way up so they have designed this uh
1:39:51stage so that you start in the back
1:39:54where you have a little bit of of
1:39:56privacy in some sense you don't hear the
1:39:58speakers and can have a chat and then
1:40:01others won't hear you in that area if
1:40:02they up here on that stage so that's
1:40:03good but um you then actually have to
1:40:06climb those stairs and these stairs are
1:40:08really not accessible no matter whether
1:40:10using a keyboard and mouse like I do
1:40:13here in this example or a VR headset
1:40:15which I'll do in some of the other
1:40:17examples in the future so stairs really
1:40:20is an interesting issue to me me and
1:40:24something we should think about more and
1:40:25potentially avoid as much as
1:40:28possible and so then I was thinking okay
1:40:32um let's explore a little bit more uh of
1:40:34this venue and I decided we were
1:40:37attending this conference for a while
1:40:38and then I said to sha hey why don't we
1:40:41go for a walk and explore some of their
1:40:44bigger virtual spaces and so we ended up
1:40:47on this Georgia Tech campus now I
1:40:50haven't actually been to the Georgia
1:40:52Tech campus so I don't know how much it
1:40:54really resembles reality but I could
1:40:57imagine that unless they had a major
1:41:00investment um it probably doesn't look
1:41:02like this so in many ways they
1:41:04reinvented reality here uh with and that
1:41:08makes sense I mean I guess I mean the
1:41:10Big B and and some of these virtual
1:41:12screens floating around so it's an
1:41:13interesting mix of like this is a selfie
1:41:16from Blair one of the organizers and a
1:41:18friend who I hope doesn't mind that I'll
1:41:21critique and review this a little bit
1:41:23and to look at our guidelines and which
1:41:25of these can apply and then we were
1:41:28going for a walk and there were some
1:41:31issues again it's a ramp here it's not
1:41:33stairs but um I could teleport uh with
1:41:36my mouse pretty easily and then uh scha
1:41:40was uh using a Macbook and didn't have
1:41:44right click so she couldn't actually
1:41:46navigate that space uh with mouse and
1:41:49teleport so I had to walk all the way
1:41:52and um for a novice user Trader is not a
1:41:55novice user but for somebody who
1:41:57explores uh such a world and VR for
1:41:59their first time uh figuring out how to
1:42:03move in a space when it's unclear and
1:42:05that you have this option potentially of
1:42:07teleporting if it's available that is
1:42:10actually really quite uh an important
1:42:12step uh and obviously the more
1:42:15experience you have in these spaces the
1:42:17the more of that experience translates
1:42:20uh between spaces and experiences and so
1:42:22maybe it won't be so much of an issue
1:42:24later but um onboarding is really quite
1:42:28difficult still and and then this is
1:42:31really it's an interesting um example
1:42:33because I really don't get a good sense
1:42:36of what the Georgia Tech campus looks
1:42:37like uh apart from a few Billboards and
1:42:41um text here and um now that text is
1:42:46okay the way it's placed is actually
1:42:49quite well done it's on a billboard it
1:42:51has sufficient contrast it's not just
1:42:54floating I'll have another example later
1:42:56that I think is pretty bad um then we
1:42:59were walking around a little bit more
1:43:01exploring some of the other spaces uh
1:43:03ending up at a campfire which was really
1:43:05quite an interesting atmosphere um the
1:43:07way you navigate those spaces uh was
1:43:10interesting so the one of these tiles
1:43:12was actually a portal into a different
1:43:15world um so there's a little bit of
1:43:18inconsistency here in terms of the
1:43:20content that you see on the wall some of
1:43:22it is actually video some of it is text
1:43:24and some of this can be played I guess
1:43:26or not so it's really quite a lot of
1:43:29stuff going on and then as I said one of
1:43:31those tiles actually functions as a
1:43:33portal into different spaces so uh I do
1:43:36like that um they illustrate a lot of
1:43:40the content here and some of the hard
1:43:42work that really went behind uh creating
1:43:45this uh experience in this online
1:43:48conference essentially so why did I pick
1:43:50this example cuz I think it really allow
1:43:53us to review some of the guidelines this
1:43:55idea of not really copy pasting reality
1:43:59uh offering alternate alternative ways
1:44:01of navigating making use of established
1:44:04interface metaphors and limiting the
1:44:07amount of explicit interaction and I was
1:44:09really stuck on the stairs example and
1:44:13uh the platform uh that I was talking
1:44:16about just because it while it was good
1:44:19in terms of physical affordances I
1:44:22understood that can I can get to a
1:44:24different part of this virtual world and
1:44:26I can navigate up and down or further
1:44:29away it really reintroduced all these
1:44:32know accessibility issues from the
1:44:34physical world and that's just like a
1:44:36big uh problem in an experience like
1:44:39this so when you go to a conference like
1:44:41this you really want to learn what kinds
1:44:43of things are actually available what
1:44:45sessions are on and um also here they
1:44:49kind of replicated The Real World um
1:44:53which usually you have some kind of
1:44:55handout or some kind of boards and
1:44:57somebody from the community created this
1:44:59room essentially putting these um
1:45:03program information sheets into into the
1:45:06virtual world here but it's really a lot
1:45:09of text and hard to navigate to the
1:45:11extent that I'm like almost bumping into
1:45:13it and um it really wasn't that helpful
1:45:18after all the main purpose was not to
1:45:20have to leave the virtual reality
1:45:23experience it was a quick fix uh because
1:45:25a lot of people felt like oh always have
1:45:27to jump back to the virtual reality
1:45:29website and uh but I just want this
1:45:32information in VR so transitioning in
1:45:34and out of VR is still an issue and it's
1:45:37very disruptive to the user experience
1:45:39so I get it but the way we deal with
1:45:42text here is is not
1:45:44ideal so why do I say this I want to
1:45:46show you two examples uh one is in VR
1:45:49and one is in AR and we're going to talk
1:45:51about the format and the layout of the
1:45:52text and I'm just using some standard
1:45:54text with as little format and layout as
1:45:56possible and then I'll show you the
1:45:59different experiences in in VR and
1:46:01AR so my first example is laurum ipsum
1:46:06placeholder text uh in VR and I just
1:46:09really placed it into the scene and you
1:46:10can see it in steroscopic view here and
1:46:14it's really hard for me to actually view
1:46:16it uh at at any one time and then this
1:46:20idea of using your hands and fingers as
1:46:23reading aid uh only partially works with
1:46:26this really big VR controller here now
1:46:29that may improve a little bit of hand
1:46:30tracking but um it's still kind of an
1:46:34issue so here I'm showing you an
1:46:36augmented reality example where I
1:46:38inserted the same text in the studio and
1:46:41there happens to be this light source
1:46:42that I obviously need to liit the scene
1:46:44and make sure that you can see me on
1:46:46camera but um and this could be just the
1:46:49sun in the in in in the real world it
1:46:51really introduces a lot of issues with
1:46:53reading this text and just seeing it so
1:46:56really you need to pay attention to the
1:46:58environment both the virtual and the
1:47:00physical um the distance of the text is
1:47:03kind of okay I put it at 1 M away um it
1:47:07shouldn't be further away than 2 m it
1:47:10what is what people say the scale is
1:47:12still a little big uh for my taste but
1:47:15if I'm scale it down it really
1:47:17introduces new issues so text in general
1:47:20there is no really good solution if need
1:47:23to be adaptive I guess and know a lot
1:47:25about the user and maybe their
1:47:27prescription and all kinds of things so
1:47:29just try to avoid it and again reading
1:47:31aids here like using my finger uh my
1:47:34finger appears behind the screen just
1:47:36because we don't actually in this
1:47:37example have finger based or hand
1:47:40occlusion and finger based occlusion and
1:47:43now we're getting there and improving
1:47:44some of this but it's not available on
1:47:46all platforms and even then I don't
1:47:48think this part of improving the
1:47:51technology would actually solve the
1:47:52issue
1:47:53so my last example is avatars so let's
1:47:56think about the design space that we
1:47:58have in terms of avatars I show two
1:48:00examples here uh one is with permission
1:48:02from my friend Jeremy Nelson um who
1:48:05appears uh in his kind of like
1:48:07reconstructed Avatar all he did was
1:48:09upload a photo to spatial and meeting
1:48:12kind of software so that's what you see
1:48:14on the right and then on the left you
1:48:16see some of the avatars that are used
1:48:19for example in Mozilla hubs and and
1:48:23those are the ones that I chose from
1:48:25when I was actually going to the I EVR
1:48:27conference so when you compare those
1:48:30types of Avatar you definitely see that
1:48:32the ones on the left are more abstract
1:48:34but appear to be more consistent and the
1:48:36one on the right and more realistic but
1:48:39there were some issues with the scale
1:48:41and the position of Jeremy so he
1:48:43appeared in my desk for example and I
1:48:46always look down on him which normally
1:48:49isn't the case we are almost the same
1:48:51height so it was was a little disturbing
1:48:54and confusing now these things will
1:48:56probably be improved in the future but
1:48:59sometimes the more abstract avatars are
1:49:02actually just good enough and we don't
1:49:04necessarily need to see and need to
1:49:06recreate the whole reality just because
1:49:09we have all these tracking issues which
1:49:11are then even worse um the closer you
1:49:14are uh into to reality the higher your
1:49:17expectations are and the more disruptive
1:49:20the user experience is if something goes
1:49:22wrong wrong you know if this little
1:49:24avatar on the left doesn't move properly
1:49:27that's weird but it's okay but if
1:49:29Jeremy's arms or legs Bend in the weird
1:49:32way you don't want to imagine that so
1:49:34here I'm going to show an example of
1:49:36where I chose my avatar and wasn't happy
1:49:38initially I was going to this conference
1:49:40it was a little funny because I did
1:49:42spend quite some time on my outfit and
1:49:46choosing there was one particular
1:49:48character I was looking for uh it was
1:49:50kind of like uh this Iron Man look
1:49:53metal guy and I think I found him on the
1:49:55last page but this was probably the
1:49:57closest or it wasn't available that was
1:50:00also disappointing that an avatar that I
1:50:03used to use was not available in this
1:50:05specific experience and so it really in
1:50:08terms of consistency was a little
1:50:11disruptive so these were some of the
1:50:13examples uh that I put together for you
1:50:16to illustrate some of the guidelines I
1:50:18didn't have an example for each and
1:50:20every guideline but take a look at them
1:50:24explore them a little bit more detail
1:50:26and again as I said use them as a
1:50:28checklist to understand and identify
1:50:32opportunities for improvement to your XR
1:50:34designs and again treat them as a
1:50:36starting point uh maybe you disagree
1:50:39that's fine maybe you have a particular
1:50:41example where this guideline um doesn't
1:50:44really apply so the rule doesn't apply
1:50:47um but I think it's important that we
1:50:49have some kind of guidelines these are
1:50:51the ones that formed in my head uh over
1:50:54a few years of working with uh novice
1:50:57designers particularly students and then
1:51:00also based on my own experience like
1:51:01going to this i e VR conference and I
1:51:04don't want to be too negative obviously
1:51:05it was a cool experience but as a
1:51:08designer when you look at some of the
1:51:10artifacts created that way like the
1:51:12virtual environments that you looked at
1:51:14and and really they didn't have a lot of
1:51:15time they had to put this together very
1:51:17quickly and I assume little user testing
1:51:21um and then working with this new medium
1:51:23uh and VR for conferences is still
1:51:25pretty new so all that um is obviously
1:51:29clear so I'm not attacking them in that
1:51:31way it's just a great example to to
1:51:34study how much do you want to push the
1:51:36realism and this trade-off between
1:51:38making it like a reality and then all
1:51:42the issues that come with it when you
1:51:43don't actually live up to that
1:51:45expectation when you actually break the
1:51:47Illusions uh be it through your design
1:51:50and some shortcomings of it or maybe
1:51:52issues with the technology like limited
1:51:54tracking or other kinds of issues so I'm
1:51:57going to leave this video a little bit
1:51:59more open-ended just because I do think
1:52:01that these guidelines don't need to
1:52:03evolve and at least now we have a way to
1:52:05think about guidelines and how to assess
1:52:07the source of guidelines and uh and when
1:52:11they apply and don't apply so I'm going
1:52:13to leave you there and think about the
1:52:16stairs which still I think it is
1:52:19fascinating to me and um if you get into
1:52:22the this if you get to this stage where
1:52:24you start obsessing about some of these
1:52:27design decisions and why things are
1:52:29there are so questioning some of the
1:52:31design decisions then I think it shows
1:52:34that you are starting to think
1:52:35critically about design and that is a
1:52:38really good thing and would make me
1:52:39proud if that's what I have accomplished
1:52:42today with this lecture
1:52:58finding your XR
1:53:00process really how do you create these
1:53:03XR experiences Michael that's I get this
1:53:06question a lot so I want to walk you
1:53:08through some potential processes and
1:53:11actually the methods that really believe
1:53:13in as part of the
1:53:14segment okay so if you think about
1:53:17interaction design traditionally um as a
1:53:21process it is is really starting from
1:53:23requirements which is essentially need
1:53:25finding understanding what users want
1:53:27from that creating alternative designs
1:53:30um you need something to experiment with
1:53:33and then you actually again work with
1:53:34users to understand which of these
1:53:37designs play out you create interactive
1:53:39prototypes they can be tested they can
1:53:41actually still be in paper they don't
1:53:43have to be implemented uh in technology
1:53:45yet or you create these interactive
1:53:47prototypes with digital tools you can do
1:53:49that as well and then you do more
1:53:52thorough evaluation potentially finding
1:53:54new requirements and this whole thing
1:53:56becomes a cycle now you've probably
1:53:58heard of this this is traditional
1:53:59interaction design
1:54:02process okay going a little bit deeper
1:54:04on the interaction design process we
1:54:06usually talk about four steps framing
1:54:08the problem exploring the solution space
1:54:11finding a good solution and refining the
1:54:13solution and then as we're going through
1:54:16this process we actually need to diverge
1:54:18initially consider lots of alternative
1:54:21options and then converge so first we
1:54:23generate options and choices and then we
1:54:26actually make choices a final part we
1:54:30often teach about the traditional
1:54:31interaction design process is that it's
1:54:33iterative in nature once you've gone
1:54:34through one of these Cycles you may
1:54:36iterate and improve the product now I
1:54:38will argue that in this traditional
1:54:40interaction design process there is a
1:54:42lot of value that we can actually
1:54:44translate to
1:54:45XR so now let's translate what we've
1:54:48learned about the interaction design
1:54:49process to the XR process I think of
1:54:51four four big steps in this process need
1:54:55finding and brainstorming where you are
1:54:57essentially framing the problem via
1:54:58scenarios and use cases personas and
1:55:01competitors and if you don't know what
1:55:02those mean I'll actually expand on those
1:55:05in later
1:55:07segments storyboarding and prototyping
1:55:09which is a significant Second Step
1:55:12requires several iterations creating
1:55:13markups using paper and digital tools
1:55:16and then even involving XR devices the
1:55:19third step is development and testing
1:55:21this is where you're creating the
1:55:23application using either web XR Unity or
1:55:27unreal or some kind of native SDK
1:55:29building on mobile development platforms
1:55:32or specific VR or ar tool kits and
1:55:35platforms I'll expand on those in other
1:55:37segments as well and then finally
1:55:40deployment and analytics I think so once
1:55:42the application has been developed and
1:55:44is deployed with users we need to
1:55:46continue evaluate how well it's doing so
1:55:48deploying the application on an xile
1:55:50device is the first step but then
1:55:51collecting during usage and analyzing
1:55:53how well the XR application is doing and
1:55:56what could be improved so identifying
1:55:57opportunities for improvement based on
1:55:59how users interact with the application
1:56:01that's the goal of the fourth step and
1:56:03this is really the whole thing is really
1:56:05the life cycle of an XR application so
1:56:08now I want to expand a little bit more
1:56:10on prototyping when I think of
1:56:12prototyping really I think of
1:56:15storyboarding where I create story
1:56:17boards and sketches to frame the problem
1:56:20and develop the story that I want my
1:56:23users to
1:56:24experience I think of physical
1:56:26prototyping so where we may use Play-Doh
1:56:29paper cardboard transparency anything
1:56:32beyond paper essentially to become
1:56:36spatial and explore the opportunities of
1:56:38XR interactions so these physical
1:56:40prototypes is really something that I
1:56:42believe strongly in I'll we spend some
1:56:44time on it they allow us to design the
1:56:46concept and explore the interactions
1:56:48before we Implement them in any digital
1:56:51tools now obviously prototyping then
1:56:53finally also includes digital
1:56:55prototyping and with digital tools to
1:56:58test the feasibility and experiment with
1:57:00XR specific features things that we can
1:57:03only really explore once we have it on
1:57:05the devices now I think all these steps
1:57:08are really important and we're going to
1:57:10explore them in more depth in future
1:57:12segments so now a term that I've used
1:57:15previously is Fidelity and I want to
1:57:17explain what I mean Low by low Fidelity
1:57:20and by high fidelity
1:57:22so when I create a new XR experience I
1:57:24usually start on paper I like sketching
1:57:27out things on paper thinking it through
1:57:29and there are some templates out there
1:57:31one of the issues that I have with
1:57:33working on paper is that I'm not very
1:57:35good at working in 3D on paper most XR
1:57:38experiences actually all of them are 3D
1:57:41and so it really requires this step so
1:57:44what I like to do instead I still stay
1:57:47in physical prototyping world and I
1:57:49create a diarama a diarama is really um
1:57:52the way I think about it it's like a
1:57:54miniature kind of prototype of my stage
1:57:58I be can can become physical I use
1:58:00things like Play-Doh and cardboard and
1:58:02transparency to really put together the
1:58:05XR experience and then explore it
1:58:08physically now the fastest way to jump
1:58:12from physical prototyping to digital
1:58:14prototyping especially for VR is
1:58:17essentially 360 so this is a 360 camera
1:58:21um that has two lenses and with one
1:58:23click I essentially take a 360 photo um
1:58:28if I use a video recording I can even
1:58:30move in space otherwise I'm like
1:58:32anchored at this location uh this is
1:58:34also one of the main limitations of 360
1:58:37is that it actually doesn't have depth
1:58:40so if you now want to create if you want
1:58:42to transition forward in your experience
1:58:45increase the Fidelity The Next Step from
1:58:47here onward is actually going into 3D
1:58:50for 3D just making that jump from 360 to
1:58:533D there are lots of tools out there and
1:58:55I'll introduce you to some of them but
1:58:57the harder jump is actually into VR and
1:59:00AR and the way I've positioned VR and AR
1:59:03here U is as if the Fidelity of AR is
1:59:06higher than VR and what I wanted to
1:59:09express is that I think creating a high
1:59:12fidelity AR experience is actually quite
1:59:15hard and so I want to explore this um
1:59:18along this graph quickly okay we're
1:59:20talking now about the fality to which I
1:59:22that I've just talked about and then we
1:59:25also talking about the skills required
1:59:27to actually create those experiences at
1:59:29those fatalities so paper is really on
1:59:33the low end the diarama the physical
1:59:35prototype um a little bit higher just
1:59:38requires a little bit more time the jump
1:59:40from diarama to 360 is relatively easy
1:59:42in fact we could take a 360 inside our
1:59:46physical model and then work from there
1:59:48digitally from 360 to 3D there's really
1:59:52not a lot of good tool support out there
1:59:54it really requires us to do to build a
1:59:57scene in 3D and in fact the jump from
2:00:00physical to 3D or even from 360 to 3D is
2:00:04where most of my students actually spend
2:00:07a lot of time on we we really need to
2:00:09explore how to do 3D and even in these
2:00:11courses here we really spend a lot more
2:00:13time on thinking through 3D once we are
2:00:16in 3D the jump to VR is still quite a
2:00:19jump but it's doable and then to R you
2:00:22really need quite a lot of skills the
2:00:24problems and the differences between VR
2:00:26and AR in VR we can control the entire
2:00:30environment and we need to design it now
2:00:32that actually requires a little bit of
2:00:34skill but it's arguably technically even
2:00:37harder to really work in AR where we
2:00:40need to sense the environment around us
2:00:42and then have our experience fit in
2:00:44nicely and so this is why I have plotted
2:00:48it out this way and I think really it's
2:00:50a good overview of the steps uh in the
2:00:53process the Fidelity you want to achieve
2:00:55and then the skills you may require for
2:00:57each of the
2:00:58steps from this placeholder content I
2:01:01often bring in 3D models here is an
2:01:04example of a nicely looking 3D cube
2:01:07still relatively easy to model this but
2:01:09it has at least a nice color shade and
2:01:12this is an example of more polished
2:01:14content so I'll start replacing The
2:01:16Primitives that I've used before as
2:01:17placeholders and bringing the 3D models
2:01:19that I really want what I have at at
2:01:22that stage is really just the content
2:01:24and now I need to think about the
2:01:25interactions so when we think about
2:01:27interactions we need to distinguish
2:01:29between implicit and explicit
2:01:31interactions I give you an example of an
2:01:34something that starts out implicit and
2:01:36then becomes an explicit interaction so
2:01:39when we use something like Google
2:01:40cardboard um where we can uh look around
2:01:43and actually Target content and then
2:01:45potentially click it with a button that
2:01:47we have on top of the device um we
2:01:50usually indicate whether user is looking
2:01:52with a cursor so let's assume here's our
2:01:54cursor and now the user is targeting
2:01:57this Cube so they move over they press
2:01:59the button it performs a click and we
2:02:02adjust the 3D model and we've registered
2:02:04The Click now this was a combination of
2:02:06an implicit interaction that actually
2:02:09became so me targeting the object and
2:02:11then actually became an explicit
2:02:13interaction as I clicked so we to handle
2:02:16both and this is really a very simple
2:02:18example with just using cardboard but
2:02:21these are the kinds of things you really
2:02:22need to think about when it comes to
2:02:24increasing the Fidelity and then adding
2:02:26interactions to your scene as well okay
2:02:29now what I've done is talk about media
2:02:32the different kinds of skills involved
2:02:33and then also have given you examples
2:02:35what this means in terms of how the
2:02:37content evolves and the interaction the
2:02:39last thing I want to do is actually
2:02:40putting it in a timeline here I'm going
2:02:42to give you an example of the XR process
2:02:45how I often implemented in my classes so
2:02:48we usually start out with a design
2:02:50concept where we really determine the
2:02:51problem frame and think about whether
2:02:54something is AR or VR from there we
2:02:56create a first prototype usually still
2:02:59quite low Fidelity with potentially
2:03:01basic interactions and really what we
2:03:03need to figure out here is the platforms
2:03:05and the tools that we use for our
2:03:06prototype this is just a V1 so then we
2:03:09get maybe some initial feedback from
2:03:11users or the instructor in my case and
2:03:13we work on a second version really what
2:03:16I often do is I scope it down to two to
2:03:18three key interactions I think it helps
2:03:20a lot to think about an our experience
2:03:22in terms of the interactions that we
2:03:24want to support and what is also
2:03:26interesting you may actually may have to
2:03:28switch some of the tools sometimes
2:03:30because if you have to work on a more
2:03:32tricky interaction um it'll dictate
2:03:34which tools you need to
2:03:36use so after these one or two uh maybe
2:03:40even three versions of our initial
2:03:42prototypes we may create something that
2:03:44we can at this stage call a minimum
2:03:46viable product or MVP which we can use
2:03:49to run some initial user tests get some
2:03:51feedback back and from that what we've
2:03:53done from an initial design concept that
2:03:55we've prototyped out maybe first on
2:03:57paper then created a few digital
2:03:59versions finally tested with users what
2:04:02we have now is actually a design
2:04:04specification potentially in the form of
2:04:06a report where we can reflect on some of
2:04:08the lessons that we've learned and we
2:04:10can use this design spec and actually
2:04:12give it to developers to implement it
2:04:14further or if we realized we actually
2:04:16have to do a lot of improvements this
2:04:19was just one design Sprint so we could
2:04:21do another design Sprint so when I think
2:04:23of design Sprints I usually think of one
2:04:26week or two weeks of work following this
2:04:29process and iterating through this
2:04:31process multiple times will eventually
2:04:33give you a really cool XR experience so
2:04:36in the following segments of this course
2:04:38we actually focus more on prototyping
2:04:41and then development and testing so
2:04:43development and testing really what are
2:04:45the three big options you have here if
2:04:47you remember from previously well it's
2:04:49the web- based route through web XR it
2:04:52is the kind of like Game Dev route if
2:04:54you want through unity and unreal or you
2:04:56do the hardcore development route maybe
2:04:59if you're an experienced mobile
2:05:00developer that's your thing you can do
2:05:02Native uh development with the Oculus
2:05:04SDK or the Vive SDK or also some of the
2:05:07AR Solutions out there so uh in the
2:05:10following we'll spend more of our time
2:05:12on web XR and unity and that's it for
2:05:15this course but I'll provide resources
2:05:17so that you can also learn more about
2:05:19the
2:05:20Alternatives and finally fin once we
2:05:22have created our XR experience we can uh
2:05:25deploy it with users and we should
2:05:26really think about analytics that's what
2:05:28I've thought about and this stage I keep
2:05:30it very high level just for us to Think
2:05:32Through what are the three main ways we
2:05:35can do this well one we could actually
2:05:38Implement custom logging and perform a
2:05:40little bit of log analysis this is
2:05:42something that a lot of researchers do
2:05:43and is actually uh practiced by a lot of
2:05:46people even just for user experience
2:05:49testing and usability testing you may
2:05:51wonder whether there are some tools out
2:05:53there yes so some of them are built into
2:05:56some of the tools we were using so for
2:05:57example Unity has analytics built in
2:06:00these are platform based analytics tools
2:06:02that are actually focused more on like
2:06:04how many people are downloading your app
2:06:06and what are they doing with it roughly
2:06:08they may be too limited to really
2:06:10understand where the issues are with the
2:06:12interactions for that there is a new
2:06:15range of more dedicated XR analytics
2:06:17tools those that would allow you to
2:06:19analyze how users interact uh when
2:06:22interactions fail so when for example
2:06:24people wanted to interact with content
2:06:25it wasn't really there when it caused
2:06:27frustration potentially when users
2:06:29appeared confused this is however an
2:06:33area of active research and only a few
2:06:35tools out there that are still evolving
2:06:38so I provide a few pointers and links
2:06:40for you but we won't spend too much time
2:06:43on these things as part of the course
2:06:45here
2:06:59in this video we're going to talk about
2:07:01XR design thinking now design thinking
2:07:04is actually quite an established method
2:07:06that has been around for many years
2:07:07coming out of the Stanford design school
2:07:10and it's often applied to Innovative
2:07:13product development and uh making user
2:07:17experiences if you will but it's really
2:07:19about a mindset uh building empathy with
2:07:23the users really getting really focusing
2:07:25on the user and then everything else
2:07:27will follow that is really the idea uh
2:07:30it's it's it's more of a let's say more
2:07:33of an approach and uh a way of thinking
2:07:37than its own kind of method or set of
2:07:41techniques it actually is associated
2:07:43with common interaction design and user
2:07:44experience design techniques um at least
2:07:46that's the way I think about it now what
2:07:48I want to accomplish here though is that
2:07:51we are importing that Knowledge from
2:07:53design thinking and applying it to XR
2:07:56there's a couple of things that are
2:07:58quite different for example when it
2:08:00comes to Innovation and problem framing
2:08:03um because XR is really like this
2:08:06technology that could potentially solve
2:08:09a lot of problems but um it's more like
2:08:11we're throwing that technology at
2:08:13everything it's like our Hammer as I
2:08:15said previously our hammer and now
2:08:16you're trying to find all the nails that
2:08:18you can hit with it and and sometimes
2:08:20you'll be successful and uh sometimes
2:08:22you're not and so design thinking is
2:08:26actually uh is a specific way of framing
2:08:29a problem by really building empathy
2:08:31with your users now your users will not
2:08:34feel like oh I need AR or no oh I need
2:08:37VR now AR or VR might be the method or
2:08:41the Tool uh if not the technology to uh
2:08:44potentially solve the problem now I want
2:08:46to be very careful because technology in
2:08:49itself doesn't actually solve any
2:08:51problems but um it might actually be an
2:08:54enabler and um that's how I want us to
2:08:57think about that so here I'll give you
2:08:59an overview of the design thinking
2:09:01method in the first step you'll
2:09:03empathize you'll try really you
2:09:05empathize with your users you're trying
2:09:07everything to find out about their
2:09:09problems what are their pain points uh
2:09:12what is currently working for them what
2:09:14is currently not working for them you're
2:09:16getting to learn the context and the
2:09:17setting in which current Solutions are
2:09:19applied to really identify a outc
2:09:21comings and gaps and therefore
2:09:23opportunities and this will help you
2:09:25define the problem really the first few
2:09:28steps the first three steps to be
2:09:30precise are really about problem framing
2:09:32they're not so much about solution
2:09:34finding you're actually trying to
2:09:37understand the problem not just the
2:09:39symptoms of the problem but actually the
2:09:43problem and so there are common
2:09:44techniques there terms of interviewing
2:09:47and observing and doing focus groups and
2:09:50there are all kinds of standard let's
2:09:52say standard traditional human computer
2:09:54interaction and user research uh
2:09:56techniques and then we are entering the
2:09:58ideation stage and the ideation stage is
2:10:01usually closely tied with prototyping as
2:10:04well so ideation really doesn't just
2:10:07happen in your brain or just through
2:10:10discussion what I'll preach here more or
2:10:12less is like that ideation happens
2:10:14through sketching and through
2:10:17storyboarding and through wireframing so
2:10:19these are some of the common techniques
2:10:21that I really want you to embrace when
2:10:23it comes to the ideation stage and that
2:10:25then actually helps us transition into
2:10:27prototyping and so in prototyping you
2:10:29usually work with paper and building
2:10:32really paper prototypes I'll show you
2:10:34some examples of this especially in the
2:10:36XR context this is really useful when
2:10:38you have these first prototypes you can
2:10:41go also from paper to digital but even
2:10:43just the paper prototypes you could
2:10:45actually test with users and keep in
2:10:47mind that the nature of the Prototype
2:10:49really also some frames the feedback
2:10:53that you will get so for example if you
2:10:54have something that looks very low
2:10:56Fidelity initial stage sketchy if you
2:11:00want you probably get very different
2:11:02feedback not so much about the colors
2:11:04but maybe more about the flow and maybe
2:11:07the organization roughly of your
2:11:08interfaces and so this is a very
2:11:11important thing to keep in mind so you
2:11:13don't want your prototypes to look too
2:11:15polished because it may also there is
2:11:17this observation that if something looks
2:11:20too finished it's close to being done
2:11:22then your testers might be reluctant to
2:11:25provide more feedback because they
2:11:26anticipate a much higher cost involved
2:11:29in making changes and now that's an
2:11:31interesting aspect as well and with this
2:11:34testing phase concluded although testing
2:11:37to be honest never really finishes cuz
2:11:39you go on and Implement here when we
2:11:42talk about implementation I usually
2:11:44think of prototypes as something we're
2:11:46implementing a prototype and we're
2:11:48testing that prototype and then we're
2:11:50doing potentially another iteration here
2:11:52this idea of implementation really then
2:11:54you're going to use the Technologies in
2:11:57our case we would actually go and try
2:11:59things out with XR devices directly now
2:12:03I'll introduce you to methods that allow
2:12:05you to actually use XA devices also in
2:12:08the prototyping and ideation stages even
2:12:11so for example through immersive
2:12:13prototyping where you actually create
2:12:15interfaces directly on an EXA device and
2:12:17can test it as you're designing
2:12:19concurrently testing it as you're
2:12:21designing and so that is a really really
2:12:23cool thing that you can do that is
2:12:25almost unique to XR so now that we have
2:12:28established this six-step method with
2:12:30the empathize Define ideate prototype
2:12:34test and Implement steps let's look at
2:12:37some of the issues when it comes to XR
2:12:40in each of these steps now I said a
2:12:42common way to empathize is to interview
2:12:46users and to observe users and
2:12:49understand really their challenges and
2:12:51what are the pain points now the problem
2:12:54with XR is that XR is usually not
2:12:57already a solution so if you're refining
2:12:59an existing XR prototype yes then maybe
2:13:02you can observe how users use it but if
2:13:05you're looking at an at at an existing
2:13:07problem and you're trying to bring in XR
2:13:09as a potentially new solution you can't
2:13:12really observe the users directly uh
2:13:15it's very different from like uh web
2:13:17based and mobile based design where we
2:13:21often the the users really work with
2:13:23this technology like on a daily basis XR
2:13:26is really not like that I mean I don't
2:13:28know a lot of people that actually use
2:13:29VR on a daily basis and or ar on a daily
2:13:32basis and then certainly not to achieve
2:13:34anything productive I mean we moving
2:13:36into meeting scenarios and carrying out
2:13:40meetings in uh in XR or in AR or VR um
2:13:44but we don't really work yet at least
2:13:48not yet in these kinds of virtual SPAC
2:13:51with virtual offices and if we did then
2:13:54the more traditional techniques of
2:13:56observation interviewing uh and really
2:13:59like shadowing and doing kind of like
2:14:01field studies but the reality here at
2:14:04the moment is that you cannot really
2:14:06observe the issues directly as they
2:14:08happen with
2:14:10XR so in the definition uh phase to me
2:14:14it's very important that you don't just
2:14:16look at the problem or don't just look
2:14:18at the symptoms of the problem and then
2:14:20trying to
2:14:21whatever it it takes to you know to
2:14:24press that problem into an AR or VR
2:14:26solution U many of us have a preference
2:14:29so somebody maybe likes VR more than AR
2:14:32and then they're trying to solve
2:14:33everything with VR or solve everything
2:14:35with ar for me it's very important that
2:14:38we keep an open mind so don't just
2:14:40commit to AR or VR it might very well be
2:14:43that after going through these initial
2:14:46kind of need finding strategies that you
2:14:50figure out that that AR neither AR nor
2:14:53VR is actually a solution and that would
2:14:55actually be a very good result it could
2:14:57avoid a lot of cost and and save you a
2:15:01lot of time and energy so as we are
2:15:04moving into the ideation stage what is
2:15:07very very important to me here and I
2:15:09really say this to my students all the
2:15:11time is that when you ideate uh and you
2:15:15already have like experience for example
2:15:17using devices like holl lens and you
2:15:19know of all the technical limitations of
2:15:21these devices so for example uh the
2:15:24frequency with which they scan the
2:15:26environment although that can be
2:15:27configured or the accuracy with which
2:15:29they can represent the environment or
2:15:31the lack of semantics that we have um so
2:15:34it's really hard to detect objects
2:15:36reliably objects we've never seen before
2:15:38and then know that the computer knows
2:15:41that this is this or that object um if
2:15:44you if you walk into a design or an
2:15:46ideation session with these technical
2:15:49constraints in mind you're really
2:15:52limiting the ideation you're are
2:15:54basically biased and uh and and you you
2:15:57really do want to avoid that I really
2:15:59like working with a diverse set of uh
2:16:02teams in these ideation settings where
2:16:06uh some of the students often that I
2:16:08work with uh or other kinds of
2:16:11practitioners or other faculty
2:16:12professors where they have actually
2:16:15limited experience with XR where they're
2:16:17thinking of it as being something that
2:16:19is magical and super powerful because it
2:16:22doesn't hold them back in their ideation
2:16:24and I think that is really really
2:16:26important when it comes to design
2:16:27thinking for
2:16:29XR now as we're entering the uh
2:16:32prototyping stage uh when I work with
2:16:36students and we we have like these
2:16:38design concept ideas we have ideated we
2:16:41have already some kind of like story
2:16:43boards and and wire frames and now maybe
2:16:46we're moving already from paper to
2:16:48digital then uh one of the things that
2:16:51that um there's a tendency to do the
2:16:54basic things like the things that you
2:16:56know already to do them digitally like
2:16:58for example a login screen I always
2:17:00tease my students don't do the login
2:17:02screen first of all all login screens
2:17:04look more or less the same so what's the
2:17:06point of prototyping those digitally you
2:17:09won't learn a lot more what you should
2:17:11do instead is really focus on the
2:17:13unknown unknowns okay so there might be
2:17:17actually a lot of gaps in your current
2:17:19design idea design concept and use
2:17:22prototyping to really figure out uh how
2:17:25to fill those gaps and that's the best
2:17:27way to approach it so you the
2:17:30prototyping uh doesn't need to happen uh
2:17:33in a complete prototype uh way um it can
2:17:37be the more like a horizontal prototype
2:17:41where it's more like the facade so
2:17:42you're trying to figure out uh the whole
2:17:44front end and all the options kind of uh
2:17:47shown to the user and if you're taking
2:17:50this approach then you you're less
2:17:52focused on the vertical on the
2:17:54technology and more focused on figuring
2:17:56out the entire user facing uh parts of
2:17:59the interface and uh ultimately fill all
2:18:02the gaps but really even then don't
2:18:05start with the known uh interface parts
2:18:09and then just do them uh digitally I
2:18:12mean maybe you can do this for warm-up
2:18:14but what you should really focus on is
2:18:16the things that you don't know yet how
2:18:18they would materialize in the user pH
2:18:21for example so focus on those things in
2:18:23as you're prototyping uh your XR
2:18:26experience so then when it comes to
2:18:29testing uh testing is very very
2:18:32interesting because as we are using XR
2:18:36we could actually use XR Technologies to
2:18:39fill the gaps so think of an interface
2:18:41that is nearly complete but then you
2:18:43want something magical to appear and you
2:18:45don't know yet how to prototype it now
2:18:48you could do this in a video prototype
2:18:49so some kind of through some kind of
2:18:51overlays uh using some video editing
2:18:53software but you may also use some more
2:18:56advanced uh AR or VR tools to really
2:18:59bring in some of these overlays so uh I
2:19:02want you to think of XR Technologies to
2:19:05also fill the gaps in your XR prototypes
2:19:08and so this is a very Innovative use of
2:19:11using XR Technologies to support
2:19:12prototyping and testing to really uh the
2:19:15idea is to so the idea in this testing
2:19:19phase is to simulate an almost complete
2:19:22user experience to simulate a the
2:19:25experience of a working prototype where
2:19:28everything has been implemented uh and
2:19:29you can fill these uh gaps if you don't
2:19:32have a full prototype um implemented you
2:19:35can fill these gaps uh using humans as
2:19:39Wizards um so this idea of The Wizard of
2:19:41Oz to really fill the gaps so wizard of
2:19:44O prototyping and testing is a really
2:19:46key idea that I think also makes a lot
2:19:49of sense in XR and especially in XR so
2:19:52for example one of the things I have
2:19:54done is like jumping into a VR
2:19:56experience and I was like more or less
2:19:58invisible to the VR user and I could
2:20:01move things in VR to produce like a
2:20:03virtual like a heads up display to them
2:20:07and they thought that oh these screens
2:20:08are animated and they're like moving and
2:20:11this interface is responsive this
2:20:12personal assistant really understands me
2:20:15but in fact I was actually there in the
2:20:17background as a wizard um and
2:20:21or the computer uh to really fill in the
2:20:24gaps and so I think that is a very uh
2:20:26cool way of simulating a more complete
2:20:29prototype that we then can test that way
2:20:32so it could be a really intelligent
2:20:34system that we're prototyping that
2:20:36understands all kinds of gestures and
2:20:37speech commands and uh in this kind of
2:20:40testing phase to really simulate the the
2:20:44almost final user experience even though
2:20:47we haven't fully implemented the system
2:20:48yet finally when it it comes to
2:20:52implementation I think one of the common
2:20:54mistakes here is that many of us are
2:20:57very familiar with one tool so for
2:20:59example Unity or unreal uh some of us
2:21:03may know both but then we we don't
2:21:05actually consider other options and we
2:21:07try to do everything with these tools
2:21:10because we think Unity or unreal is the
2:21:12best tool now the point I want to make
2:21:14here is that you should use the best
2:21:16suited tool so for example an early
2:21:19stage prototype you really don't have to
2:21:20run and and go into unreal and use all
2:21:24their high-end rendering capabilities or
2:21:27Unity uh you also don't need to jump
2:21:29into Unity to get started no you can
2:21:32choose other kinds of prototyping tools
2:21:36uh maybe more like geared towards
2:21:37immersive prototyping so really have a
2:21:39variety of options and don't be afraid
2:21:42uh even though it's not always easy to
2:21:43transition between some of these tools
2:21:46uh having this flexibility in terms of
2:21:49uh tools and knowing a variety of tools
2:21:52is really actually a strength and that
2:21:54would enrich also your the way you think
2:21:56about the problem um not just like with
2:21:59this one tool but from multiple
2:22:01different
2:22:16perspectives so next I'm going to
2:22:18quickly give a high level overview of
2:22:20the common methods and tools that we
2:22:21would be using so the first thing we
2:22:24really want to the first deliver in some
2:22:26sense we really want to accomplish is
2:22:28the problem statement and the way we do
2:22:30a problem statement or we go about a
2:22:32problem statement is through scenarios
2:22:33and use cases and maybe diary studies so
2:22:35you're asking users to take notes about
2:22:37their daily lives and Reporting back to
2:22:39you what kinds of issues they had for
2:22:41example with uh as they were using your
2:22:44AR VR prototype you may be wanting to do
2:22:47some surveys so a little bit more larger
2:22:49scale on online or in specific user
2:22:52communities you may want to do
2:22:54face-to-face interviews or actually put
2:22:56multiple people together in a focus
2:22:58group and uh putting these different
2:23:00stakeholders together to really frame
2:23:03the problem so that's the problem
2:23:05statement and then we really want to
2:23:07understand the user we want to under we
2:23:09want to Define we want to Define who the
2:23:12user really is uh common technique here
2:23:15is to use personas now personas don't
2:23:18think of personas as these uh sheets
2:23:21that we fill in where every Persona has
2:23:23like a cute name and maybe some
2:23:24characteristics I think of personas more
2:23:27like as The Meta behind creating these
2:23:30personas that is actually where the rich
2:23:31information is so not the sheet that I
2:23:33present to you but actually the way I've
2:23:36constructed these personas and if you
2:23:38done if you've done this in the right
2:23:40way it would be based on facts and data
2:23:43um and I'll talk a little bit more about
2:23:44personas later as well another way to
2:23:47define users and really understand who
2:23:49we're designing for and who we're not
2:23:51designing for is a task analysis and if
2:23:54you have already an existing prototype
2:23:56uh you could use that to collect the
2:23:58data to some kind of logging and then
2:24:01log analysis um you may want to do user
2:24:04Journeys uh similar to The Diary studies
2:24:06but in this case maybe a little bit more
2:24:09uh visual where people really think
2:24:11through their daily activities and you
2:24:14trying to find out what are the
2:24:16activities and what are the kinds of
2:24:17events and how do they in their in their
2:24:19day-to-day activities and how do they
2:24:21impact those activities so user Journey
2:24:23Maps is something that is very popular
2:24:25in the user experience design community
2:24:28and that's something that you could also
2:24:29apply to XR uh similarly stakeholder
2:24:32maps and there are all kinds of ways of
2:24:34really Framing and understanding uh our
2:24:37users next we really want to understand
2:24:40the competitive landscape so uh what are
2:24:44the solutions out there uh what are the
2:24:46factors where they are good what are the
2:24:49good features uh what kinds of features
2:24:51are missing and a common way to get it
2:24:53this competitive analysis is a SWAT
2:24:56analysis so you're looking at the
2:24:58strengths the weaknesses the
2:24:59opportunities and the threads uh and
2:25:03doing this kind of analysis which is
2:25:05more like coming from a business
2:25:06perspective but it's often now applied
2:25:08to user experience design and
2:25:10interaction design as well you could
2:25:13obviously look at existing products and
2:25:15uh download them and try them out uh if
2:25:19you don't have access to some of the
2:25:20tools out there you may want to base
2:25:23your knowledge on uh existing reviews
2:25:26out there so there might have already be
2:25:28an existing user community and there
2:25:31might be forums associated with a
2:25:32software solution where people complain
2:25:34a lot about uh something and so that's a
2:25:37way to look at it uh you could look at
2:25:39stack Overflow or some of these common
2:25:42sources where people report um issues
2:25:45and try to help each other you could
2:25:47also do a more systematic uh research
2:25:50BAS bed approach kind of a literature
2:25:52review um and then really uh look at the
2:25:56things that are described maybe in white
2:25:58papers in blog posts or academic papers
2:26:01as well and that way build your
2:26:02knowledge of you really want to get the
2:26:05sense of the uh technology or the
2:26:08solution landscape out there and really
2:26:11who are the
2:26:13competitors as you're going about your
2:26:16design uh and you're drawing from the
2:26:19knowledge that you generated from the
2:26:21problem statement and reframing the user
2:26:23and then also understanding the
2:26:25competitors um you're going to do
2:26:28something uh about design synthesis so
2:26:32really like getting all the data and
2:26:35facts together and Performing what I
2:26:37refer to as design space analysis and
2:26:40here I introduce you to a technique
2:26:42called uh qoc analysis so questions op
2:26:46options and criteria so questions
2:26:49options and criteria
2:26:51and you're asking questions about the
2:26:53options that you have available and then
2:26:55you're determining criteria that will
2:26:57help you actually choose the best option
2:27:00if you want uh I think this is a very
2:27:02cool way of analyzing a design space and
2:27:05it's very
2:27:06useful if you're more into software
2:27:09engineering or systems you usually think
2:27:12in terms of functional and nonfunctional
2:27:14requirements lists uh so functional
2:27:17really like about the system and things
2:27:18you can like measure like system
2:27:21performance response time and those
2:27:23kinds of things uh and then
2:27:25non-functional requirements as well so
2:27:27more associated with user
2:27:29characteristics or the soft things um
2:27:32that are hard to
2:27:33implement so to give you an example of
2:27:36scenarios here is uh here are two
2:27:39scenarios that we've used in our work on
2:27:42Cross device augmented reality and we
2:27:44were basically using uh these at the
2:27:47time more futuristic scenarios what if
2:27:50family wanted to uh Place Furniture in
2:27:53the living room what if they could do
2:27:54this collaboratively using different
2:27:56kinds of augmented reality devices or
2:27:58what if uh a bunch of friends wanted to
2:28:02collaboratively play some kind of laser
2:28:04shooting uh game where they're shooting
2:28:07inspired from Robo raid on the Holland
2:28:09where they're shooting these kinds of
2:28:11bugs from the future uh how could we
2:28:15enable that uh collaboratively and so we
2:28:18basically did a scenario B based
2:28:20analysis so we threw these scenarios
2:28:22visually as images uh at our
2:28:25participants and then we really asked
2:28:27them to try to solve these scenarios and
2:28:30think through some of the issues whether
2:28:33they are designers or developers or end
2:28:36users so we really studied with all
2:28:37these three types of populations and
2:28:40really un really wanted to understand
2:28:42how they think about these scenarios and
2:28:44that helped us uh really drive a lot of
2:28:47the requirements and then build a
2:28:49solution
2:28:50um for cross device augmented reality
2:28:53it's a paper that if you I I'll share
2:28:55with you if you feel like you'd like to
2:28:57take a look so when we talk about
2:28:59scenarios we really need to take into
2:29:01account five key elements so really the
2:29:04setting so the situation or the state of
2:29:07the system the context of use basically
2:29:09the physical environment in which uh the
2:29:12activity occurs so then you need to look
2:29:14at the agents or the users uh so what
2:29:17are they trying to accomplish as the
2:29:20performing these
2:29:21activities um you need to think about
2:29:23the goals that they have so what are
2:29:25their objectives and what are they
2:29:27trying to accomplish and then what
2:29:30actions do they take uh to really
2:29:33accomplish their goals and if there are
2:29:35any kinds of outside Events maybe
2:29:37triggered by the system or somehow by
2:29:39the setting on the environment really
2:29:41things that happen to the users while
2:29:43doing their activities and if you bring
2:29:45in these five different scenario
2:29:47elements you'll actually really explore
2:29:50a future scenario of use of your
2:29:54interface in a very comprehensive way
2:29:58most scenarios that I read uh when
2:30:01presented to me are focused on just the
2:30:03user which is similar to just the
2:30:05Persona or just a setting but I don't
2:30:08learn enough about the goals and the
2:30:10actions and and if I learn more about
2:30:12the actions I can also Envision already
2:30:15a potential solution and the system
2:30:18behind the scenes so it's really
2:30:19important to address all these five
2:30:21elements as you're doing some kind of
2:30:24scenario-based
2:30:25analysis next I want to talk about
2:30:27personas personas are really commonly
2:30:29used to communicate uh users uh of our
2:30:34system and uh personas in a more
2:30:37traditional sense as I said in a user
2:30:40experience design and research uh s uh
2:30:45personas would really be used to distill
2:30:49all the facts and all the knowledge we
2:30:51have about our
2:30:52users um that we have gathered through
2:30:55interviews
2:30:57observations any kinds of surveys and
2:31:00other kinds of uh instruments that you
2:31:02may have implemented to really figure
2:31:04out the issues but at the core Persona
2:31:08is really used to to describe to other
2:31:12designers who we are designing for and
2:31:14who we're not designing for so common
2:31:16elements of a Persona are uh some kind
2:31:19of descriptive Persona
2:31:21name um a photo maybe that shows the
2:31:25user um or the context attributes so
2:31:28it's very important uh to learn more
2:31:30about the
2:31:31scenario uh and then often what helps in
2:31:34a Persona is a quote that best
2:31:36represents this Persona and then usually
2:31:39you list the goals you list their
2:31:41frustrations and needs um if they told
2:31:44you explicitly or based on your
2:31:47interpretation of what they told you and
2:31:50then also their motivations incentive
2:31:52fear achievement growth power these are
2:31:55kinds of common attributes um kind of
2:31:58that you find in a Persona template and
2:32:01so we need to distinguish different
2:32:03types of personas here um so commonly we
2:32:06talk about primary and secondary
2:32:08personas so primary personas really
2:32:11refer to users where we believe that our
2:32:14design completely satisfies uh their
2:32:17goals and needs uh as opposed to
2:32:19secondary users where we think that our
2:32:21design mostly but not completely uh
2:32:24meets their goals and needs and then we
2:32:28also have this idea of negative personas
2:32:31and this really helps to communicate to
2:32:33our stakeholders who we are not or not
2:32:37yet designing for so our design at this
2:32:40stage doesn't actually Target any of
2:32:42their goals or needs so personas is a
2:32:45really common technique in user
2:32:47experience research and uh design and uh
2:32:52what is unique or more special to XR so
2:32:55I think when we construct personas more
2:32:57specifically for an XR experience we
2:33:00need to think about how much experience
2:33:03do these personas actually have are they
2:33:06a novice or an expert XR designer
2:33:10developer or user and then what are good
2:33:13metrics to express their experience or
2:33:16they could be uh kind of like uh
2:33:18self-rated skill sces um or they tell
2:33:21you how many months of experience they
2:33:24have or how many times per week or even
2:33:27per day they use an XR experience and
2:33:30then they should also give you examples
2:33:33um of devices and applications that they
2:33:36have used so I think if you ask for all
2:33:39these kinds of things as background
2:33:41information and if you specify that for
2:33:43your personas then this is a very good
2:33:46way to frame uh and identify who are
2:33:49your primary and your secondary users so
2:33:52in a lot of my research I often targeted
2:33:55novice designers to enable them to be
2:33:58able to create XR experiences without
2:34:01being uh a developer or a programmer and
2:34:06so that's how I would normally fill in
2:34:09uh the personas now I do think that uh
2:34:13developers could be the secondary
2:34:15personas in a lot of my research because
2:34:18uh a lot of my work as focused on rapid
2:34:20prototyping and while my techniques are
2:34:23targeted at uh more like novice uh
2:34:26designers or novice XR creators even
2:34:30experienced uh XR creators could benefit
2:34:33from some of the techniques so I would
2:34:34consider them as secondary users and in
2:34:38a lot of my research uh I focus more on
2:34:41designers and developers rather than uh
2:34:44users or end users um and so um an end
2:34:49user user so somebody who uses that
2:34:52design created by the designer that is
2:34:55using my tool so that end user would be
2:34:59an example of a negative Persona in a
2:35:01lot of my work doesn't mean that I don't
2:35:04like XR users it just means that I
2:35:07currently don't really design uh for
2:35:09them that my solution is really geared
2:35:12towards XR
2:35:14creators so talking more about personas
2:35:17and constructing personas um I have a
2:35:21list of dos and don'ts here um so what
2:35:25you should really do is clarify how
2:35:27these personas were created are they
2:35:30constructed or are they facts based um
2:35:34um ideally you should actually base them
2:35:36on data and facts but often personas are
2:35:40just used as a tool to communicate to
2:35:43other stakeholders who we are designing
2:35:44for as I said and so you should just be
2:35:46very clear and honest about that and and
2:35:49what is very important is that you
2:35:51actually let those personas reflect your
2:35:54diverse Target user groups so don't uh
2:35:56if you think in terms of three personas
2:35:58don't make them all the same uh in the
2:36:01activities that I usually do with my
2:36:03students uh when we are constructing
2:36:05personas I really want them to think
2:36:07about the most uh unique and very
2:36:10different types of users that they might
2:36:12be designing for to really explore that
2:36:15space uh a little better and in addition
2:36:19to primary and secondary personas we
2:36:22really also should always include
2:36:23negative personas to make it clear to
2:36:26others who we are not designing for so
2:36:28what you shouldn't do is just make
2:36:30assumptions you shouldn't stereotype
2:36:32personas so don't bring in this Persona
2:36:35of this old person that can't use
2:36:37computers for example um or even bring
2:36:40in certain genders and discriminate
2:36:42against them or or race and even though
2:36:46um I teach in a very diverse and
2:36:48inclusive environment I still see this
2:36:50happen all the time among my students
2:36:52and I want you to be careful finally
2:36:55don't confuse negative personas with
2:36:57people you don't want to uh use your
2:37:00solution now negative personas are
2:37:01really the people that you're currently
2:37:03not targeting in your design doesn't
2:37:05prevent them from using your solution
2:37:08but um maybe in a future iteration of
2:37:11your design you may Target them more
2:37:12specifically and then cater more to
2:37:15their uh needs and support more of their
2:37:18goals
2:37:32so what I often use is a kind of
2:37:35template where I list out the
2:37:37competitors on one side and then all the
2:37:39kinds of features or even non features
2:37:43bugs uh on this side and I call them uh
2:37:46factors so you do this as the columns
2:37:50and the competitors as the rows and the
2:37:52competitors really are the solutions
2:37:54let's say the XR applications that are
2:37:56out there and um rather than just like
2:37:59putting indicators which of these
2:38:01factors are supported I usually write
2:38:03how well they are supported and how they
2:38:05are in fact implemented and this
2:38:07actually differs between Solutions but
2:38:09for the purposes of this presentation um
2:38:11let's just put uh xes as checks and
2:38:15supports this vector or not and uh like
2:38:19with personas um I think of competitors
2:38:23as direct competitors partial
2:38:25competitors indirect parallel and
2:38:28analogous so this seems very fine grain
2:38:31here but there are actually significant
2:38:32differences between each of these so
2:38:34direct and partial makes uh sense Direct
2:38:37in some sense is really overlapping they
2:38:40this uh competitor really provides the
2:38:42same uh features and services and
2:38:45targets the same users partial is um
2:38:48there is is a subset of the features for
2:38:51example supported or the only target a
2:38:53subset of the users that you have
2:38:55identified through your personas now an
2:38:58indirect competitor is somebody who goes
2:39:01about the same problem in and so in that
2:39:06sense is similar but goes about it in a
2:39:08different way and as and because they're
2:39:10still targeting the same problem uh and
2:39:13even though they go about it in a
2:39:15slightly different way they are still a
2:39:17competitor now a parallel now a parallel
2:39:21competitor is similar to what you have
2:39:22in mind but they're not necessarily a
2:39:25direct competitor and finally an
2:39:27analogous competitor is actually a
2:39:29non-competitor this is basically
2:39:31somebody who is is basically this is
2:39:34basically a solution that is targeting
2:39:37different users and different goals and
2:39:39needs but there is something in their
2:39:41design that inspires you and so what I
2:39:44usually do with my students is I really
2:39:45encourage them to think about the direct
2:39:47and the partial compe s and maybe some
2:39:51indirect and parallel competitors as
2:39:52well but R find those analogous
2:39:55competitors uh existing interfaces that
2:39:58that may address a different problem but
2:40:00they they there's something very
2:40:01interesting and inspiring about the
2:40:03design and that is something that maybe
2:40:06you want to think about adapting and
2:40:07incorporating Into Your solution and
2:40:10that is fine um usually all right and so
2:40:13then the way you go about these comp
2:40:16competitors and the competitive analysis
2:40:17as I said is through SWAT analysis where
2:40:20you identify the strengths and
2:40:21weaknesses and opportunities and threats
2:40:24and uh now here is a common Pitfall that
2:40:27I see among a lot of my students uh
2:40:29we're doing this comparative analysis
2:40:31and we're drawing out this whole
2:40:33landscape and then our solution is
2:40:36trying to do it all and just do
2:40:38everything better um we really like all
2:40:42these factors are important and so we're
2:40:44trying to build this super solution and
2:40:47that is really not how you should think
2:40:49about about it and what you should do
2:40:50instead is maybe find the gaps there
2:40:54must be gaps between some of the factors
2:40:56They Don't Really cater to the users's
2:40:58needs They Don't Really address their
2:41:00goals and that's where you need to
2:41:03innovate so don't try to do a dashboard
2:41:05that integrates all everything but
2:41:08instead try to find a more focused
2:41:10solution that really establishes new
2:41:13factors and that's the niche and that's
2:41:16what you're gearing your XR solution
2:41:18towards that's that um we now have
2:41:21looked at the scenarios I've shown you
2:41:24examples of personas we've done on
2:41:26comparative
2:41:27analysis and um I will have a separate
2:41:31video and separate modules dedicated
2:41:33toward storyboarding so that's not
2:41:35covered here but these four uh types of
2:41:38input really form what we then call the
2:41:41concept model or your design concept and
2:41:44that is really how you think about the
2:41:46solution as the designer now keep keep
2:41:49in mind that that mental model of that
2:41:51system that you are creating um the key
2:41:54challenge is to really invoke that
2:41:56mental model also in your users and in
2:41:58fact most systems fail because users
2:42:01often have a very different manal model
2:42:04quite different from the concept model
2:42:05that the designer had in mind so the
2:42:07designer's mental model of the system
2:42:10but if they match then you're really
2:42:12doing a very good job think about how
2:42:15you think about how an ATM works yeah
2:42:17you you're putting in your card you're
2:42:20asking for money it's going to give you
2:42:22back your card and then you're going to
2:42:25get the money uh a designer of that ADM
2:42:29might think a about it very differently
2:42:31much more fine grain very specific all
2:42:34kinds of error scenarios and um let's
2:42:37say if that ADM never Aid your card you
2:42:42didn't even know that this was possible
2:42:44your mental model is that ATMs are
2:42:47really nice they always give you money
2:42:49you've never experienced not having
2:42:51money uh on the account or maybe you
2:42:53have and so think about these different
2:42:55user experiences so eating your card
2:42:59insufficient funds or maybe some error
2:43:01there's actually enough money but
2:43:02something went wrong with the bank and
2:43:04think about all these user experiences
2:43:07and uh their mental models of ATMs might
2:43:10be very different from someone who
2:43:13always had enough money uh whose card
2:43:16was never eating was never eaten and
2:43:19they never actually had an issue with an
2:43:21ATM and also think about this the next
2:43:24time you're going to use the next ATM
2:43:26one one once one of these disruptive uh
2:43:30user experiences has happened to you you
2:43:32will be very afraid I have more examples
2:43:35like this uh for example check out lines
2:43:38uh self checkout lines specifically I
2:43:40always get very nervous about the
2:43:41machines in those instructions and so
2:43:44this is these are really examples where
2:43:47the uh design concept if you think about
2:43:50it in principle might make sense from
2:43:52the designer's perspective but all these
2:43:55disruptive user experiences lead to very
2:43:57different mental models of these systems
2:43:59and there's this interesting clash and
2:44:01so there is a good way to go about this
2:44:03design synthesis and actually building
2:44:06this concept model uh this concept model
2:44:08or your design concept is really in some
2:44:12sense uh the specification that you want
2:44:15to give to a developer so this is how I
2:44:17think about the system this this is what
2:44:19we need to implement for our XR
2:44:22solution so design space analysis I
2:44:25really like uh this idea of asking
2:44:28questions considering the options that
2:44:31we have for any kinds of questions like
2:44:33how to display something we can display
2:44:34it permanently or appearing and then
2:44:37really uh you know putting out the
2:44:40criteria when would we use which option
2:44:42and what does which option actually mean
2:44:44surveying these options and then finally
2:44:47actually making a a design choice for
2:44:50one of the options so for example here
2:44:52appearing and so there's work on that
2:44:54and I'll provide the reference uh about
2:44:57this design space analysis and this
2:44:59really often happens implicitly among
2:45:01designers we ask questions all the time
2:45:03we consider options all the time so the
2:45:05hard part when it comes to XR is really
2:45:07asking the right questions how to
2:45:09display things how to track things how
2:45:12to travel in VR or in AR how to
2:45:15manipulate things and how to collaborate
2:45:18these are like some of the the big
2:45:20questions that I uh usually come up with
2:45:23or go through when I create a new XR
2:45:26experience and let's just look at one
2:45:28example here so how to display obviously
2:45:30we just had permanent or appearing but
2:45:32when we think about appearing it's not
2:45:34just whether something appears on screen
2:45:36or not in XR we can also place things in
2:45:39the world and so in AR that might be
2:45:42somewhere tied to a physical location or
2:45:45in VR somewhere tied to a virtual
2:45:47location it may not always be visible to
2:45:49the user and may only be appearing when
2:45:51the virtual camera moves it into view we
2:45:54may also have this kind of like
2:45:55compromise solution where once we see it
2:45:58it follows you around so as you move the
2:46:00camera it kind of like travels with you
2:46:03and so these are some of the options
2:46:05that you are considering uh even just
2:46:07for this appearing example uh and how to
2:46:10display things in XR
2:46:26now design thinking as I said is really
2:46:28about a mindset uh building empathy with
2:46:32your users and then trying to define the
2:46:36problem and really
2:46:38innovating in in a meaningful way and
2:46:41that to me is very very important in the
2:46:42XR space the meaningful part uh often
2:46:46comes too short uh and um I was thinking
2:46:50what are four principles that I could
2:46:52convey to you and so maybe you take them
2:46:54with you and you incorporate them into
2:46:56your design thinking and so these four
2:46:59principles uh and they they have been uh
2:47:02I have been shaped in many different
2:47:04ways where I've been you know studying
2:47:07computer science at eth Zurich for
2:47:09example I went to the human computer
2:47:10interaction Institute at Carnegie melon
2:47:12I was teaching there a little bit and
2:47:14now I've been a professor for several
2:47:16years at the University of Michigan
2:47:18teaching user experience design and
2:47:20research and now obviously also arvr
2:47:22courses and so from these various
2:47:25experiences uh I have actually
2:47:27Incorporated these four principles into
2:47:30my design thinking and maybe this helps
2:47:32you as well so here's the first
2:47:34principle the user is not you if you're
2:47:38a designer you really have to think
2:47:41about that design from not like what is
2:47:44what is easy for you and what what are
2:47:46the kinds of things you can detect and
2:47:48recognize and what gestures work well
2:47:51from a computational perspective no you
2:47:53have to think about it from the user
2:47:55perspective and this is the primary
2:47:57challenge so my second Principle as a
2:47:59designer is it's really hard to unlearn
2:48:02and I often have to remind myself
2:48:04because I tend to forget how hard it is
2:48:06to do for example basic web stuff I mean
2:48:09I say now basic web stuff because it is
2:48:11hard for me to imagine this initial
2:48:13stuff or the first time you open up
2:48:14Unity or unreal and and how intimidating
2:48:19some of this software can be and now
2:48:21that I've used these tools for many
2:48:23years um I've learned so much and it's
2:48:26really hard to unlearn and so uh the the
2:48:29way I want to illustrate that point is
2:48:30some of you may have seen this optical
2:48:32illusion before and um maybe you are
2:48:35able to see two people in this optical
2:48:40illusion most of you will see one um if
2:48:45you haven't seen this before you will
2:48:47either see a young lady or an old lady
2:48:51and that really depends on how much time
2:48:53you give yourself and which elements you
2:48:55focus on in this uh optical illusion but
2:48:59let me uh let me sketch out the young
2:49:01woman basically uh is this is her chin
2:49:05her eye this is her hair and she's
2:49:08looking over her right shoulder and now
2:49:11the old woman is uh this becomes the
2:49:14nose you see one eye here and one eye
2:49:17there and this is then the the older
2:49:19lady's chin and this is her mouth and so
2:49:23this is the old and the young lady's
2:49:26hair and so now I've shown both uh these
2:49:32uh women to you and the next time you
2:49:34will see this I promise you you will at
2:49:36least remember oh there are two people
2:49:38in there and then a second later you
2:49:39will be able to see both again and it's
2:49:41really hard for you to unlearn and so
2:49:43it's the same as a as a designer so if a
2:49:46user expresses issues to that seem
2:49:49trivial or like why didn't they know
2:49:51that keep that in mind and just because
2:49:54you know that and you have learned that
2:49:55um so the problem with that is that it
2:49:58actually makes it very hard for you to
2:50:00put yourselves into the shoes of a
2:50:02novice user H and that's just something
2:50:05to keep in
2:50:08mind my third principle is about this
2:50:11difference between how Junior and senior
2:50:14designers approach uh this design and
2:50:17this is an example that I've borrowed
2:50:19uh and a junior designer really usually
2:50:22thinks that is one path and then they
2:50:24will stick to that path and eventually
2:50:27get to that solution now there is no
2:50:29direct uh line from the problem to the
2:50:32solution uh that the started line here
2:50:34doesn't actually exist um and uh but we
2:50:39we we may just like follow uh this lead
2:50:42and then we're we're doing a little bit
2:50:43of iteration and then really stuck on
2:50:46that idea and that is really like how
2:50:48you end up with this kind of path but a
2:50:50more experienced designer would really
2:50:52explore alternate paths so different uh
2:50:55designs and then take each some of these
2:50:58more promising designs further and then
2:51:00explore some of these branches and maybe
2:51:02even go a step back and figure out an
2:51:04alternative to the alternative and then
2:51:07finally get to a really good solution
2:51:10and this is how you should go about
2:51:12design uh be a more senior designer if
2:51:15you will my fourth principle and this is
2:51:19something that I just say from a lot of
2:51:21experience um and when I work with
2:51:23students especially they always think
2:51:26about prototype is just a prototype and
2:51:29and then what's the product and and and
2:51:32so I just say to you everything is a
2:51:35prototype even the latest Exile device
2:51:38that you may hold in your hand right now
2:51:40that is a product it's advertised as a
2:51:42product it is still a prototype we're
2:51:44still very far away from this final
2:51:46vision of XR device and XR experiences
2:51:50Incorporated maybe into our daily lives
2:51:52enriching our activities and so just
2:51:55keep that in mind and don't really draw
2:51:58don't really try to draw the line
2:52:00between prototype and product just be
2:52:02honest so that's my fourth principle
2:52:05that uh everything is a prototype and
2:52:08there is no final product um I use the
2:52:13term final prototype more like as this
2:52:15is the last thing that we prototype it
2:52:17doesn't mean that it's the finished
2:52:19product and so maybe that's a good way
2:52:22to think about uh prototyping and
2:52:25product and also there's really no
2:52:27difference in terms of the methods and
2:52:28tools that they're using to create
2:52:30prototypes and that they're using to
2:52:32create a product I think it becomes a
2:52:34product the more you invest in service
2:52:37and advertising and these are things
2:52:39that you usually don't do with
2:52:41prototypes um and prototypes really
2:52:43should be you should be okay to let them
2:52:45go to throw them away a throw away
2:52:47prototype is still a good prototype but
2:52:49a throwaway product a failed product is
2:52:52really not what you want so Embrace
2:52:55these different methods and tools that I
2:52:58summarized here in this XR design
2:53:00thinking lecture and um hopefully find
2:53:04good meaningful designs that eventually
2:53:07would be regarded a useful and
2:53:10applicable product
2:53:27in this video we're going to talk about
2:53:29XR design ethics now design ethics is
2:53:33really really important uh we we need to
2:53:36really think about that in the space of
2:53:38XR um ethical and responsible design you
2:53:42have actually a a great responsibility
2:53:45as you're designing a virtual reality
2:53:47around your user users you have to make
2:53:49sure that they are safe that they are
2:53:51protected and uh that's hard uh
2:53:54especially given that you don't know
2:53:56much about the environment in which the
2:53:59users are using your applications and
2:54:01when it comes to AR as well you have to
2:54:03really make a lot of trade-offs in terms
2:54:06of how much do you need to know about
2:54:10the environment the people around the
2:54:12users about the users themselves the
2:54:15devices that they're using the context
2:54:17where they using this and so when you
2:54:20have all that information do you really
2:54:23make use of it does it really
2:54:24significantly improve the design of your
2:54:27application now in many cases you won't
2:54:29have access to all that data just
2:54:31because the technology isn't there yet
2:54:33and we always feel like oh yeah we need
2:54:35all that information but we should
2:54:37really think about how much information
2:54:39we need in order to provide a compelling
2:54:42design and our rationale or our goal
2:54:45should be to work with the minimum data
2:54:49rather than trying to you know get it
2:54:51all and then see what we can do with it
2:54:54you really have to figure out how to
2:54:56minimize the risk and how to maximize
2:55:00the safety uh of your users and uh as I
2:55:04was designing this lecture I was
2:55:06thinking about these following
2:55:09questions what when users can't tell
2:55:12what's real and what's virtual
2:55:14anymore what is the role and
2:55:17responsibility of the XR designer how do
2:55:20we hold XR designers
2:55:22accountable what happens when users go
2:55:24against your intended design all these
2:55:27things combined I was thinking what are
2:55:29good ethics principles of XR design uh
2:55:33this is a space that I'm just thinking
2:55:35about a lot and there are not so many
2:55:39really good guidelines so there's not
2:55:41like a a a a list of 10 things to follow
2:55:44and it'll be perfect no there's an
2:55:47intuition that you have to develop as a
2:55:49designer uh you really have to know the
2:55:51Technologies you really have to
2:55:53understand the design space and then you
2:55:56have to make really really good choices
2:55:59in the interest of the users so I wanted
2:56:02to start with design ethics principles
2:56:04and quite fundamentally there are like
2:56:06three autonomy so the user is in control
2:56:10at all times transparency the user is
2:56:13informed about how to control or be in
2:56:15control and stay in control and safety
2:56:18the user is protected from danger risk
2:56:21and injury so autonomy transparency and
2:56:24safety these are like the really high
2:56:26principles if you look at existing
2:56:28design guidelines like from all the main
2:56:31vendors safety and that is both the
2:56:35physical but also the psychological
2:56:38safety um those are really values that
2:56:41are held up very high is highly it's
2:56:43really emphasized but it's hard to find
2:56:46concrete guidelines that help you make
2:56:49good decisions and
2:56:51unfortunately um it is often through
2:56:54misuse of a design that we learn about
2:56:56the negative consequences of our design
2:56:58and by misuse I just mean that's
2:57:01something maybe you haven't considered
2:57:03in your design process so we have to
2:57:05think about how we approach design more
2:57:08holistically so one way to think about
2:57:11design or responsible design in this in
2:57:14this case is to not just focus on how to
2:57:18best use your solution so improve and
2:57:20optimize the design of your solution so
2:57:22that is really good for users um and
2:57:26following design patterns established
2:57:28design patterns or helping by creating
2:57:30new designs and shaping new design
2:57:33patterns now we should also consider how
2:57:36your solution could be misused so the
2:57:39dark patterns of design so now the
2:57:42problem with both design patterns and
2:57:44Doc patterns is that they're both still
2:57:46really emerging now there's not like a a
2:57:49list of these patterns and then these
2:57:52dark patterns and be careful um it's
2:57:55it's not like that I mean here's a
2:57:57here's a way to think about it h you
2:57:59need to think
2:58:01holistically so you need to think about
2:58:03the context the situations and the
2:58:04conditions in which the users are and in
2:58:06which your experience that the one that
2:58:08you're designing will be run by the
2:58:11users you need to think about the tasks
2:58:13that you want your users to be able to
2:58:15do what kinds of actions or interactions
2:58:17can they do what what are the kinds of
2:58:18the events that might be generated by
2:58:20the system and how might users respond
2:58:23to those kinds of
2:58:25events then obviously you need to really
2:58:27think about the device cameras the
2:58:29sensors and the algorithms now as a
2:58:32designer um you're designing for a kind
2:58:34of like a a a device and you could say
2:58:36that you are actually a user of that
2:58:39device since you're not necessarily
2:58:41improving the algorithms on these uh
2:58:44devices and platforms but you can still
2:58:47be careful and you can still make good
2:58:51decisions in terms of the devices that
2:58:53you build on so we also need to think
2:58:56about data collection processing
2:58:58management pipelines especially as a
2:59:00researcher you really aim to get as much
2:59:03data as possible also obviously as a
2:59:05designer working for a business uh
2:59:08because information is key um but here I
2:59:11would really encourage you to think
2:59:13about do we really need all that data so
2:59:17Frameworks libraries and tools that you
2:59:19are using for application development
2:59:21that is really something that you are
2:59:23making active choices and so you should
2:59:27again know what's going on behind the
2:59:28scenes and make informed decisions then
2:59:32obviously the user interface and
2:59:34interaction design um this whole course
2:59:36is really focused on user interface and
2:59:38interaction design and U but this is not
2:59:40the only place where we really need to
2:59:43pay attention and make sure that we are
2:59:46for example not in danger uring our
2:59:48users not promoting weird um
2:59:52interactions and then not only at the
2:59:54user interface but obviously also at the
2:59:56data level we really need to think about
2:59:57data ownership the governance so really
3:00:00what happens with this data once
3:00:01collected where it's stored how it's
3:00:04being processed and then also and this
3:00:08is really important future use of that
3:00:10data I mean just because a user might
3:00:13agree to um give you the information so
3:00:16that they can run that XR session and
3:00:19place a couple of objects in their
3:00:21living room or kitchen or
3:00:23bedroom um doesn't mean that you should
3:00:26be using that data uh in the future you
3:00:29as the designer vendor these are things
3:00:32to think about and uh there may be many
3:00:34more but this is actually what I come up
3:00:36with as I think about both the design
3:00:39and the development of a new XR
3:00:43application and a user experience that I
3:00:46think is safe and potentially uh helpful
3:00:51and useful to users uh so what I'd like
3:00:53to do now is actually bring in a
3:00:54different perspective as a researcher I
3:00:57often have like new ideas and some of
3:00:59them might be crazy and a little bit
3:01:03borderline right some of them may
3:01:05involve deception and that's something
3:01:07that's okay in research as long as you
3:01:08debrief and also get informed consent
3:01:11from users now that is something that
3:01:13I'd like to emphasize a little bit what
3:01:15I really find interesting that in
3:01:18research in scientific research at least
3:01:21in North America um which is what I'm
3:01:24most familiar with at the moment uh we
3:01:26do have established ethical review
3:01:29processes um but it's it's really
3:01:32important that you navigate all these
3:01:34things um so to actually so that
3:01:37actually users uh rights so participants
3:01:39study participants rights are protected
3:01:42well one way to think about the ethical
3:01:43review process is that it's really a
3:01:45risk assessment are they users are the
3:01:49users or the participants in this case
3:01:51are they exposed to risk that is more
3:01:55than what they are usually encountering
3:01:57in daily life and so it is a very
3:01:59interesting lens that may help us or
3:02:02maybe you as a designer also to better
3:02:05understand what could be ethical design
3:02:24so I'm going to walk you through ethical
3:02:26review and I'm going to try to apply
3:02:28that process as far as I understand it
3:02:31to XR so when you want to do a new user
3:02:33study you really have to define the
3:02:35purpose the procedure and the duration
3:02:38so that's really key information so what
3:02:42is the purpose of any data that we
3:02:44acquire that's what I would like to ask
3:02:46you here and then also what is the
3:02:48procedure for you to collect that data
3:02:51we you think this through and then also
3:02:53how long do we need to collect that data
3:02:56so is it just one session is it every
3:02:58session and how long do we keep it and
3:03:01do we intend future use and do we tell
3:03:04our users and not in the small print of
3:03:07some kind of really long document about
3:03:10the policy and the terms of use and all
3:03:13that no we probably need to think about
3:03:15better ways of integrating and
3:03:17incorporating those kinds of things in
3:03:18the user interface while if you're a
3:03:20little bit more advanced as a designer
3:03:22you you know the consequences of each of
3:03:25these things but as a user you may not
3:03:28and so is the user really making an
3:03:29informed decision well all they want is
3:03:31run your app in most cases so um but
3:03:35that's not good enough and that from an
3:03:37ethical perspective that's not good
3:03:40enough it has to be informed consent
3:03:43so the other main goal with an ethical
3:03:46review is to really understand whether
3:03:49there are direct benefits to users and
3:03:51uh so what you want to do is actually
3:03:53maximize the benefits and limit the
3:03:55trade-offs so what are the expected
3:03:58benefits to users and are they direct so
3:04:00can they perceive them so what are and
3:04:03I'm going to inject acceptable tradeoffs
3:04:06in realism comfort and
3:04:08safety and are these reasonable uh the
3:04:12third aspect of an ethical review is
3:04:14really a demonstrated effort that you're
3:04:16trying to minimize the risks of your
3:04:18participants so what are the content
3:04:21induced risks to users and associated
3:04:24with that is the question of how to
3:04:26limit the data and any entities that
3:04:28collect process and manage it and by
3:04:30entities I really mean like the
3:04:32algorithms all the steps in the pipeline
3:04:36um all the kinds of different um
3:04:39Solutions uh and service providers that
3:04:41are involved in making you track the
3:04:44gestures and understand the speech
3:04:46commands and all that
3:04:48um really you want to keep that very
3:04:51very small as a researcher I usually say
3:04:54okay the only person or people that have
3:04:58access to the research data is me as the
3:05:00investigator and then my research team
3:05:03and finally I think um and this is maybe
3:05:07a little bit unique here or a little bit
3:05:09of a different view on ethical review um
3:05:12this idea of agency in autonomy to
3:05:15really Empower participants so how do
3:05:17you
3:05:18so how do you enable your users to stay
3:05:20in control well as a designer for
3:05:23example you may really strive for this
3:05:26high level of immersion just because you
3:05:28think virtual reality has to be like
3:05:31really realistic but like uh imagine
3:05:35ways for users to control the level of
3:05:37detail just like you have in any
3:05:38computer game you can usually control
3:05:41the graphics so now this was my version
3:05:43of an ethical review um of XR and now I
3:05:46want to switch gears and actually talk
3:05:48about this idea of doing a design ethics
3:05:51review and provide a couple of ideas of
3:05:53how to approach it and think about it so
3:05:56the first really big thing to think
3:05:58about and there are four in total in my
3:06:01opinion at least is context and
3:06:03situation where is the Exile device
3:06:05being used who is around the user the
3:06:08second important thing is the sensory
3:06:09information and the data that we are
3:06:10collecting so what does the Exile device
3:06:13see what kind of data is acquired so
3:06:15these are really important questions
3:06:17that you should be answering for your
3:06:19design and uh it may really vary
3:06:22depending on context and situation so
3:06:23there's an interesting dependency
3:06:25between the first and the second kind of
3:06:27quadrant here so then we need to think
3:06:30about the processing Pipeline and this
3:06:32is something as a designer you may
3:06:33struggle a little bit maybe because
3:06:35you're using all kinds of solutions that
3:06:37are kind of like a blackbox to you but
3:06:39that's where I would encourage you to
3:06:40really learn more about the technology
3:06:43so what happens on device is a really
3:06:45really important question to be answered
3:06:47so what kind of data is collected and
3:06:50then processed on device and how is that
3:06:53data processed otherwise if not on
3:06:55device well it has to be somewhere
3:06:58online in the cloud and that means it's
3:07:00going to be transferred to a third party
3:07:03I mean from the users perspective it's a
3:07:04third party because that's maybe
3:07:06something they didn't know that party is
3:07:09involved and and so that's uh a little
3:07:12bit of a risk isn't it so what happens
3:07:14in the cloud who manages the cloud what
3:07:17kind of of uh what kind of protection is
3:07:20provided what is the service level
3:07:22agreement uh so these are all kinds of
3:07:24things you need to think about so the
3:07:26fourth big topic of our design ethics
3:07:28review has to be about data ownership
3:07:30and governance so who owns and manages
3:07:33the data and how is the data act and how
3:07:36is the user informed about this and is
3:07:39it explicit or implicit or do we never
3:07:42tell them or have you not thought about
3:07:44it okay so I'll stop teasing you there
3:07:47but really this is something that we
3:07:49should think about and um really really
3:07:52important is that this design ethics
3:07:54review happens continuously throughout
3:07:57design not just at the end when all the
3:07:59decisions have been made so this is
3:08:02something this is a mindset that you
3:08:04have to apply as you're uh making design
3:08:08decisions and forming a rationale about
3:08:10uh those design decisions uh my first
3:08:13example is home decoration uh this is
3:08:17inspired from the Ikea place app and so
3:08:20now we have a family here they're all
3:08:22using their devices and they're deciding
3:08:24to gather what kind of furniture to get
3:08:27and where to place it and the example
3:08:29here is a living room but it could be
3:08:31all kinds of other rooms so let's
3:08:33analyze this a little bit do a little
3:08:35bit of a design ethics review given what
3:08:37we just learned so the context and
3:08:40situation here okay I said it's living
3:08:42room and it's family so let's just say
3:08:45it's the family and every member of that
3:08:47family agreed that they will all want to
3:08:49come together and use the devices in
3:08:51this interesting way and then the
3:08:54sensory information and data well user
3:08:57bystanders and these bystanders are
3:09:00family members we do see Furniture we
3:09:03may be able to infer something from that
3:09:05furniture is it expensive furniture how
3:09:07much Furniture how old furniture so um
3:09:12that is something that is potentially
3:09:13sensitive information and then the
3:09:16geometry yes so the app needs to
3:09:19understand the geometry of the
3:09:21environment and then it can do a basic
3:09:24object detection mostly in terms of the
3:09:26furniture that it knows so in this case
3:09:29probably Ikea furniture that it has like
3:09:31access to as a catalog so but lots of
3:09:34things could go wrong here in terms of
3:09:35like um other kinds of objects being
3:09:38introduced and then
3:09:40misclassification Etc so then we're
3:09:43looking at the processing pipeline so in
3:09:45this example actually the original was
3:09:47marker base but now we have marker less
3:09:49implementations so basically the devices
3:09:52do their own scanning of the environment
3:09:55relatively Limited at the moment in
3:09:57terms of seen understanding finally data
3:10:00ownership and governance so here
3:10:02obviously the data ownership H might be
3:10:05with the app developer I don't actually
3:10:07know who developed Ikea play maybe it
3:10:09was Ikea and then when it comes to the
3:10:12question of how is the user informed um
3:10:15obviously that is through some kind of
3:10:17language terms of use when you like find
3:10:20that application and download it and
3:10:22then a little bit through visualization
3:10:25because for example it'll render a
3:10:27little bit of a pattern across the
3:10:29surfaces that it detects and gives you a
3:10:31sense of what the device sees but don't
3:10:34be fooled that is just a visualization
3:10:37of the kinds of information that the
3:10:39device is relatively positive about
3:10:42relatively sure that this is a surface
3:10:45it still sees many other things that it
3:10:46may not
3:10:47visualized to the user so keep that in
3:10:50mind and that's our little first this is
3:10:53our first little design ethics review
3:10:55for this scenario so now I'm going to
3:10:57Port us into a different scenario we're
3:10:59still kind of like in the same realm and
3:11:02people uh together in a living room for
3:11:05example and now they're playing a
3:11:06collaborative game so one of the big
3:11:08differences in this scenario is that
3:11:10you're not paying as much attention to
3:11:13what kind of things you're looking at
3:11:15when you're when you are designing and
3:11:17decorating your home you're really in
3:11:19control of that device but when you're
3:11:21playing a game and that game makes you
3:11:23like makes like objects appear there and
3:11:26there and even places that maybe you
3:11:28didn't want the camera to see but just
3:11:30to participate in that game you you you
3:11:33do this and obviously your movement is a
3:11:35lot faster so it makes it a lot harder
3:11:37for devices to really sense things and
3:11:40understand it but we're getting there so
3:11:42the other thing I wanted to express in
3:11:44this scenario is now we actually using
3:11:46very different display technology you
3:11:47see a hland user here on the left some
3:11:50kind of hland is actually hland one here
3:11:53and we're using some kind of projection
3:11:56and uh camera based system so a connect
3:11:59here so this is a Pro Cam setup and so
3:12:03that a user even without device can
3:12:05participate and see this kinds of
3:12:07experience and then we see a smartphone
3:12:09user here on the right so all kinds of
3:12:12very different display Technologies and
3:12:14each of these have very different
3:12:16tracking technology is involved now that
3:12:19is interesting I want to bring in
3:12:21another consideration which is like if
3:12:24these were in different environments and
3:12:27now each user in their own home are
3:12:29participating in this collaborative game
3:12:32what does that mean for the data and now
3:12:35there is no way to handle it locally
3:12:38well this kind of
3:12:39information gathering and processing and
3:12:42coordination among these devices so to
3:12:44get them into the same coordinate frames
3:12:46and making sure that these devices can
3:12:48collaborate that already requires some
3:12:50kind of server side infrastructure but
3:12:53that server could still be part of the
3:12:54home network now in this remote setup
3:12:58that's
3:12:59different now we have to go online and
3:13:02now we have to leak no process and now
3:13:06we have to transfer that data between
3:13:09participants between users and now that
3:13:11introduces interesting new issues so for
3:13:15this scenario I'm not going to do a
3:13:17design ethics review I just wanted to
3:13:19point out a few differences and I'm
3:13:21going to do the same for two more
3:13:24variations of a second scenario so the
3:13:25second scenario imagine a classroom
3:13:28lecture so this is now more like a
3:13:30public context so you're like going to a
3:13:33lecture hall maybe you're a student or
3:13:36you're going to a conference and there's
3:13:37a speaker up front and so that speaker
3:13:40wants you to participate in this really
3:13:42cool AR experience where you can see the
3:13:45solar system and you're using all kinds
3:13:47of devices here the different users and
3:13:49they're kind of engaged so pretty cool
3:13:51so public space all kinds of different
3:13:53devices both handheld hand and headworn
3:13:57could also be kind of ma based and now a
3:13:59different variation of this scenario H
3:14:02it's now a team meeting so you're going
3:14:04with team members into a smaller uh room
3:14:10it's could be um some office space or
3:14:14some public building or even your home
3:14:17uh so how would that change how you
3:14:19would use this AR application so things
3:14:22to consider here are this could be a
3:14:25more private or still a public
3:14:27environment you may or may not know
3:14:30those users and you don't know what
3:14:32kinds of applications that have
3:14:33installed on these devices and also you
3:14:37need to think
3:14:39about uh these different devices coming
3:14:42together here and all that to support a
3:14:46human Anatomy application so that we can
3:14:49learn about human anatomy um so all
3:14:52kinds of interesting things to consider
3:15:10here so the last thing I wanted to throw
3:15:13in is the consideration of design impact
3:15:16and that's something you have to
3:15:17consider at different stages so you have
3:15:19to think about it before during and
3:15:22after the users are experiencing your
3:15:25experience so your application so before
3:15:28and this could already be in the App
3:15:30Store if that's the distribution channel
3:15:33uh you should really set the
3:15:35expectations uh of the users for that
3:15:38new reality so whatever it is that
3:15:40you're putting them into some fancy or
3:15:43scary virtual world or how you may
3:15:45change manipulate
3:15:47modify diminish whatever you do to their
3:15:50real world in an augmented reality
3:15:52version of your
3:15:53application and so you need to prepare
3:15:55them so that they know and then
3:15:58obviously during the experience you
3:16:00really need to balance the realism
3:16:02comfort and safety and uh the trade-off
3:16:05should really be around the realism and
3:16:07what you want to maximize is safety
3:16:11comfort and then maybe realism you
3:16:13didn't even see that so maybe realism is
3:16:15there finally you need to think about
3:16:18what happens after they have tried out
3:16:19your experience so the impact of your
3:16:21experience afterwards it may actually
3:16:23shape uh the next kinds of interactions
3:16:26that they uh encounter in daily life or
3:16:29with another kind of XR experience and
3:16:33yes okay maybe you feel like your
3:16:34responsibility ends there but you really
3:16:37need to think about that as well so help
3:16:39users transition back to the old reality
3:16:42and this could be through some kind of
3:16:44debriefing or even during the experience
3:16:47asking them to take breaks to remind
3:16:49them of the real world because being
3:16:52addicted while it might be good for
3:16:54revenue is not really good from a design
3:16:58perspective especially from an ethical
3:17:00design perspective okay and now I wanted
3:17:03to give three examples from my own
3:17:05research and two of these examples are
3:17:08actually about projects that I've worked
3:17:09on where we had to make very interesting
3:17:11design ethics decision and the third
3:17:13example is one where I decided not to
3:17:15pursue it I just just because from an
3:17:18ethical perspective I really didn't feel
3:17:21this was a good thing to do the first
3:17:23example is from a mixed reality crisis
3:17:26simulation and Tage training app and I'm
3:17:28not going to show any pictures here but
3:17:30we really had to deign how realistic
3:17:33should we make the injuries now that was
3:17:35an interesting project we were working
3:17:37with a professional in nursing also a
3:17:39professor here at the University of
3:17:41Michigan really understood all that and
3:17:44she actually helped us create the
3:17:46virtual injuries by using a mannequin
3:17:49and some kind of like blood uh and then
3:17:52taking realistically looking photographs
3:17:55and sending me all these pictures
3:17:57basically the task that I said the
3:17:59students was like place these injuries
3:18:02so that they appear realistic and
3:18:04anatomically correct and so I'm not
3:18:07going to actually show any of these
3:18:08materials here as I said but um I really
3:18:12thought we need trigger warnings even as
3:18:15I was working with my students students
3:18:17and we really had to make the trade-off
3:18:19between making it super realistic and
3:18:22making it just help uh inform the triage
3:18:25decision making which follows a specific
3:18:28protocol so we were going so we were
3:18:29designing two levels of injuries and we
3:18:31allowed users to pick it and uh one
3:18:34important consideration here though is
3:18:36that uh if you're a medicine student and
3:18:39the further you are along in your
3:18:41studies the more things you have seen
3:18:43and now with our design what we could
3:18:46enable able is early uh students in
3:18:49their career as well as more advanced
3:18:51students who have obviously a higher
3:18:53threshold now um work together and that
3:18:57app was still supposed to invoke some
3:19:00stress to make it a realistic scenario
3:19:03because in a disaster you really need to
3:19:05be calm and that is something you need
3:19:08to train and if if we just show kind of
3:19:12like comic style injuries you're not
3:19:15going to feel stressed and so that was a
3:19:17very interesting uh scenario to navigate
3:19:21and the second example is from a master
3:19:22thesis recently computed in my lab and
3:19:25that was with a really cool student who
3:19:28uh came from Colombia and also wanted to
3:19:30do a little bit of a field study there
3:19:32in a museum of memory and the idea of
3:19:35this project was to build empathy but
3:19:39avoid revictimization as they're using
3:19:42augmented reality to bring some of these
3:19:43stories and The Narrative of those
3:19:45stories close closer to the users now uh
3:19:49without going into too much detail that
3:19:51is a really really difficult to do that
3:19:54is a really difficult thing to do and in
3:19:57this project we really worked together
3:19:58with the staff of that Museum of memory
3:20:01who had significant experience and the
3:20:03important thing is we didn't want to
3:20:05design this naive AR application where
3:20:07just like Place images everywhere make
3:20:10it immersive as much as possible no this
3:20:13is really something where you have to
3:20:15understand the technology very well well
3:20:16that's why I was so fascinated by the
3:20:18project and we had to make very very
3:20:20careful decisions in the end it was a
3:20:23very very complex project and and we are
3:20:25still um processing uh some of these
3:20:28research findings and thinking about how
3:20:31to communicate that research to the
3:20:33community but I wanted to include it
3:20:35here because obviously it was a recent
3:20:37project and and work that uh my student
3:20:39did and I was very proud of her and that
3:20:41work and so I wanted to mention it here
3:20:44the third project is one that I did not
3:20:46Pres student walks into my office really
3:20:48great student we both learned from each
3:20:51other that we had lost loved ones uh
3:20:55actually deceased family members and
3:20:58what if we could bring them back uh how
3:21:00could we how could we even bring them
3:21:03back what would it be like to bring back
3:21:04a deceased family member in VR and uh
3:21:08that is a project that I was like wow
3:21:13okay I mean I can probably do this I
3:21:17mean I can think about the technology
3:21:19and is it going to be 360 or is it going
3:21:21to be virtual reality or is it going to
3:21:23be augmented reality and what are the
3:21:25consequences of mixing that experience
3:21:28so bringing back the deceased person and
3:21:30making them appear in real life so in
3:21:33augmented
3:21:34reality and um yeah so I didn't want to
3:21:38do this and I think and the student
3:21:40understood and we made the decision
3:21:42together and I think um that is fine now
3:21:45we have not done this but as I said
3:21:47somebody else has and now I feel like
3:21:50somebody else has to live with that
3:21:53decision and uh the other way to look at
3:21:56this is like had we done it maybe we
3:21:59could have explored that project in a
3:22:03really good way make very good ethical
3:22:06decisions and then inform or somehow
3:22:09influence that other project that
3:22:11happened without us so that's maybe the
3:22:13one regret that I have here so this last
3:22:15part of the lecture was a little you
3:22:18know thinking about things and uh I hope
3:22:23um that you understand the importance of
3:22:26the topic of Ethics I hope that you
3:22:29forgive that it's not like this
3:22:30checklist it's it's it's not that easy
3:22:33and so I want to go to my last Slide the
3:22:36things we should keep in mind as we
3:22:38think about design ethics and exod
3:22:40design ethics specifically so in every
3:22:42decision you have to make consider the
3:22:45cost benefit ratio to the users okay and
3:22:48especially the cost so with all design
3:22:51decisions you make carefully weigh the
3:22:53tradeoffs don't make critical decisions
3:22:55for users enable them to make those so
3:22:59you should really think about how do you
3:23:01actually provide for a uh customization
3:23:05support how do you communicate
3:23:07customization options to users and how
3:23:09do you enable and Empower your users to
3:23:13make those decisions rather than you
3:23:15making all the decision decisions around
3:23:17the immersion and the level of realism
3:23:19and how you blend objects into their
3:23:21environment and the kinds of information
3:23:25and handes and speech and whether you
3:23:28identify all the people around them and
3:23:30what you do with that information no you
3:23:32think about how to communicate that
3:23:35users and that is actually a really
3:23:37really difficult design problem and now
3:23:40we're talking all right obviously then
3:23:43you need to stay aware so always learn
3:23:46and these ethics considerations must
3:23:48evolve as EXA Technologies evolve so in
3:23:52a few months or years from now we may
3:23:54know better and maybe I have to really
3:23:56redo that lecture but this is the best I
3:23:59could do at this stage now think a
3:24:01little bit about my background I'm a
3:24:04computer scientist I am trained in human
3:24:07computer interaction but more on the
3:24:09ethical side and if somebody else has a
3:24:12very different opinion uh about EXO
3:24:15design eth
3:24:17and I would really like to talk to that
3:24:20person I'm sure there are both good
3:24:22elements in what I talked about and
3:24:24elements that you could uh critique and
3:24:27I'm open to critique but I hope that I
3:24:30provide at least some kind of thought
3:24:32framework U for you to navigate that
3:24:34space which is a really difficult space
3:24:37so build these ethics considerations
3:24:39directly into your workflow adapt both
3:24:41so really not an afterthought okay
3:24:44directly part of your process
3:24:46embedded in every decision you make and
3:24:50then finally and this is more like
3:24:52speaking to you guys out there
3:24:55developing technology and then you guys
3:24:58out there designing for this technology
3:25:00you really have to know the TAC and
3:25:03don't overpromise with what you say Your
3:25:05solution can do okay I've been in so
3:25:08many discussions around the things that
3:25:10devices see and how tracking actually
3:25:14works and while it's very scary scary so
3:25:17for example to use a VR device like that
3:25:19in the background with all the cameras
3:25:20that it has for inside out tracking
3:25:23while that may be very very scary for
3:25:25passers by the average person that may
3:25:28not fully grasp the reality of these
3:25:31devices and understand what's going on
3:25:33technically you have that responsibility
3:25:35to inform the users and the non-users
3:25:38around that device um that something is
3:25:42going on and this idea of like some kind
3:25:44of red blinking light so that you know
3:25:47that we are recording and the device is
3:25:49seeing something that is the most naive
3:25:52way to approach this topic so that is
3:25:55not the solution and the other thing you
3:25:58really have to be clear about I've seen
3:26:01and the other thing I wanted to touch on
3:26:02is like I've seen so many promotional
3:26:04videos that really annoy me just because
3:26:07they don't advertise the the product
3:26:11correctly the device can actually not
3:26:14see anything when you have your hands
3:26:15down here
3:26:16and the field of VI is like this rather
3:26:18than like that and whatever they suggest
3:26:21and how they have composed that video
3:26:24and um yeah uh there's also something
3:26:27wrong there from an Ethics
3:26:30perspective and uh yeah so don't
3:26:33overpromise and then as a researcher
3:26:36this is really clear I will never get my
3:26:37stuff published unless I really talk
3:26:39about the limitations and in order to
3:26:41talk about the limitations you have to
3:26:43explore the limitations so every
3:26:44solution has limitations
3:26:47and you really need to learn about them
3:26:49and then you need to be upfront about
3:26:51them you can't fix them all some of
3:26:54these limitations may have to do with
3:26:55the technology that you're choosing or
3:26:57somewhere deep down in some kind of
3:26:59framework that you're relying on or some
3:27:03platform and um yeah so it's just
3:27:06important that you know and that you
3:27:09consider this and so this is where I end
3:27:11with my lecture on XR and design ethics
3:27:15so this is really a topic that is
3:27:17evolving a lot of people have talked
3:27:19about it from the user perspective
3:27:22realism and you know all the kinds of
3:27:25side effects and is it good for some
3:27:28kind of therapy and what are the
3:27:30long-term use effects and what are the
3:27:32effects on children and I wanted to do a
3:27:34lecture uh that helps us as designers
3:27:37think through this now I probably didn't
3:27:39get this super right but if I
3:27:41accomplished something I made you think
3:27:44more about the design decisions that
3:27:46you're making and also trying to hold
3:27:48them up to some kind of Ethics standard
3:27:51uh a code of conduct that we really
3:27:54still have to develop together in this
3:27:56field and uh I hope that um some of my
3:28:00efforts and then obviously all the other
3:28:02amazing people out there that are
3:28:04working on accessibility and other kinds
3:28:07of issues around privacy and security uh
3:28:10that we all coming together in this and
3:28:12we're finding really good uh ethical
3:28:15Frameworks that help us think through
3:28:17those things and avoid making mistakes
3:28:20when it comes to design
3:28:39decisions so xra design guidelines is a
3:28:42really really important topic um and the
3:28:44problem that we have right now is that
3:28:46EXA design is really a moving Target so
3:28:49many new kinds of devices coming out and
3:28:52some of them are changing really the way
3:28:54we interact now that we have hand
3:28:57tracking available and eye tracking as
3:28:59input uh on both AR and VR devices it is
3:29:03uh it is very interesting uh where this
3:29:06leads in terms of design Evolution and
3:29:08so I really struggled a little bit when
3:29:11I wanted to put this lecture together
3:29:14here I was hoping
3:29:16uh that I can sh share something
3:29:19practical and really wanted to have
3:29:21concrete guidelines as much as possible
3:29:24and I've looked at all the guidelines
3:29:25that are out there and we're going to
3:29:27review some of them here as well but the
3:29:31issue is really that um there is no
3:29:35perfect guidelines out there so uh the
3:29:38moral of the story and the essence of
3:29:41the message here in the end is that you
3:29:43you just have to try it out and you have
3:29:45to think from the perspective of a user
3:29:48uh not just design and implement the
3:29:51things that you think are easy as a
3:29:52designer no you really have to look at
3:29:54what what is um what is Meaningful what
3:29:57is useful
3:29:58usable um and also consider long-term
3:30:01use and um well hopefully develop an
3:30:04intuition over time that allows you to
3:30:07determine whether something is a good or
3:30:09a bad
3:30:11design but yeah that's just like my my
3:30:13perspective I've still tried to put a
3:30:16few guidelines for us to get started and
3:30:19and we can use these guidelines to
3:30:21review each other's designs um
3:30:24throughout the course and uh but think
3:30:27of it really as a a source a starting
3:30:29point a source of guidelines that can
3:30:32evolve hopefully over time and and
3:30:34that's really how design works so we're
3:30:36still in this part of the course uh
3:30:38where we're more on the design thinking
3:30:40side so what's the best way to learn
3:30:42about EXA design so I think it's uh
3:30:44actually studying design guidelin so
3:30:46what is the knowledge that is out there
3:30:48and which of these guidelines are really
3:30:51like something we have to adhere to and
3:30:53which of these we can bend to
3:30:54potentially innovate in design uh we
3:30:57need to learn how to critique existing
3:30:58designs and actually that critique will
3:31:01really involve um a lot of
3:31:05constructive criticism and suggestions
3:31:07for
3:31:09improvement a lot of this should be
3:31:11based on guidelines as much as possible
3:31:13it should be objective and um finally we
3:31:16want to be able to create our own design
3:31:18so I'd say in all of these efforts it's
3:31:21really important to learn about
3:31:23guidelines so when we have a look around
3:31:25there are actually quite a few
3:31:26guidelines out there and and I think we
3:31:28can distinguish them into like four
3:31:31types of guidelines or sources uh those
3:31:34by vendors that I think are mostly
3:31:36platform driven those by designers uh
3:31:39the good thing about those is that
3:31:41they're really user oriented and they're
3:31:43coming more from a human computer action
3:31:47perspective then there are those created
3:31:49by practitioners so really
3:31:51experience-based not necessarily uh with
3:31:55uh scientific rigor and that's why I
3:31:57would draw the line to those established
3:31:59by researchers um usually empirically
3:32:02derived to some kind of uh lab based
3:32:04user studies and controlled experiments
3:32:07and um so there are all those guidelines
3:32:11out there and I'll provide references to
3:32:13some of the promising examples
3:32:16in this video I'm going to review a few
3:32:19of the vendor based guidelines and then
3:32:23I'm going to share some of my own
3:32:26experience and insights and uh we're
3:32:29going to actually go on a little trip uh
3:32:31as part of a recent experience where I
3:32:32went to a remote conference i e virtual
3:32:36reality conference and so I thought it's
3:32:38actually quite cool to look at how
3:32:42researchers and designers Implement um
3:32:45current
3:32:46virtual worlds for example to support
3:32:48something like a remote conference
3:32:50attendance and so we're going to look at
3:32:52that and that's one of the examples I
3:32:53have here as well so let's look at a few
3:32:55guidelines first uh I picked Oculus
3:32:58guidelines uh they're Mo mostly focused
3:33:01on VR uh then I picked uh some from
3:33:04Google they are focused on AR core and
3:33:06so handheld AR and then I was also
3:33:10looking at uh other sources Apple um
3:33:14Microsoft
3:33:16and uh and then Mozilla and so the apple
3:33:19and the Microsoft ones are actually
3:33:21quite good but for copyright reasons
3:33:22cannot show them here I'll link them and
3:33:25I added the Mozilla ones because they
3:33:26are really in the realm of web XR and
3:33:29this idea of bridging multiple devices
3:33:31and platforms they're a little bit more
3:33:33technical as well but I think together
3:33:36comprehensively quite a nice set of
3:33:38guidelines uh driven by vendors and
3:33:41device uh creators and then also Open
3:33:44Standards uh in initiatives and
3:33:47institutions like Milla so the first
3:33:49example we're going to look at is here
3:33:52from the Oculus uh guidelines and so I I
3:33:57browsed and selected my device first so
3:34:00let's say I'm going to develop for the
3:34:01Oculus Quest and they're going to tell
3:34:03you a little bit about uh how rendering
3:34:06Works how steroscopic images work um and
3:34:10then um it says things like don't
3:34:14include uh the screen with the HUD that
3:34:16makes lot of sense and don't draw uh HUD
3:34:19elements behind uh scene elements so
3:34:23these are some really good uh tips and
3:34:25this sometimes happens by accident and
3:34:27then I think we're entering a very
3:34:28practical portion here where it talks
3:34:30about um basically the field of view and
3:34:33how Vision Works and both the horizontal
3:34:36so the horizontal field of view and the
3:34:38vertical field of view that that we have
3:34:40and where we see best and what's really
3:34:41a good area and then I think those
3:34:44guidelines focused on hand tracking are
3:34:46really a part where the Oculus
3:34:48guidelines stand out so let's focus on
3:34:51these a little bit and explore them in
3:34:53in more depth so here you see a concept
3:34:55video of how um the App Store might work
3:35:00where you exploring one of the games
3:35:02like beats saber here H using hand
3:35:05interaction and then they're talking a
3:35:07little bit about the principles
3:35:10behind um hand based
3:35:13interaction and they're using ter terms
3:35:16that I like obviously informed by Don
3:35:18Norman so affordances signifiers and
3:35:21feedback some of the core design
3:35:23principles that we often teach or at
3:35:25least hear about when it comes to user
3:35:27experience and interaction
3:35:29design uh for example then talking about
3:35:32how to constrain inputs to improve
3:35:35usability so the degrees of freedom uh
3:35:38which can actually be very useful but
3:35:40sometimes you don't want to move objects
3:35:41along all axes and talking about uh some
3:35:45of those things also sending this
3:35:47reminder that hands aren't controllers
3:35:50and so you should design differently for
3:35:52hands as you if you previously did
3:35:55controller based VR interaction then
3:35:56there's quite a few things you need to
3:35:58adapt and really cool illustrations here
3:36:02so overall quite a few helpful tips and
3:36:05very useful illustrations talking about
3:36:08both their principles and uh interaction
3:36:12Styles and then also talking about user
3:36:15interface components uh really bringing
3:36:18it down from the theory and to the
3:36:22practice and then specifically for the
3:36:26Oculus platform which I think is pretty
3:36:29useful um even though it is obviously
3:36:32vendor driven and platform driven uh
3:36:36many of these things could still
3:36:38translate to other platforms as well so
3:36:41our next example is augmented reality
3:36:43guidelines uh from Google
3:36:46AR core I really like those because uh
3:36:49really useful illustrations and to the
3:36:51point um telling us a little bit about
3:36:54how AR core works and how to think about
3:36:56environmental
3:36:58limitations uh both in terms of like
3:37:00obstacles that could be there for users
3:37:02and also where where the platform has
3:37:04trouble actually detecting and
3:37:07recognizing uh different types of um
3:37:10surfaces so sometimes to do with
3:37:13lighting or reflection really that makes
3:37:16it hard and then a few practical tips
3:37:19regarding how to design an AR experience
3:37:22are you designing for tabletop are you
3:37:23designing for a room scale or a world
3:37:27scale experience um and then ideas of
3:37:32how to promote interactions and
3:37:34communicate um where interactions can
3:37:36take place so surface detection and
3:37:38visualizing uh surfaces and then also
3:37:41adapting to the available play area
3:37:44talking about how to promote um well how
3:37:48to promote users uh exploration and
3:37:52potentially having fun talking about
3:37:54different types of movement and how you
3:37:56actually have control a little bit
3:37:59through Visual cues you really have even
3:38:02though obviously AR and VR are really
3:38:04about agency and giving autonomy to
3:38:06users as a designer you still have a lot
3:38:09of influence over how users move and
3:38:12interact and you can drive that through
3:38:15visual cues and audio cues and you can
3:38:18also combine these in useful ways to
3:38:21address accessibility issues and as I
3:38:24said safety and comfort is really at the
3:38:26core of most guidelines and what I like
3:38:28here about some of the ones from Google
3:38:31is that they illustrate really to the
3:38:33point like what are the things you want
3:38:35to avoid don't have people move
3:38:37backwards for example and talking about
3:38:39realism and how to achieve more realism
3:38:41through modeling texturing and then
3:38:44material design and um really good tips
3:38:46also for XR designers who don't
3:38:49necessarily have a 3D design and
3:38:51modeling background um cool examples um
3:38:55in comparison so really quite uh useful
3:38:58reference here in terms of design
3:39:00guidelines and to the point our next
3:39:03example is a a specification and some
3:39:06guidelines around web XR in this case um
3:39:09authored and edited by Mozilla and so
3:39:12the web XR specification is still
3:39:14evolving and even when you look at this
3:39:17uh draft uh you will still find portions
3:39:19that are
3:39:20Unwritten um but really use for
3:39:22illustrations around how tracking Works
3:39:25how uh headsets uh for VR work
3:39:29stereoscopy and um designing for virtual
3:39:33reality and uh physical risks um so
3:39:37motion sickness or virtual reality
3:39:38sickness they call it here and physical
3:39:40risks and um earlier versions of these
3:39:43guidelines had really concrete value in
3:39:45terms of like where you should place for
3:39:48example text and uh really emphasizing
3:39:51the fact that you shouldn't move um the
3:39:53virtual camera unless the user really
3:39:57wants it and then thinking about fading
3:39:59in and out to uh or blurring the
3:40:02periphery in order to reduce motion
3:40:05sickness what I really like about uh
3:40:08this document here is the approach it
3:40:11takes uh towards crossplatform and
3:40:14thinking about both AR and VR uh and
3:40:17then coming down to basically what's
3:40:19common which is virtual cameras and how
3:40:22you manipulate the view and how you can
3:40:26basically achieve kinds of effects that
3:40:29are similar to film making with good
3:40:31illustrations so it actually helps us
3:40:34understand a lot of the material better
3:40:36um so I think uh this is a unique aspect
3:40:40to some of these guidelines um a little
3:40:43technical maybe for the Taste of some
3:40:46designers but overall I'd say pretty
3:40:49good and still evolving uh guidelines
3:40:52helpful for crossplatform web- based AR
3:40:55and VR so this is going to be a very
3:40:57useful resource for us uh in the future
3:41:00as well so these were just uh three
3:41:03examples uh one FOC one from Oculus
3:41:05focused on VR and one from Google AR
3:41:07core focused on smartphone based
3:41:09handheld AR maress in this case and then
3:41:14also in example from Mozilla which is
3:41:17really at this intersection of platforms
3:41:19and focused more on the web XR standard
3:41:22as I said a little bit more technical so
3:41:24just reviewing these guidelines uh we
3:41:26can see a lot of the interesting
3:41:29challenges that there are right now uh
3:41:31about these guidelines these were all
3:41:33relatively practical and to the point
3:41:35examples I've seen a lot more vague
3:41:38guidelines so here's the trouble that I
3:41:40have with a lot of the XR design
3:41:42guidelines and again keep in mind that I
3:41:44showed some really good examples here so
3:41:47I'm not critiquing those specifically at
3:41:49the core um we really don't have
3:41:52established best practices they are
3:41:54actually really rapidly evolving still
3:41:57XR design is still a moving Target lots
3:41:59to still learn about it there's a lot of
3:42:01research and design going on in the
3:42:03space we only have very slowly emerging
3:42:06if at all design patterns and then those
3:42:09patterns really often change with a new
3:42:11OS update I've seen this with oculus
3:42:13multiple times for example and then we
3:42:15already have major differences between
3:42:17device platforms so even when you look
3:42:19at VR devices and compare Oculus and
3:42:22Vive guidelines or when you look at AR
3:42:24guidelines and look at the Apple ones
3:42:25and the Google ones for AR kit and AR
3:42:27core you really see uh quite a a a big
3:42:32gap and major differences between some
3:42:34of these platforms so not only between
3:42:36AR and VR as platforms but also between
3:42:40uh different vendors and providers at
3:42:42the core however they all agree that
3:42:44obviously designing for comfort and
3:42:46safety is really really
3:42:48important so I would say there is a
3:42:51problem still with scoping guidelines um
3:42:53just in the examples I shared some of
3:42:55them were quite technical so the Ulus
3:42:57and the Mozilla ones um and uh the ones
3:43:01from Apple and the ones from Google were
3:43:04a little bit more high level like keep
3:43:06the user safe and comfortable um that's
3:43:09not I mean they then went into some of
3:43:11the detail so as a this more like a
3:43:14principle than a guide line and so
3:43:16overall it's a little tricky at the
3:43:18moment to figure out really uh what's
3:43:21the best way to scope and present
3:43:23guidelines many of them are true
3:43:25application or device or vendor specific
3:43:27so in the absence of a lot of really
3:43:31clear and established guidelines I would
3:43:34just say that the best guidelines come
3:43:36from learning by doing and it is really
3:43:39important that you test your designs
3:43:40early and often and learn from your own
3:43:43mistakes realize that you made a mistake
3:43:45when you test it with users and then I
3:43:48think at the core it's still important
3:43:50to uh mind the differences between VR
3:43:53and AR so really designing a virtual
3:43:56world but still taking care of the
3:43:58physical world and then augmenting
3:44:01enhancing uh the virtual world with ar
3:44:04uh and maybe less of a focus on the
3:44:08differences between specific devices and
3:44:11vendors in the end we really need to
3:44:13figure out what's the good what are the
3:44:14good things to do in VR and what other
3:44:16good things to do in AR and what should
3:44:18we really
3:44:32avoid so keeping with this idea of
3:44:34learning by doing I wanted to add two
3:44:36more examples here uh one is actually
3:44:40taking some of the Google a design
3:44:42guidelines and uh putting it into AR
3:44:46allowing you to explore in C2 basically
3:44:50in front of you what some of these
3:44:52guidelines actually mean um it's an
3:44:54interesting example of Storytelling n AR
3:44:57and and for the purpose of learning and
3:44:59educating designers so I really like it
3:45:02here for example illustrating nicely the
3:45:04differences between table scale room
3:45:07scale and then world scale
3:45:19okay and how to communicate the size or
3:45:23the scale of the experience already in
3:45:25the App Store so that people know before
3:45:28they download and run an app and then
3:45:31how to deal with the size of objects and
3:45:34how this is really quite a cute example
3:45:36with his dinosaur that constantly wants
3:45:38your attention um but I mean it's really
3:45:42hard to keep him in uh in view him or
3:45:44her
3:45:45and
3:45:47um um so they also talk about how to use
3:45:52the size of objects and the fact that
3:45:54parts of them are hidden to uh encourage
3:45:58users to move around and explore more of
3:46:01the AR experience they also show how to
3:46:05design a multiplayer AR experience and
3:46:07what to pay attention to and here really
3:46:09had to uh I just like couldn't really
3:46:12see what they were doing on their
3:46:14smartphones these two characters so I
3:46:16had to uh interfere a little bit and go
3:46:19in there in the end turned out they were
3:46:21playing chess and they were exchanging
3:46:23spatial anchors to as a form of
3:46:25calibration to agree on a shared
3:46:27coordinate system and that's something
3:46:28they'll tell you about and I think it's
3:46:30really quite cool so it's really a
3:46:32playful way of exploring design
3:46:34guidelines and actually seeing the
3:46:36effect um as you're learning about these
3:46:37guidelines so I really like um how they
3:46:41have taken these uh guidelines that we
3:46:43looked that we browsed earlier and
3:46:45looked at on the screen and now we can
3:46:48learn about it in the world now here's
3:46:51the example with this idea of promoting
3:46:54users uh movement um allowing them to
3:46:57explore more and creating
3:46:59enticing uh captivating immersive AR
3:47:03experiences just through the size of
3:47:05objects and uh this froggy is really
3:47:08quite cute so another great way to learn
3:47:11about XR design is through some of the
3:47:13examples that are out there there so
3:47:15here we're looking at the hand
3:47:17interactions example as part of the
3:47:19mixed reality to
3:47:21kit uh from
3:47:26Microsoft and so I think it's just great
3:47:28to see an example of all the kinds of
3:47:31widgets and how to design a scene and
3:47:34lay out lay it out in front of the user
3:47:36and then label each of these widgets and
3:47:39uh going through all the design options
3:47:42so these examples are super practical
3:47:45especially when it comes to digital
3:47:47prototyping and implementation so even
3:47:49more concrete than some of the previous
3:47:51examples so this is also focused on hand
3:47:55interactions and really telling you all
3:47:58about the different kinds of ways in
3:48:00which we can use hand interactions to
3:48:02support zoom and scrolling and then uh
3:48:05translation of
3:48:07objects and then hand tracking also to
3:48:10play the piano orth thought that
3:48:12experience is not as great as it looks
3:48:15because it's really lacking the tangible
3:48:18aspect and haptic
3:48:21feedback it's great to see those
3:48:23examples of buttons and relatively
3:48:25common user interface widgets like the
3:48:27slider here and the buttons
3:48:31earlier so it's really great to see how
3:48:34the hand inter actions can be configured
3:48:37and then how for example a 3D model like
3:48:40uh the Earth and Earth's core here can
3:48:43uh be annotated with labels and what
3:48:45that would look like and it's it's
3:48:49really a nice little scene that allows
3:48:51us to explore various examples uh
3:48:53including traditional buttons and more
3:48:56um Unity user interface examples that
3:48:58would also work but they really feel
3:49:00quite different and just seeing them
3:49:02mixed in here also suggest to designers
3:49:05well first of all this is possible but
3:49:07second of all the experience is quite
3:49:09different and maybe you should not mix
3:49:10them although this is a little bit
3:49:12implicit and probably more based on my
3:49:14my experience than exp stated in this
3:49:17example anyway it's a great example for
3:49:20us to learn and explore just like the
3:49:22one I showed earlier Google's AR
3:49:24elements that you can run on your
3:49:25smartphone and this one here runs on
3:49:27holens 2 the mix already took it hand
3:49:30interactions example so having those
3:49:33pre-built scenes and examples is just
3:49:36very good way to learn about AR VR
3:49:38design
3:49:52so over the past few years I've really
3:49:54worked on quite a few AR and VR projects
3:49:57and I was thinking about what kinds of
3:49:59guidelines could I share with you my
3:50:03attempt of making it concrete and
3:50:05hopefully useful and less vendor and
3:50:08platform specific and more focused on
3:50:11really the issues that I see in xra
3:50:13design and I classified it into the dos
3:50:17and the don'ts and we're going to go
3:50:19over them so here they are um really
3:50:23it's like a list you can just explore it
3:50:25and I'll walk you over each of these uh
3:50:27issues so first I talk about offer
3:50:30alternative ways to travel so what I
3:50:33mean by this is like for example
3:50:34allowing users to go to travel to places
3:50:38through a menu or um through gay spas
3:50:41interaction by clicking locations in the
3:50:43world and land marks or different kinds
3:50:46of teleportation but what you should
3:50:48really try to avoid is to move users
3:50:52without their input so you should really
3:50:54not move them unless they indicate they
3:50:57want to be moved it would feel really
3:51:00awkward the other thing that's also
3:51:02weird is if the virtual character is
3:51:03moving and the user uh is actually
3:51:06sitting so it really conflicts with
3:51:09proception so our understanding of what
3:51:12our nerves and senses are actually doing
3:51:15I think you should use interface
3:51:17metaphors as much as possible but you
3:51:19should also pay attention to using
3:51:22modern interface metaphors H and um it's
3:51:26really important not to try to reinvent
3:51:29reality so I know the idea of virtual
3:51:31reality is sometimes to Port you into
3:51:34this completely different space but if
3:51:36that space doesn't adhere to physics and
3:51:39any kinds of things that we know from
3:51:40The Real World what's the point of
3:51:43calling it a virtual reality now you can
3:51:45actually change gravity in all those
3:51:46kinds of aspects but the basic laws of
3:51:49physics should still apply it should
3:51:51still feel um like a world that is known
3:51:55to us and familiar to us I think you
3:51:57should also make as much use as possible
3:52:00of physical affordances to really
3:52:02communicate to users how they can pick
3:52:05up an object and whether an object can
3:52:07be moved rotated and scaled or not I
3:52:10think um after pickup obviously
3:52:13providing feedback but the these initial
3:52:15physical affordances are really
3:52:16important to allow users to confidently
3:52:19explore uh an AR of scene I would
3:52:23caution and advise against this idea of
3:52:25making it a copy of the real world so an
3:52:28example I share with you later is stairs
3:52:31so I've really thought a lot about why
3:52:33we have stairs in virtual reality for
3:52:35example I mean we can teleport we can
3:52:38use a menu to get to places why do we
3:52:41have stairs and I think it has to do
3:52:43with this idea of physical affordances
3:52:45to communicate to us uh there is
3:52:47actually a level that you can go up or
3:52:49down and just helps us read the space
3:52:52and understand the the environment so I
3:52:55get that but um really think about um
3:53:00the amount of stairs we need to navigate
3:53:02and that stairs are actually really not
3:53:04accessible in in the virtual world uh
3:53:07and also not in the physical world as
3:53:09you know you should minimize or reduce
3:53:11the amount of required explicit
3:53:14interaction
3:53:15so a user doesn't really want to like
3:53:16have to speak a lot a lot of voice
3:53:19commands or repeatedly perform a very
3:53:22specific set of gestures I think the
3:53:25hand interaction example I just showed
3:53:27from the Holland mix 2 Kit 2o I think
3:53:30it's great because there's a lot of
3:53:32implicit camera based interactions and
3:53:34then like grabbing a slider and then
3:53:37moving it but being able to like going
3:53:39out of that position so not having to
3:53:41hold it and just continue I think those
3:53:44are great examples of reducing ergonomic
3:53:46effort on users so I think that is
3:53:48really important also don't rely on
3:53:51gestures or speech so I've seen a lot of
3:53:54user interfaces where everything was
3:53:55gesture based and no way of using speech
3:53:59or the other way around where everything
3:54:01is was was voice based commands and then
3:54:04some of the more traditional gestures
3:54:06that you would expect for scaling
3:54:08rotating Etc elements um didn't work so
3:54:11the challenge is really to find the
3:54:13balance between
3:54:15explicit and implicit interactions and
3:54:18then also within the explicit
3:54:19interaction space the the amount of
3:54:22gesture speech and
3:54:24multimodel um so I think that's that's
3:54:27still quite a challenge so here's a
3:54:30relatively standable one but I think
3:54:31it's important put important elements in
3:54:33the foreground avoid the periphery um
3:54:37and so many people talk about this I
3:54:38really just think um it's like you know
3:54:422D screen based design you put
3:54:43everything that's important above the
3:54:45fold you lose people uh the more they
3:54:48have to scroll in interfaces and this
3:54:51applies to arvr as well really don't
3:54:54have users read a lot of text and I'm
3:54:56going to come back to that one because I
3:54:58think it's fundamental and I'll and I'll
3:55:00show you examples of how bad this
3:55:03actually is in AR and VR provide Visual
3:55:05and audio cues to guide users through
3:55:08task don't just rely on one sense really
3:55:10use this mix I've seen a lot of VR
3:55:13experiences that are really Visual and
3:55:16no audio and audio makes such a big
3:55:18difference on the AR side if you're
3:55:20working on a smartphone users often
3:55:23choose to maybe mute their phones so
3:55:25maybe not as important but in general I
3:55:27think you should aim again for this mix
3:55:29and having both uh types of cues really
3:55:33influence how users navigate and
3:55:35interface and and operate an interface
3:55:37and finally but this one should be
3:55:39relatively clear uh anyway but it still
3:55:42happens I've seen it in in a lot of
3:55:43interfaces so don't remap uh the
3:55:46controller buttons uh just because the
3:55:49the interface is now in a different mode
3:55:51it's really difficult uh for users to
3:55:53relearn all the functions and then
3:55:55remember which mode they are in and
3:55:57really try to avoid breaking any kinds
3:55:59of design conventions or going against
3:56:02mainstream unless you're really
3:56:03confident about your design choice in
3:56:06which case I would encourage you to go
3:56:07about it and maybe you revolutionized
3:56:09design that way but most of the time it
3:56:12really doesn't actually for design and
3:56:15makes it really hard for users as
3:56:30well so we've covered a lot of the dos
3:56:33and don'ts uh based on my experience and
3:56:36that experience is coming from doing a
3:56:38lot of research in that space and then
3:56:40also teaching a lot of students about XR
3:56:43design and obviously operating ating a
3:56:44lot of arvr applications myself but um
3:56:48while I strongly believe that these are
3:56:50good guidelines um it's just a starting
3:56:52point uh just based on my experience and
3:56:55maybe there are people out there that
3:56:57disagree but overall as a starting point
3:57:00I think you can apply those guidelines
3:57:03look at your designs and then look at
3:57:05these guidelines as a checklist and see
3:57:07how many dues you have done and how many
3:57:09don'ts you have done and then think
3:57:11about ways of avoiding the don'ts um
3:57:14which I think might be a good way to
3:57:16improve
3:57:17design so still thinking about design
3:57:19guidelines I wanted to look at an
3:57:21example and I chose a recent experience
3:57:24that I had when I was attending the i e
3:57:27virtual reality conference in
3:57:292020 H which was supposed to happen in
3:57:32Atlanta but due to covid-19 had to move
3:57:35to remote and online and we're going to
3:57:37look at some of their spaces that they
3:57:40have created some of their virtual
3:57:41environments that they have created and
3:57:43I have invited my uh student here scha
3:57:46who is going to join me in some of these
3:57:49discussions and Explorations as well and
3:57:51a couple of the things we're going to do
3:57:52is we're going to climb some stairs
3:57:55which I think leads to interesting
3:57:57issues um we're going to explore the
3:58:00world going for a walk together and then
3:58:03you will also see me picking avatars and
3:58:06uh looking at a lot of text and then
3:58:08we're going to review some of the design
3:58:09decisions and I'm going to compile a few
3:58:12more uh tips and maybe guidelines if you
3:58:15will to also avoid those things in the
3:58:19future so here I'm looking down the
3:58:21venue this is the stage that they have
3:58:24and I'm waiting for my student sh ah
3:58:26there she is and I see her and I can see
3:58:30how she is essentially trying to make
3:58:33her way up so they have designed this
3:58:35stage so that you start in the back
3:58:38where you have a little bit of of
3:58:40privacy in some sense you don't hear the
3:58:42speakers and can have a check that and
3:58:44then others won't hear you in that area
3:58:46if they up here on that stage that's
3:58:47good but um you then actually have to
3:58:50climb those stairs and these stairs are
3:58:52really not accessible no matter whether
3:58:54using a keyboard and mouse like I do
3:58:57here in this example or a VR headset
3:58:59which I'll do in some of the other
3:59:00examples in the future so stairs really
3:59:04is an interesting issue to me and
3:59:08something we should think about more and
3:59:09potentially avoid as much as
3:59:12possible and so and I was thinking okay
3:59:16um let's explore a little bit more of
3:59:18this venue and I decided we were
3:59:20attending this conference for a while
3:59:22and then I said to sha hey why don't we
3:59:25go for a walk and explore some of their
3:59:28bigger virtual spaces and so we ended up
3:59:31on this Georgia Tech campus now I
3:59:33haven't actually been to the Georgia
3:59:36Tech campus so I don't know how much it
3:59:38really resembles reality but I could
3:59:41imagine that unless they had a major
3:59:44investment um it probably doesn't look
3:59:46like this so in many ways they
3:59:48reinvented reality here uh with and that
3:59:52makes sense I mean I guess I mean the
3:59:53Big B and and some of these virtual
3:59:56screens floating around so it's an
3:59:57interesting mix of like this is a selfie
4:00:00from Blair one of the organizers and a
4:00:02friend who I hope doesn't mind that I'll
4:00:05critique and review this a little bit uh
4:00:07to look at our guidelines and which of
4:00:09these can apply and then we were going
4:00:12for a walk and and there were some
4:00:15issues again it's a ramp here it's not
4:00:17stairs but um I could teleport with my
4:00:21mouse pretty easily and then uh sha was
4:00:25uh using a Macbook and didn't have right
4:00:28click so she couldn't actually navigate
4:00:30that space uh with mouse and teleport so
4:00:33I had to walk all the way and um for
4:00:37noice user TR is not an office user but
4:00:40for somebody who explores uh such a
4:00:42world and VR for their first time uh
4:00:45figuring out how to move in a space when
4:00:48it's unclear and that you have this
4:00:50option potentially of teleporting if
4:00:52it's available and that is actually
4:00:54really quite uh an important step uh and
4:00:57obviously the more experience you have
4:01:00in these spaces the the more of that
4:01:02experience translates uh between spaces
4:01:05and experiences and so maybe it won't be
4:01:07so much of an issue later but um
4:01:10onboarding is really quite difficult
4:01:12still and and then this is really it's
4:01:15an interesting um example because I
4:01:18really don't get a good sense of what
4:01:20the Georgia Tech campus looks like uh
4:01:23apart from a few Billboards and um text
4:01:26here and um now that text is okay the
4:01:31way it's placed is actually quite well
4:01:33done it's on a billboard it has
4:01:35sufficient contrast it's not just
4:01:38floating I'll have another example later
4:01:40that I think is pretty bad um then we
4:01:43were walk walking around a little bit
4:01:44more exploring some of the other spaces
4:01:47uh ending up at a campfire which was
4:01:49really quite an interesting atmosphere
4:01:51um the way you navigate those spaces uh
4:01:54was interesting so the one of these
4:01:56tiles was actually a portal into a
4:01:59different world um so there's a little
4:02:02bit of inconsistency here in terms of
4:02:03the content that you see on the walls
4:02:05some of it is actually video some of it
4:02:07is text and some of this can be played I
4:02:10guess or not so it's really quite a lot
4:02:12of stuff going on and then as I said one
4:02:15of those tiles actually functions as a
4:02:17portal into different spaces
4:02:19so I do like that um they illustrate a
4:02:23lot of the content here and some of the
4:02:26hard work that really went behind uh
4:02:29creating this uh experience in this
4:02:31online conference essentially so why did
4:02:34I pick this example cuz I think it
4:02:36really allows us to review some of the
4:02:38guidelines this idea of not really copy
4:02:41pasting reality uh offering alternate
4:02:44alternative ways of navigating making
4:02:47use of established interface metaphors
4:02:50and limiting the amount of expit
4:02:52interaction and I was really stuck on
4:02:54the stairs example and uh the platform
4:02:59uh that I was talking about just because
4:03:01it while it was good in terms of
4:03:04physical affordances I I understood that
4:03:06I can I can get to a different part of
4:03:09this virtual world and I can navigate up
4:03:12and down or further away it really
4:03:15reintroduced all these non accessibility
4:03:17issues from the physical world and
4:03:19that's just like a big problem in an
4:03:22experience like this so when you go to a
4:03:24conference like this you really want to
4:03:26learn what kinds of things are actually
4:03:28available what sessions are on and um
4:03:32also here they kind of replicated The
4:03:34Real World um which usually you have
4:03:38some kind of handout or some kind of
4:03:40boards and somebody from the community
4:03:42created this room essentially putting
4:03:45these um program information sheets into
4:03:49into the virtual world here but it's
4:03:52really a lot of text and hard to
4:03:54navigate to the extent that I'm like
4:03:56almost bumping into it and um it really
4:03:59wasn't that helpful after all the main
4:04:03purpose was not to have to leave the
4:04:06virtual reality experience it was a
4:04:08quick fix uh because a lot of people
4:04:10felt like oh always have to jump back to
4:04:12the virtuality website and uh but I just
4:04:15want this information in VR so
4:04:18transitioning in and out of VR is still
4:04:20an issue and it's very disruptive to the
4:04:22user experience so I get it but the way
4:04:25we deal with text here is is not ideal
4:04:29so why do I say this I want to show you
4:04:30two examples uh one is in VR and one is
4:04:33in AR and we're going to talk about the
4:04:35format and the layout of the text and
4:04:37I'm just using some standard text with
4:04:38as little format and layout as possible
4:04:41and then I'll show you the different
4:04:43experiences in in VR and
4:04:45AR so my first example is laurum ipsum
4:04:50placeholder text uh in VR and I just
4:04:53really placed it into the scene and you
4:04:54can see it in stereoscopic view here and
4:04:58it's really hard for me to actually view
4:05:00it uh at at any one time and then this
4:05:04idea of using your hands and fingers as
4:05:07reading a uh only partially works with
4:05:10this really big VR controller here now
4:05:12that may improve a little bit of hand
4:05:14tracking but um it's still kind of an
4:05:18issue so here I'm showing you an
4:05:20augmented reality example where I
4:05:22inserted the same text in the studio and
4:05:25there happens to be this light source
4:05:26that I obviously need to liit the scene
4:05:28and make sure that you can see me on
4:05:30camera but um and this could be just the
4:05:33sun in the in in in the real world it
4:05:35really introduces a lot of issues with
4:05:37reading this text and just seeing it so
4:05:40really you need to pay attention to the
4:05:42environment both the virtual and the
4:05:44physical um the distance of the text is
4:05:47kind of okay I put it at 1 meter away um
4:05:51it shouldn't be further away than 2 m it
4:05:54what is what people say the scale is
4:05:56still a little big uh for my taste but
4:05:59if I'm scale it down it really
4:06:01introduces new issues so text in general
4:06:04there is no really good solution it
4:06:06would need to be adaptive I guess and
4:06:08know a lot about the user and maybe
4:06:10their prescription and all kinds of
4:06:12things so just try to avoid it and again
4:06:14reading aids here like using my finger U
4:06:18my finger appears behind the screen just
4:06:20because we don't actually in this
4:06:21example have finger based or hand
4:06:24occlusion and figure based occlusion and
4:06:27now we're getting there and improving
4:06:28some of this but it's not available on
4:06:30all platforms and even then I don't
4:06:32think this part of improving the
4:06:34technology would actually solve the
4:06:36issue so my last example is avatars so
4:06:39let's think about the design space that
4:06:42we have in terms of I show two examples
4:06:44here one is with permission from my
4:06:46friend Jeremy Nelson um who appears uh
4:06:50in his kind of like reconstructed Avatar
4:06:52all he did was upload a photo to spatial
4:06:55and meeting kind of software so that's
4:06:58what you see on the right and then on
4:06:59the left you see some of the avatars
4:07:02that are used for example in Mozilla
4:07:05hubs um and those are the ones that I
4:07:08chose from when I was actually going to
4:07:10the I EVR conference so when you compare
4:07:13those types of avatars you definitely
4:07:16see that the ones on the left are more
4:07:17abstract but appear to be more
4:07:19consistent and the one on the right and
4:07:21more realistic but there were some
4:07:23issues with the scale and the position
4:07:26of Jeremy so he appeared in my desk for
4:07:28example and I always look down on him
4:07:31which normally isn't the case we are
4:07:34almost the same height so it was a
4:07:37little disturbing and confusing now
4:07:40these things will probably be improved
4:07:41uh in the future but sometimes s the
4:07:44more abstract avatars are actually just
4:07:47good enough and we don't necessarily
4:07:49need to see and need to recreate the
4:07:51whole reality just because we have all
4:07:53these tracking issues which are then
4:07:56even worse um the closer you are uh into
4:08:00reality the higher your expectations are
4:08:02and the more disruptive the user
4:08:04experience is if something goes wrong
4:08:07you know if this little avatar on the
4:08:09left doesn't move properly that's weird
4:08:12but it's okay but if Jeremy's arms or
4:08:14legs Bend in the weird way you don't
4:08:16want to imagine that so here I'm going
4:08:18to show an example of where I chose my
4:08:21avatar I wasn't happy initially I was
4:08:23going to this conference it was a little
4:08:25funny because I did spend quite some
4:08:27time on my outfit and choosing there was
4:08:31one particular character I was looking
4:08:33for uh it was kind of like uh this Iron
4:08:36Man looking metal guy and I think I
4:08:38found him on the last page but this was
4:08:41probably the closest or it wasn't
4:08:43available able that was also
4:08:44disappointing that an avatar that I used
4:08:47to use was not available in this
4:08:49specific experience and so it really in
4:08:52terms of consistency was a little
4:08:55disruptive so these were some of the
4:08:57examples uh that I put together for you
4:09:00to illustrate some of the guidelines I
4:09:02didn't have an example for each and
4:09:04every guideline but take a look at them
4:09:08explore them in a little bit more detail
4:09:10and again as I said use them as a
4:09:12checklist to understand and identify
4:09:16opportunities for improvement to your XR
4:09:18designs and again treat them as a
4:09:20starting point U maybe you disagree
4:09:23that's fine maybe you have a particular
4:09:25example where this guideline um doesn't
4:09:27really apply so the rule doesn't apply
4:09:31um but I think it's important that we
4:09:33have some kind of guidelines these are
4:09:35the ones that formed in my head uh over
4:09:38a few years of working with uh novice
4:09:41designers particularly students
4:09:44and then also based on my own experience
4:09:45like going to this I EVR conference and
4:09:48I don't want to be too negative
4:09:49obviously it was a cool experience but
4:09:52as a designer when you look at some of
4:09:54the artifacts created that way like the
4:09:56viral environments that we looked at and
4:09:58and really they didn't have a lot of
4:09:59time they had to put this together very
4:10:01quickly and I assume little user testing
4:10:05um and then working with this new medium
4:10:07uh and VR for conferences is still
4:10:09pretty new so all that um is obviously
4:10:13clear so I'm not attacking them in that
4:10:15way it's just a great example to to
4:10:18study how much do you want to push the
4:10:20realism uh and this trade-off between
4:10:22making it like a reality and then all
4:10:26the issues that come with it when you
4:10:27don't actually live up to that
4:10:29expectation when you actually break the
4:10:31Illusions uh be it through your design
4:10:34and some shortcomings of it or maybe
4:10:36issues with the technology like limited
4:10:38tracking or other kinds of issues so I'm
4:10:41going to leave this video a little bit
4:10:43more open-ended just because I do think
4:10:45that these guidelines don't need to
4:10:47evolve and at least now we have a way to
4:10:49think about guidelines and how to assess
4:10:51the source of guidelines and uh and when
4:10:55they apply and don't apply so I'm going
4:10:57to leave you there and think about the
4:11:00stairs which still I think it is
4:11:03fascinating to me and um if you get into
4:11:06this if you get to this stage where you
4:11:08start obsessing about some of these
4:11:11design decisions and why things are
4:11:13there are so questioning some of the
4:11:15design decisions then I think it shows
4:11:18that you are starting to think
4:11:19critically about design and that is a
4:11:21really good thing and would make me
4:11:23proud if that's what I have accomplished
4:11:26today with this
4:11:41lecture in this video we're going to
4:11:43we're going to talk about XR design jams
4:11:45now design jams are like hackathons just
4:11:49that they don't last like an entire
4:11:51weekend or whatever you may have heard
4:11:53of hackathons in my mind the design Jam
4:11:56is like coming together uh solving a
4:11:59design task collaboratively over the
4:12:02time span of perhaps like 3 hours um and
4:12:06it's actually really a cool activity now
4:12:09the key idea behind design gems is to
4:12:10bring together stakeholders
4:12:13um that might benefit from whatever it
4:12:16is that you are designing uh so these
4:12:18stakeholders include some of your target
4:12:20users but also maybe members of your
4:12:22design team as well in fact I've done
4:12:24some of the coolest design jams in my
4:12:26own uh research and also work as a
4:12:29professor so in the classroom uh when
4:12:31I've actually involved other clients and
4:12:34members of my own research team so
4:12:36pretty cool and that's what we're going
4:12:38to learn about uh today I'm going to
4:12:40share a few lessons I'm going to talk
4:12:42about both from the perspective of
4:12:45planning and running a design jam and
4:12:47then also participating in a design Jam
4:12:50so we are still exploring how to do XR
4:12:53design and I've already covered design
4:12:56guidelines and critiques and now we're
4:12:59talking about design gems and design
4:13:00gems are really about getting your hands
4:13:02dirty and so creating your own XR
4:13:05designs that's what we're going to talk
4:13:06about so as I said I've done a lot of
4:13:09design jams in my research in the past
4:13:12so this is one of of the first ones I've
4:13:14done when I joined the University of
4:13:15Michigan it was around a project called
4:13:18Pro to AR I really wanted to learn how
4:13:21to make design gems more physical and
4:13:23bring arvr into design gems and actually
4:13:26provide also digital tools that help the
4:13:28students transition more quickly from
4:13:31physical so from paper to digital so
4:13:34into a tool called prot to and I'll link
4:13:37it in the references and resources if
4:13:39you want you can read about it there's a
4:13:41paper about it as well so but that's
4:13:43just one project um we've over the years
4:13:46used it in many different ones so this
4:13:48one is 360 Proto which is essentially a
4:13:51followup of pro to AR so the idea was to
4:13:55still build this system that would help
4:13:57designers create interactive arvr
4:14:00prototypes Now using paper um just paper
4:14:03and specific templates and so this
4:14:05design gem then for example involve
4:14:07paper prototyping and then also arvr
4:14:10devices uh directly as you can see here
4:14:13a preview on a Google card board um so
4:14:16basically you just take a picture of
4:14:19your prototype on the left that you see
4:14:21on the left and then it'll appear in
4:14:23your smartphone when you put it uh into
4:14:25a Google cardboard and we've also used
4:14:28design gems to design more complex
4:14:31systems together with uh students and
4:14:34also clients I show here an example
4:14:36where we invited students actually all
4:14:38of these students then ended up working
4:14:39in my research Labs so it's also a very
4:14:43powerful and useful recruiting tool to
4:14:45actually attract new talent to your
4:14:47research or design teams so let's try to
4:14:49Define design gems a little bit what are
4:14:51the main ingredients so whenever I
4:14:53design a design gem I think of the goals
4:14:57um the tasks so really what do I want my
4:15:00participants to do um what do we want to
4:15:02learn about um what should they deliver
4:15:05so each of these tasks should somehow
4:15:07lead to a clear a clearly specified
4:15:11deliverable um which kind of AR VR
4:15:14devices are we going to involve um what
4:15:17tools do I provide to my um participants
4:15:20so is it just uh physical materials like
4:15:24paper transparency cardboard Play-Doh
4:15:28those are tools is it specific paper
4:15:30templates still tools or is it any kind
4:15:32of digital tools and then pair it with
4:15:35some kind of devices for
4:15:37example um and finally also how much
4:15:39time should we spend so you should
4:15:40really structure it even even 3 hours
4:15:44even though 3 hours sounds like a very
4:15:46long time this is actually over very
4:15:47quickly if you do a good design gym the
4:15:49participants will feel very productive
4:15:51and have a lot of fun and actually uh
4:15:54often times in my classes when I ran
4:15:56design gems students just didn't want to
4:15:58go home so that's a cool little uh side
4:16:02effect as
4:16:03well so what I thought i' share is a
4:16:07little bit of a photo story here of a
4:16:09design G that we've uh run so this is
4:16:12the the one around around
4:16:13prot and that you can read about as I
4:16:16said and uh the the way I describe it in
4:16:19the research paper is obviously quite
4:16:21different because the focus of the
4:16:23research was uh somewhere else but here
4:16:25you see the typical elements like story
4:16:27boards and then we becoming increasingly
4:16:30physical and mixing in Playdoh and then
4:16:33Playdoh to create 3D models and
4:16:36basically creating 3D scenes and the
4:16:38task here was to design a furniture
4:16:40placement app similar to IKEA play um
4:16:44and we also essentially primed the
4:16:46students by showing them existing apps
4:16:49that are similar to what we had in mind
4:16:51and you get a lot of really cool assets
4:16:53that way including for example really
4:16:56nice furniture this sofa even though
4:16:58it's miniature still has a very special
4:17:00place in our lab so now I'm going to
4:17:03take the perspective of an organizer of
4:17:05a design Gem and then later I'm going to
4:17:06talk about what it's like more if you're
4:17:09participating in a design gem so let's
4:17:11talk about planning design gems first
4:17:14so the first thing I always do is
4:17:15actually I start by thinking about who
4:17:17should participate who am I actually
4:17:19designing this design gem for uh
4:17:22obviously you can make it open for
4:17:23everyone but a focus group approach so
4:17:26really like being more selective uh in
4:17:28terms of like what kind of experience do
4:17:30you want uh your participants to have
4:17:33how uh Broad and diverse should it be or
4:17:36should it Target specific technical
4:17:37skills or specific domains um that is
4:17:41going to really significant iFly impact
4:17:43um how the whole design gam is going to
4:17:45go and the data and the results that
4:17:47it's going to
4:17:49produce I also think about what happens
4:17:52before the design Gem and then also
4:17:54after the design gem obviously uh most
4:17:56of my tips will be about the during so
4:17:58while you're doing the design gem but
4:18:00it's important uh to really think about
4:18:03are there any tasks that participants
4:18:05should do before or even after design
4:18:07gems so as an example you might ask
4:18:10participants to fill in a background
4:18:11questionnaire before or after a design
4:18:13gam so that you can focus the design Jam
4:18:16Time really on the actual prototyping
4:18:19and if you do it before um it can also
4:18:21be a recruitment tool that way and allow
4:18:23you to select
4:18:25participants you should really think
4:18:27about the data you want to collect about
4:18:29your participants and um and I usually
4:18:32treat design gems part of my research
4:18:35projects and therefore I get IB approval
4:18:38and so it's very important to think
4:18:40about what kind of data are you going to
4:18:42collect and how you're going to protect
4:18:44um the rights of your participants um
4:18:47how you're going to De identify and
4:18:49anonymized data Etc and while the design
4:18:52gem should feel like a really fun
4:18:55activity there could still be a
4:18:57relatively strict protocol behind the
4:18:59scenes um to make sure that you're doing
4:19:01it in a rigorous uh
4:19:03way so then and now we're getting closer
4:19:07to the event and how do I actually now
4:19:10make sure that everybody I bring in if
4:19:12it's a rather diverse pool of
4:19:14participants how to make sure that
4:19:15everybody is on the same page and the
4:19:18way I usually do this is I usually start
4:19:20a design gem off by showing something to
4:19:23participants giving a demo of some kind
4:19:24of AR or VR technology so that they have
4:19:27at least seen it um or a specific type
4:19:29of AR or VR application and sometimes
4:19:32really even an authoring tool if you're
4:19:34trying to critique something there so in
4:19:36your case if you already have something
4:19:38designed that could be the starting
4:19:40point of the design Jam um and you
4:19:42should really provide references and any
4:19:44kinds of materials to participants so
4:19:47that they can refer uh back to it um
4:19:51because sometimes um not everything is
4:19:53clear in the beginning and then the
4:19:54questions really come throughout the
4:19:56design Jam so it's important to be able
4:19:58to actually go back uh and um have
4:20:02access to those
4:20:05materials speaking of materials it's
4:20:07very important to organize all the
4:20:09materials that you anticipate
4:20:11participants will need to have them
4:20:13ready for them so you actually have a
4:20:15lot of influence over how efficiently
4:20:17and effectively participants will be
4:20:18able to participate and contribute and
4:20:22so that is um really the key here so
4:20:27from doing a lot of design James I
4:20:28realized I really have a lot of
4:20:31influence on how quickly and effectively
4:20:34participants are going to navigate the
4:20:36tasks and so it's it's really good to
4:20:38plan up front and have those materials
4:20:40ready for them so for example here are
4:20:42all the materials that we provided to
4:20:44participants for this furniture
4:20:46placement uh n AR design Jam uh so there
4:20:51were there was plain paper no particular
4:20:53templates but we encouraged participants
4:20:56to think about uh the scenarios and
4:20:58sketch out storyboards and wireframes
4:21:01but again no particular templates um at
4:21:04that point yet and then after some
4:21:06period in the design gem we introduced
4:21:08Play-Doh so that participants were more
4:21:11focusing their attention on creating and
4:21:14crafting 3D models uh using Playdoh not
4:21:17actual 3D modeling tools and that
4:21:19allowed everybody to bring in uh their
4:21:22their fun childhood fun experience if
4:21:25you will and was really fascinating to
4:21:27observe in that particular design gem
4:21:29was how participants were able to do the
4:21:313D modeling in parallel to some of the
4:21:34discussion around uh how to design this
4:21:36mobile AR app and most of the discussion
4:21:39was really about userflow and um how to
4:21:43transition between screens and so 3D 3D
4:21:46modeling if you have ever done it it is
4:21:49actually a really intense task it's not
4:21:52something you usually do in parallel to
4:21:55uh discussing uh some of the other key
4:21:57things so that was quite fascinating in
4:21:59this design jam and it's one of those
4:22:01examples that I'd like to share because
4:22:03it really shows the value of design Gams
4:22:05to me this is something I didn't
4:22:07anticipate in advance I really had
4:22:09dedicated time slot for 3D model in with
4:22:13Play-Doh and then it just like naturally
4:22:15flew into some of the other activities
4:22:17and so we just let it
4:22:19happen so this was already leading into
4:22:22the running design gems a little bit I'm
4:22:24going to give you a few tips here so I
4:22:27think it's important to start by giving
4:22:29a problem statement uh you really don't
4:22:32want your participants to spend most of
4:22:33their time trying to clarify what it is
4:22:37that you want them to do and what the
4:22:39problem really is and so the problem
4:22:41framing shouldn't be to them it should
4:22:43actually be given to them I think this
4:22:45will save a lot of time and potentially
4:22:47also frustration among participants and
4:22:50believe me I'm speaking from experience
4:22:51here because I've made that mistake
4:22:53before for example when I was running
4:22:55the kai
4:22:562019 um AR VR prototyping design Jam uh
4:23:00we were designing a remote conference uh
4:23:03experience app uh thinking about AR and
4:23:07VR and then most of the discussions were
4:23:09like about the tasks and whether remote
4:23:13attendance is actually something that is
4:23:15Meaningful and useful because the
4:23:18community and the participants were from
4:23:20that Community they were really torn and
4:23:22split whether you should go in person or
4:23:25should attend remotely and I should have
4:23:28anticipated that beforehand but I really
4:23:30didn't and so I feel like I should have
4:23:33said okay imagine Kai is going to be
4:23:36remote and this is what we're designing
4:23:38for bam and there you go and associated
4:23:41with that is this next really important
4:23:43tip which is having a clear design
4:23:46prompt make it short and precise
4:23:49definitely leave room for novelty and
4:23:51creativity but also challenge your
4:23:55participants and then it's very
4:23:57important throughout the design Jam to
4:23:58really use a consistent structure so for
4:24:01each activity start with a goal provide
4:24:03participants uh with a description of
4:24:05each task that they should be doing
4:24:08unless unless the approach that they
4:24:11choose is part of what you're trying to
4:24:13figure out so maybe then can you can
4:24:15leave it a little bit um Loosely
4:24:17specified but you should really still
4:24:19specify clearly specify the deliverables
4:24:23so in a lot of my design jams the
4:24:24primary deliverable at the end is some
4:24:26kind of video that um talks uh about the
4:24:30evolution and maybe the final prototypes
4:24:32that came out and maybe also some of the
4:24:35design thinking that went behind it but
4:24:37what I really hope to see in these
4:24:39videos is some kind of creativity and or
4:24:42thinking and a prototype that helps me
4:24:45imagine what the participants were
4:24:47envisioning so now that we have
4:24:49specified the problem gave them a clear
4:24:52prompt and we have paid attention to a
4:24:54consistent structure throughout at the
4:24:56end of the design gy it's also really
4:24:58important to think about what kind of
4:25:00feedback uh do you want to collect from
4:25:02participants there should be normally
4:25:04some kind of debriefing I often do some
4:25:06kind of discussion round in combination
4:25:09maybe with a more individual feedback
4:25:11questionnaire
4:25:12um so that participants can uh answer
4:25:15questions and also share their
4:25:16experience and maybe make suggestions
4:25:18for improvement because I often run
4:25:20design gems multiple times with
4:25:22different participants and so I can
4:25:24improve iteratively basically the design
4:25:27gem as its own little design and design
4:25:31product um throughout the design gam I
4:25:35think from a facilitation perspective
4:25:37it's very important that you don't just
4:25:38do it by yourself you have a little bit
4:25:40of a team uh you have
4:25:42I often work with students and uh
4:25:45research assistants and they help me
4:25:47with timekeeping and also answering
4:25:49questions and taking observational notes
4:25:51so it's really important to plan this uh
4:25:54before you actually do it so maybe do a
4:25:56dry run that can be a lot of fun and you
4:25:58can figure out and iron out some of the
4:26:00issues and then I said it's important to
4:26:02collect feedback so I often design uh
4:26:06one page that's important feedback
4:26:08questionnaire and I'll start with the
4:26:11questions that are most relevant to um
4:26:14our project in this case around the
4:26:16research that we were doing and the
4:26:18demographic stuff I leave for later uh
4:26:20you usually want to collect both types
4:26:22of information but um and then the
4:26:25middle portion here uh is really an
4:26:27opportunity to provide more open-ended
4:26:29uh feedback but even with the open-ended
4:26:32uh feedback I usually um tell my
4:26:35participants how many items of feedback
4:26:38we are roughly looking for so two Pros
4:26:40two cons of this experience for
4:26:56example so now we're switching
4:26:58perspectives and now we're thinking
4:26:59about participating in design gems
4:27:02here's a few tips that I usually give my
4:27:04students in my classes um because
4:27:07obviously these there's a lot of design
4:27:09gems out there a lot of companies run
4:27:11design GS
4:27:13and um especially in the arvr space and
4:27:16it's sometimes as I said used as a
4:27:18recruitment tool as well so you really
4:27:19want to do well here and so here I've
4:27:22collected a few tips through my own
4:27:23observations over the years running lots
4:27:26of design gems as I said in research in
4:27:28my lab and then also in in teaching in
4:27:31the classroom I'm going to go quickly
4:27:33over each of these principles and almost
4:27:35read them out to you but I think they
4:27:38should basically stand for themselves
4:27:40making a plan is crucial so really
4:27:42distribute the rules and let them evolve
4:27:44these rules okay sketch to brainstorm
4:27:47don't just brainstorm sketch it's very
4:27:49important I've seen a lot of design
4:27:51games where basically you just talk talk
4:27:54talk and nothing happens nothing comes
4:27:56to paper but paper is such an important
4:27:58medium to communicate ideas so I really
4:28:02always emphasize this idea of uh
4:28:05sketching to idate not just
4:28:08brainstorming and thinking bubbles which
4:28:10I cannot really see
4:28:12make use of space so this is be physical
4:28:15be visual be creative divide the
4:28:18workload so not everything has to be
4:28:21done in parall try to identify
4:28:23opportunities to divide the labor and
4:28:25then maybe take turns so that nobody
4:28:27gets bored and really use the
4:28:29facilitator so so one person that really
4:28:31pushes the team uh because most design
4:28:33jams are time limited and structured
4:28:36into several sections so it helps uh to
4:28:41to move forward and actually um be
4:28:43efficient as a
4:28:46team and then um the key deliverables is
4:28:50something you should really decide on um
4:28:52so I'll list a few examples here but not
4:28:55each of them are important and if you
4:28:57have taken standard user experience
4:28:59design or interaction design classes you
4:29:00really start with scenarios personas
4:29:03competitors and yes I've talked about
4:29:05that as well but I think really just get
4:29:09down to sketching quickly sort out your
4:29:11storyboards maybe prototype something
4:29:13physically if you have the time do a
4:29:15digital prototype if that is something
4:29:17that would be of benefit and seems
4:29:19doable to you but really be creative and
4:29:22then as I said in most of my design gems
4:29:26usually the product is a video and that
4:29:29video if you have a little bit of time
4:29:31for editing and cutting not so much the
4:29:33music but like really focusing on on the
4:29:35details so cutting off the the things
4:29:37that would be distracting don't focus on
4:29:40nice video over overlays and title
4:29:43slides and all this kind of stuff no
4:29:46keep it kind of so that it's like ad hoc
4:29:49you feel like you're there you you're
4:29:51part of that experience you see that
4:29:53prototype for yourself even though it's
4:29:54just a video H so keep it Dynamic and
4:29:58don't make it too
4:30:01professional and then throughout the
4:30:03design Jam one of the really really
4:30:05important things I cannot emphasize
4:30:07enough is that you should really
4:30:09experiment try out alternative design
4:30:12ideas you can do this through parallel
4:30:13prototyping and then maybe choose some
4:30:15of the best ideas but don't judge
4:30:18immediately let everybody see a little
4:30:20bit and come together so these are some
4:30:22of the best ways I've seen uh teams
4:30:24organize themselves and then when it
4:30:27comes to uh documenting design you
4:30:29should really think of documenting the
4:30:31evolution so don't just take the
4:30:32pictures at the end when you're like
4:30:34really proud of what you've accomplished
4:30:36and maybe you've won something and there
4:30:38reasons to celebrate or even if you
4:30:40haven't won there's always reasons to
4:30:41celebrate
4:30:42um so don't just take pictures at the
4:30:46end and then this is like from the heart
4:30:49um but because I know that companies
4:30:51increasingly see value in design gems so
4:30:54for example uh at yish and in my lab
4:30:57specifically lots of companies approach
4:30:59us and want to work with me and my
4:31:00students and uh the only way to really
4:31:04do well in these design gems is to
4:31:06practice practice practice so U
4:31:08participate in as many design gems as
4:31:10possible and then and in these design
4:31:12gems and don't treat it so much as a
4:31:15competitive kind of activity treat it as
4:31:18something collaborative and fun and try
4:31:20to feel for the boundaries you know the
4:31:24end of the envelope if you will by
4:31:27soliciting feedback from other teams
4:31:30basically your competitors or maybe even
4:31:32some of the organizers I think these
4:31:34this can lead to really cool discussions
4:31:36I usually observe it with students that
4:31:38they seem to be just focused on their
4:31:39teams and um but it's really actually
4:31:42cool to take a break and look around and
4:31:44see what others are working on because a
4:31:46lot of the value from these design gems
4:31:48and the best results in these design
4:31:50gems don't just come from sticking
4:31:53within your team and within and sticking
4:31:56to that specific idea of yours but kind
4:31:59of like seeing what the others are doing
4:32:01and incorporating some of these elements
4:32:04even though it feels like cheating
4:32:06that's boring and that's creative and so
4:32:08I wouldn't totally discourage it so so
4:32:12in summary as a participant here are a
4:32:14few things you could or really should do
4:32:17uh so before you get started make a
4:32:19little bit of a plan think about the
4:32:20time that you have Define your approach
4:32:22maybe the roles in the team and then
4:32:24also the Milestones your interpretation
4:32:26of the goals and
4:32:28deliverables really think about the
4:32:29scenarios but maybe don't spend too much
4:32:32time writing them just like try to agree
4:32:35verbally or implicitly in some way with
4:32:38your team so be clear on the users the
4:32:40agents and what they're doing and then
4:32:43another thing that I often observe is
4:32:45then a lot of design prompts and design
4:32:47Jam activities uh seem a little vague
4:32:51and they're subject to interpretation so
4:32:53rather than getting frustrated about
4:32:54that you should see it as an opportunity
4:32:57to make decisions um not assumptions but
4:33:00decisions and then clearly articulate
4:33:03why you have designed something this way
4:33:04because you were designing for this type
4:33:07of persona or for this type of uh
4:33:10context namely physical environment Etc
4:33:14so make decisions in your scenarios and
4:33:18throughout all the
4:33:20activities yes you could use personas uh
4:33:24and if you do then obviously they would
4:33:26just be used as a tool to communicate
4:33:27who you are designing for and
4:33:29importantly who you're not designing for
4:33:31storyboards I think are really key um
4:33:34for that and I'll show you in some of
4:33:36the physical prototyping activities um
4:33:39we actually have specific arvr templates
4:33:42and I think it's often a good idea to
4:33:43use some templates but then don't just
4:33:45fill in those templates break out of
4:33:47those templates if it's justified by
4:33:49your design ideas and uh push that
4:33:52creativity and Novelty Edge a little bit
4:33:55flesh out the most promising ideas and
4:33:58then really quickly transition into
4:34:00something that is more tangible like a
4:34:01physical prototype or maybe even a
4:34:03digital prototype the physical
4:34:04prototypes will allow you to quickly
4:34:06demo the envisioned interface and
4:34:08Associated interactions and the digital
4:34:11prototype will demonstrate feasibility
4:34:14if you're good with the tools but I
4:34:16wouldn't just always jump into the
4:34:18digital prototyping because it's often
4:34:20quite unrealistic to get a really robust
4:34:22and useful prototype I would often favor
4:34:25the physical over the digital especially
4:34:28in in those design
4:34:31jams so these were just a few concrete
4:34:35ideas and examples of what you could or
4:34:37should be doing I think you should
4:34:40decide for yourself and with your team
4:34:42which of these you want to prioritize
4:34:44you usually can't do all of them but
4:34:46pick those that really help you learn
4:34:48the most about the problem and then also
4:34:51the solution so the last thing I would
4:34:53like to say I said I've used design gems
4:34:55a lot in my research and every design
4:34:57gem has been different H and there is
4:35:00some kind of U really creative part
4:35:03about that methodology that I like it's
4:35:05very different from a lab based user
4:35:07study or some kind of controlled
4:35:09experiment and I really like that
4:35:12creative component both in terms of like
4:35:14what the participants produce but also
4:35:16how you run it um I think that's uh a
4:35:20very valuable part and I think lots of
4:35:22interesting kinds of data usually come
4:35:24out of these design gems so both the
4:35:26kinds of prototypes that uh participants
4:35:29have created we have analyzed a lot of
4:35:31different ways and learned that way what
4:35:32kinds of gestures interactions speech
4:35:34commands how do they think about
4:35:36multimodel interaction how do they think
4:35:37about VR how do they think about AR
4:35:39should our prototype really be an AR or
4:35:41a VR prototype so all these kinds of
4:35:44things you you can really get out of
4:35:45these design gems it's very very useful
4:35:48so think of them as a tool embed them
4:35:50early and throughout the design of your
4:35:53project uh I really can't get enough of
4:35:55design gems and I hope that's a a joy
4:35:57and a passion that you will share with
4:35:59me
4:36:13in this video we're going to talk about
4:36:14XR design critiques uh so critiquing is
4:36:18a very very important skill um design
4:36:21critiques or crits are practiced in in
4:36:24many areas including in architecture
4:36:27it's quite extensive here for example at
4:36:28the University of Michigan uh can be
4:36:31quite heated discussion you might say um
4:36:34it is usually a form of constructive
4:36:36criticism so basically a thorough review
4:36:40and you take into a account design
4:36:43guidelines and um your own experience of
4:36:45of creating um
4:36:48experiences and in that sense so uh
4:36:51we're going to talk about the critiquing
4:36:54part uh I think the best designers are
4:36:57those that uh can that have a critical
4:36:59eye and can see issues in design and
4:37:02seeing issues is actually really that's
4:37:04easy that's the easy part uh formulating
4:37:07a constructive critique that includes
4:37:10elements on of improvement so not just
4:37:14like no no no but also here is a
4:37:18different way to think about it or why
4:37:20don't you try this um so I'm going to
4:37:23introduce you to a little bit of uh that
4:37:26mindset that I adopt even though I I can
4:37:29actually be quite critical but also with
4:37:31my own work um but I think uh ultimately
4:37:36the idea is to generate feedback and so
4:37:38the other part of critique that we don't
4:37:40really talk so much about is uh and that
4:37:43actually makes a very good designer as
4:37:44well is the ability to address feedback
4:37:47and to to to to react and respond to
4:37:51feedback if you have access to a
4:37:54smartphone that is compatible with uh
4:37:57Google cardboard or any derivatives I'll
4:38:00show you a version of cardboard that I'm
4:38:02using which is actually made of plastic
4:38:04um or you can use the AR mode um so uh
4:38:09Expeditions actually runs uh on both IOS
4:38:12and Android uh phones so you do need a
4:38:15little bit of a later generation
4:38:17smartphone model here but uh we should
4:38:20be able to to work with Google
4:38:23Expeditions so I'm going to do a
4:38:25critique now of Google Expeditions and
4:38:28then I'm going to choose a few specific
4:38:30Expeditions I'm going to focus on VR I'm
4:38:32going to talk a little bit about uh 360
4:38:34so this ability to switch from VR 2360
4:38:37which I think is actually a great
4:38:38feature uh and then I'm going to talk
4:38:40about uh uh an AR exhibition and these
4:38:44are just examples but think of of
4:38:46expeditions as an app that can actually
4:38:49load actually download uh different
4:38:52kinds of expeditions these Expeditions
4:38:54can be created by other uh vendors so
4:38:58not just Google and um I think this
4:39:01should provide a playground for us to
4:39:03actually practice critiques uh
4:39:05Expeditions as a tool has a relatively
4:39:09limited number of features but it's
4:39:11interesting how these different
4:39:13Expeditions implemented and there are
4:39:16still better ways to do it and then some
4:39:19not so good ways and I'll point out some
4:39:22of the more common design issues that I
4:39:24saw across the board as examples um but
4:39:28let's uh let's just jump in and get
4:39:31started okay in this uh video I'm going
4:39:33to do a quick review of Google
4:39:35Expeditions and um so I'm going to
4:39:38launch it here uh on my uh phone and
4:39:44we're going to use cardboard and um I
4:39:47don't have cardboard cardboard with me
4:39:49right now so we're going to use this
4:39:51version of cardboard which is a im
4:39:54cardboard plastic version of cardboard
4:39:57and it's really quite cool so it's quite
4:40:01uh
4:40:01mobile and that's it it just goes into
4:40:05my head like this so I'm going to start
4:40:08Expeditions going to put it in there as
4:40:11the phone instructs me to to do it and
4:40:16and there we are and so now it actually
4:40:20looks really nice I already have this
4:40:22dinosaur here and a really cool kind of
4:40:25like background uh
4:40:28image and um so what we're going to do
4:40:31is we're going to look
4:40:34at the downloaded tourist for a sec I
4:40:37have all these tourist that I have
4:40:38downloaded here and um so quite a few
4:40:45and the one that I wanted to do is the
4:40:47New York City Tour and so this is this
4:40:52is uh New York City and you can see
4:40:56Manhattan here and there are a couple of
4:40:59landmarks so now we're looking at the
4:41:01One World Trade Center um which is the
4:41:05World Trade Center Memorial and then
4:41:08this is the Woodworth building
4:41:11and as you move away I like how it
4:41:14actually gives you these visual cues
4:41:16that you have to go back there to find
4:41:18the
4:41:19Target and uh we can continue the Tour
4:41:23by looking at Bridges and now we're
4:41:25going to find the Manhattan Bridge and
4:41:28the Brooklyn Bridge that we're looking
4:41:30at and then so again the visual cues are
4:41:35cool and let us just explore something
4:41:37maybe out of sequence or order but there
4:41:40don't doesn't seem to be too
4:41:42many uh landmarks in this scene so let's
4:41:45just go
4:41:46to a different scene so the way you
4:41:50switch scenes is uh this way now we're
4:41:53looking at middle Upper Manhattan Island
4:41:56we can scroll here and read more if you
4:42:00want
4:42:01and Central Park South let's go there
4:42:07that doesn't work um okay that one
4:42:09activated
4:42:11and for some reason the other one didn't
4:42:12work so let's see whether we can reach
4:42:14it by going next okay it does work now
4:42:17that's a little uncomfortable like so
4:42:19far
4:42:21down and I mean you can lean in but
4:42:24obviously it's not track in six Dees of
4:42:26Freedom only 3° of Freedom so um you
4:42:29can't actually really this is weird this
4:42:32kind of like motion and then the scene
4:42:35doesn't actually react to it obviously
4:42:37is a little
4:42:39uncomfortable and um
4:42:41there are other issues here when
4:42:44you sometimes when this other menu comes
4:42:47up let's see where does it it's unclear
4:42:51how to bring up this menu is really
4:42:53unclear to me um and then it is always
4:42:56in the way somehow it's
4:42:58interfering
4:43:02so
4:43:04maybe again I don't know how to bring up
4:43:07this
4:43:08menu and I just know it happens
4:43:10sometimes it comes and it like it seems
4:43:13to happen when you move in a certain way
4:43:15anyway and then you can explore
4:43:18different scenes uh here and we get an
4:43:20overview of all the scenes and we can I
4:43:23didn't want to click this so as an
4:43:25accident I actually wanted to click this
4:43:28thing um so I have to be careful cuz I
4:43:31know there is
4:43:32the sunset on the Hudson River one that
4:43:36I wanted to look at and show you and
4:43:40this is
4:43:42quite a view isn't
4:43:44it it is really nice it's good to have a
4:43:47swivel chair in order to explore this in
4:43:51a safe and fun way so I wanted to
4:43:56interrupt this uh critique for a bit and
4:43:58there were that that was a thinkaloud so
4:44:01you could listen to me uh as I was
4:44:03exploring the app and I was pointing out
4:44:05some of the confusions I had and when
4:44:07something was a misclick and or I
4:44:10intended something else so that's
4:44:12actually quite useful then this
4:44:14frustration about this menu that doesn't
4:44:16appear and um so I'm going to summarize
4:44:21some of the key issues that I've
4:44:23identified so far and and then we're
4:44:25going to look at a uh a a method to
4:44:29actually formulate a critique that
4:44:31starts with the positive and it's a
4:44:32little bit more structured so uh one
4:44:35thing I've talked about was a little bit
4:44:37of uh the menus and and that was okay
4:44:40and I mean there were some issues there
4:44:42with uh
4:44:44accidentally uh not hitting this thing
4:44:46and then something else was triggered
4:44:48but overall was fine um I talked uh
4:44:51quite a bit about the menu and scrolling
4:44:54and I think one of these items didn't
4:44:55work but um okay that's also okay and by
4:44:59the way this is obviously captured in a
4:45:02steroscopic view and you could see a
4:45:04little bit of um a shift here between
4:45:08the eyes and that is necessary uh in
4:45:11order to give you this uh depth
4:45:13illusion uh and then we have the barrel
4:45:15Distortion that also uh adds to this
4:45:18effect of
4:45:20depth so uh I was uh really pointed out
4:45:24quite a few times how I like all the
4:45:27visual cues like these arrows um they
4:45:30Point me towards uh the direction in
4:45:32which the the land mark is that I'm
4:45:35looking for so I really like that what
4:45:37was completely absent was any kind of
4:45:40audio use um I did have it enabled but I
4:45:44didn't get the narrator to work there's
4:45:45a narration function in there um so I
4:45:48skipped that part maybe it'll work for
4:45:50you and then um I did like the visual
4:45:54targets um and then how the content was
4:45:57placed to it so and I pointed out a few
4:46:00more issues but so let's look at how we
4:46:03can uh critique better to organize our
4:46:08thoughts a little bit and so I'm going
4:46:09to introduce you to a template that I
4:46:11often use and it's it's actually quite
4:46:13popular it's used in a lot of in a lot
4:46:14of design thinking and classes uh also
4:46:19heavily used in Stanford where I think
4:46:21it originated and it's often used in
4:46:24user experience and interaction design
4:46:26classes so that template is I like I
4:46:28wish what if and it works as follows you
4:46:32start with what you like about a design
4:46:35and um you really limit it to the top
4:46:38two things and really point out
4:46:40something something that doesn't feel
4:46:42like you just say something nice to say
4:46:44something nice no really pick something
4:46:46that you think that is cool something
4:46:48that inspired you something that you
4:46:50would you would adopt and incorporate
4:46:52into your own practice I think that
4:46:54makes it a very good way to start a
4:46:56critique and then you have to transition
4:47:00over into the actual critique so the
4:47:04criticism portion of the critique uh and
4:47:06this is the I wish portion so what could
4:47:09be improved again focus on two don't go
4:47:13forever don't make it outweigh
4:47:15everything you have talked about when
4:47:17you liked things I don't use the I like
4:47:21thing to get quickly into the negative
4:47:24so whenever I go to conferences it's
4:47:26like thank you thank you thank you for
4:47:28your presentation but have you thought
4:47:30about bam and so that's like um not a
4:47:35good way um this is a standard way of
4:47:39starting a critique but it's actually
4:47:41quite annoying and it's not really
4:47:42useful so pick something that you like
4:47:45and then transition over into something
4:47:47that you think needs to be improved and
4:47:50it's not only because you think that uh
4:47:52there should be guidelines or any kind
4:47:54of best practice that you can refer to
4:47:57maybe you've read something maybe
4:47:58there's a scholar who wrote about this
4:48:01and that gives your critique more
4:48:04credibility and I think that's important
4:48:05as well and uh you should really then
4:48:08come up with ways of improving it not is
4:48:10saying what needs to be improved but
4:48:13also suggesting how it can be approved
4:48:15so that is the Whata if component and
4:48:17here I usually still ask for two so when
4:48:20I do this in class and have uh students
4:48:23present XR interfaces or whatever
4:48:25interfaces to each other I've done this
4:48:27for many years now um it is often in the
4:48:30form of written feedback so I have like
4:48:31little paper sheets with just this
4:48:33asking for two likes two so for two Pros
4:48:36two cons and two suggestions for
4:48:38improvement and the last part is the one
4:48:41that I think is the most important but
4:48:43doesn't always receive enough attention
4:48:44the middle portion always seems to go
4:48:47really well and uh the first one I don't
4:48:50think is as strong as I wished it it
4:48:53would be so really as strong I like a
4:48:57good ey wish and a really strong what if
4:49:01that is how you should balance those
4:49:02things it's really important uh have you
4:49:05tried this um why don't you try this I
4:49:08think this could work and offer some
4:49:11insights and suggestions why you think
4:49:13it might
4:49:15work okay so let's apply this method to
4:49:19the Google Expeditions example that I
4:49:22just showed and let's try to be quick
4:49:24and like really uh rapid critique I like
4:49:27I wish what if so here we go I like that
4:49:31the visual cues help me find landmarks I
4:49:35is really good there was a lack of AQ
4:49:37but I didn't say that I wouldn't say
4:49:39that at this time I just say I like that
4:49:42the visual cues help me find landmarks
4:49:44and I also like that the main menu
4:49:45allows me to switch scenes I think that
4:49:48is a very good way now I wish that the
4:49:52affordances and visual feedback and
4:49:53menus were a little bit clearer so I
4:49:56wished for affordances and visual
4:49:58feedback in menus I mean there is
4:50:00something but it's not enough and then I
4:50:03also wish there were another way to
4:50:04trigger that main menu I really just
4:50:06couldn't figure it out as you could uh
4:50:09watch me do it you notice some of my
4:50:12frustration as
4:50:13well and so what if so what if we're
4:50:17going to do the following we're going to
4:50:18have hot areas around the menu items uh
4:50:21were marked so I even suggested that
4:50:23here in a little design Improvement and
4:50:25I think that's a that's a good what if
4:50:27you could sketch out an improvement uh
4:50:29or maybe with a little bit of image
4:50:31manipulation which I did here just using
4:50:33some basic overlays uh you could suggest
4:50:36how to change the design and what if the
4:50:39main menu appeared on click
4:50:41and there were no collisions that would
4:50:43be a great Improvement and and the no
4:50:44collisions part we could maybe figure
4:50:47out where other menus are and make sure
4:50:50that uh they are not overlapping each
4:50:52other because that does happen quite a
4:50:54bit and it can be a little annoying
4:50:57because you don't know which of the
4:50:58menus will actually trigger it's not
4:51:00always obvious and keep in mind uh that
4:51:04we don't actually have a z-axis we can
4:51:07work with so this is 3 degrees of
4:51:09freedom so you really just have head
4:51:10rotation that's all you have and so you
4:51:13leaning in or out doesn't make a
4:51:15difference and if if this were supported
4:51:17on on cardboard then maybe we could work
4:51:22around the issues with uh these overlays
4:51:25and the menus but as it stands it's an
4:51:27issue and here are some ways of
4:51:29addressing it now I'm going to jump back
4:51:32and continue with a little bit of that
4:51:34example and I'm going to point out some
4:51:37other features um that I think are cool
4:51:41and um I'm not going to do this same I
4:51:44like I wish what if method again but you
4:51:46could imagine doing that yourself and
4:51:49after the 360 view that I'll introduce
4:51:51you to I'm also going to show you a few
4:51:53AR examples and then uh you could
4:51:57imagine um the I like I wish What If
4:51:59part um but I think it's just maybe
4:52:02valuable for you to see how I explore
4:52:05the interface and how I critique things
4:52:08as I go
4:52:23and the other thing that this app
4:52:25actually supports which I think is also
4:52:27pretty cool is get out of the cardboard
4:52:31uh VR mode Let's just like put this away
4:52:34for a sec and then we have a button that
4:52:37essentially gets out of the steroscopic
4:52:39view and just brings us into a 360 View
4:52:43and now we can just like look around and
4:52:48still enjoy the
4:52:51sunset and um
4:52:53interestingly so it doesn't work as it
4:52:56doesn't feel as nice as the VR
4:52:59experience but um it kind of still works
4:53:02and you the nice thing about it is that
4:53:04it remember state so you're still in the
4:53:05same scene you still have the same
4:53:07landmarks activated and the interactions
4:53:11are still really the same the text is a
4:53:13little harder to reach cuz everything is
4:53:14a little smaller now this kind of
4:53:16navigation still works still feels
4:53:19awkward to look down onto an object like
4:53:23that but and this transition is like how
4:53:28am I supposed to do this
4:53:29transition okay and now I swivel too
4:53:32much and somebody might feel sick uh
4:53:35that way so it's it's interesting how
4:53:36you navigate uh these scenes um how
4:53:40about say overall it is a a cool
4:53:42experience there's a lot of expeditions
4:53:44you can explore I chose the world cities
4:53:46one here New York as one example um in
4:53:50the exercise um we we're also going to
4:53:53look at the AR version of it which I
4:53:54think is also pretty cool can't do the
4:53:56same Expeditions in AR and VR so um I
4:53:59haven't found one that is both available
4:54:01in in AR and VR but um you the the
4:54:04framework around it the the tool
4:54:06Expeditions as a tool or the app around
4:54:08it um is interesting how it works to
4:54:10compare it in AR and in VR um if you
4:54:13only have AR or VR available um so VR um
4:54:18in this case through a form of cardboard
4:54:20if you only have one of them available
4:54:22um probably AR is going to be possible
4:54:24for you then I would recommend that you
4:54:28explore some of the Expeditions and do a
4:54:31design critique of of how it is designed
4:54:34and um well try to apply some of the
4:54:38guidelines and then also look at it from
4:54:40an Ethics perspective um those are some
4:54:43of the steps we want to do in some of
4:54:44the exercises so um this should be fun
4:54:48and I'm I'm I hope you learn a lot and I
4:54:50think there are obviously other
4:54:52applications out there that you could
4:54:53explore as well but just within
4:54:55Expeditions as an app there's so many
4:54:57different things we can explore and
4:54:59learn from um so it should be quite cool
4:55:02so that was the uh Google Expeditions
4:55:05for VR and we looked at uh City's
4:55:08example New York here uh as as an
4:55:11example and that was just one of the
4:55:13Expeditions there are others um and um
4:55:16I'm now going to switch into AR and show
4:55:19you a little bit how that works and
4:55:21maybe that's supported on your phone and
4:55:24um please feel free to critique AR or VR
4:55:28experiences um I do think um uh my
4:55:32feedback is a little bit different it's
4:55:34very much focused on the environment and
4:55:38how it uh works with placing objects in
4:55:42the physical environment anchoring and I
4:55:45noticed quite a few issues around um
4:55:48rotation and scaling uh which wasn't
4:55:51really supported only some kind of
4:55:52translation along the y- axis that was a
4:55:54little confusing to me but um let's look
4:55:57at this together it's a thinkaloud again
4:55:59from my perspective uh this critique
4:56:03could be um reformulated into the I like
4:56:08I wish what if method um I would still
4:56:10go and do a thinkaloud first of any
4:56:12interface that somebody wants you to try
4:56:15out and and then you should summarize
4:56:19your uh feedback uh in this I like uh I
4:56:24wish and what if method we're going to
4:56:26view an AR and so here we're going to
4:56:29put it on the floor this is a skeletal
4:56:32system and let's look at this guy pretty
4:56:37big um
4:56:41yeah it doesn't look too
4:56:42happy and just looking at the
4:56:47sky and that is interesting how it's
4:56:50animated and
4:56:55basically goes
4:56:57apart um
4:57:00yeah and then the
4:57:04spine it's a bit scary isn't it
4:57:10and in this case we're going to raise it
4:57:11a little bit but it should be
4:57:14scaled okay so the AR interactions
4:57:17overall are pretty um limited
4:57:21Expeditions again is designed in an
4:57:23interesting way it's for a classroom use
4:57:26so you might actually run this in a in a
4:57:29classroom so with a set of students and
4:57:33Learners and uh maybe that is why there
4:57:36is no audio so that we don't actually
4:57:38have the audio really interf hearing uh
4:57:41that could be a design decision it was
4:57:43just not really explicitly
4:57:45communicated um and so that's fine so
4:57:47again keep in mind not all of the
4:57:49criticism might be valid here this is
4:57:51just like based on this one use case of
4:57:54me trying it out in the studio or you
4:57:56trying it out at home or wherever you
4:57:57are and consider the fact that it was
4:58:01designed for classroom use and it
4:58:03actually has a mode where an instructor
4:58:05could use it with a group of students
4:58:08and uh that's something to experience as
4:58:10well and and so to close I would like to
4:58:13highlight a few key points again we
4:58:15should really try to base our critiques
4:58:16on established design guidelines that
4:58:18makes them more objective and and that's
4:58:21just a very good way to critique so uh
4:58:25obviously design guidelines could be
4:58:27based on design principles so like nor
4:58:30principles of design feedback
4:58:32affordances mappings and constraints
4:58:34whether something follows uh natural
4:58:37mapping and whether the affordances
4:58:39physical forces so whether the objects
4:58:41suggest to you how you can interact with
4:58:43them and uh constraints as well you
4:58:46should take into account usability so
4:58:49learnability memorability uh efficiency
4:58:52errors and satisfaction these are common
4:58:55usability uh
4:58:57considerations you should uh also
4:59:00critique it from the perspective of
4:59:01accessibility and Equity so if an
4:59:04interface uses uh some of the most
4:59:07advanced features out there that are
4:59:09probably only ailable on the latest
4:59:11generation headsets which at this stage
4:59:12are still very
4:59:14expensive um think about whether that is
4:59:18really necessary and uh think about the
4:59:21limited reach because a lot of people
4:59:23may not be able to afford that
4:59:25experience and that's another thing that
4:59:26I like about Google Expeditions that
4:59:28it's designed for low-end devices and uh
4:59:31therefore scales also in the classroom
4:59:34potentially think about the audience um
4:59:37so the users the background
4:59:40again bring in this idea of personas and
4:59:43maybe review from that perspective and
4:59:46then also to a limited extent I would
4:59:48say bring in your personal experience
4:59:51and finally you should think about some
4:59:52of the broader concerns that you may
4:59:55have about the experience and the impact
4:59:58that it may have on its users here I'm
5:00:01just going to list a few of the common
5:00:03ones so privacy is a big concern mental
5:00:06health physical health uh anything that
5:00:09you think has to do with uh ethical
5:00:12issues so maybe the uh degree and the
5:00:16level of realism and maybe the level of
5:00:19immersion and whether this could be
5:00:21potentially dangerous think about the
5:00:24social impact can this experience be
5:00:26used in a classroom or not uh think
5:00:30about those kinds of things think about
5:00:32cultural issues that you see in the
5:00:33design so if I just uses Sports
5:00:36metaphors from baseball I would
5:00:39understand nothing and so it's one of
5:00:41the issues I had when I came to the US
5:00:43but just a side note here and uh any
5:00:45kind of environmental and sustainability
5:00:48concerns so really be broad um also in
5:00:51your review and address those uh and
5:00:55bring up those concerns just raise them
5:00:57can lead to very interesting discussions
5:01:16again welcome to the honors track this
5:01:18is the Exile design critique this is the
5:01:20exercise that is going to get us started
5:01:23as part of the honors track so uh for
5:01:25every exercise I'll provide a little
5:01:27introduction video and The Specific
5:01:30Instructions regarding the exercise will
5:01:32be shared through uh the moo uh the
5:01:35readings that we have but this video
5:01:38provides an overview and talks about
5:01:40some of the main steps and why this
5:01:43exercise actually exists so design
5:01:46critiques are very important I'll talk
5:01:48about that in its dedicated lecture uh
5:01:50it's a ritual in fact in my course when
5:01:53I teach this residentially every student
5:01:55has to critique an existing arvr app and
5:01:58it's really cool because when you bring
5:01:59all bring together all these students
5:02:01and they critique these existing apps
5:02:04everybody learns about existing apps and
5:02:06also we practice our skills of doing a
5:02:09thoughtful productive uh but also with
5:02:13constructive criticism uh a r critique
5:02:17uh something that we can Lear from and
5:02:19so that's what we're trying to exercise
5:02:21here so I'm going to walk you through
5:02:22the main steps of this
5:02:24exercise so the task should you choose
5:02:27to accept it is to critique an existing
5:02:30Expedition like XR app in 360 VR or ar
5:02:35so I say um this so that we have maximum
5:02:38flexibility you you don't necessarily
5:02:40have to have VR or ar you can just do it
5:02:44on a smartphone with 360 so and I say
5:02:47Expedition like XR app uh I'm thinking
5:02:49Google Expedition and I'll tell you a
5:02:51little bit more about Google expedition
5:02:52in a second but also uh it could be your
5:02:56favorite museum if it has an app or
5:02:58something for the city or something that
5:03:01feels like you can go on a tour um I
5:03:04want it to be not too complex an app but
5:03:06the activity is ready to critique it and
5:03:09if it's too complex might probably be
5:03:11one of the main criticisms so what I'm
5:03:14thinking in the task should be to record
5:03:16yourself describing the XR experience
5:03:19using thinkaloud it's very important so
5:03:22the thinkaloud part is really important
5:03:23yeah you should you should play around
5:03:25with it and then you play around with it
5:03:27one more time and then you record uh
5:03:29what you're thinking even your most
5:03:31often the first time impressions are
5:03:33really the ones that are really really
5:03:34cool to capture in this think aloud but
5:03:37maybe we're not used to this idea of
5:03:39thinking aloud as we're using something
5:03:42and that's something you have to
5:03:43practice and um so I'll show you an
5:03:47example of of me doing a thinkaloud
5:03:49while playing around with an existing
5:03:51Google Expedition uh I chose the solar
5:03:55uh system
5:03:56one I want you to describe the context
5:03:59of your XI experience so I want you to
5:04:01well basically share uh when you share
5:04:03the results of your critique uh whether
5:04:05you have operated this app in 360 in VR
5:04:08in AR have you been in a private or
5:04:10public setting um so if you were uh you
5:04:13don't have to be too specific but I
5:04:15think it may have impact on how you
5:04:18critique the app maybe it was not
5:04:19designed for the context in which you
5:04:20used it and so that's then uh something
5:04:23that we could also discuss as well I
5:04:26want you to critique the Expedition
5:04:28using the template the I like I wish
5:04:30what if template okay so I introduced it
5:04:33in the lecture and I'm going to briefly
5:04:36mention it again in the next few slides
5:04:38I want you to analyze the exposition in
5:04:40terms of guidelines so really design
5:04:42guidelines first and I provided some
5:04:44examples of guidelines I mean as you
5:04:46know from my lecture I'm never really
5:04:48happy with all the guidelines we have we
5:04:49don't have enough guidelines and those
5:04:51we have are not good enough so um anyway
5:04:55this a starting point I want you to also
5:04:58articulate ethical social or privacy
5:05:00concerns one might have I mean you you
5:05:03might have them you might not have them
5:05:05uh but users could have them and so I
5:05:07want you to think about that so the
5:05:10expected results of this activity and
5:05:12this exercise are that you have now uh
5:05:14provided a balance critique of positive
5:05:17and negative design aspects so really
5:05:19start with the positive keep it
5:05:22remember you have developed initial
5:05:24ideas for how to change and improve the
5:05:26design that will be very helpful for you
5:05:28as you're continuing to work on this and
5:05:30go into the next stage which is
5:05:32storyboarding and uh SP of storyboarding
5:05:34we do brainstorming and you also have a
5:05:37better sense of what to look for in
5:05:38terms of design guidelines
5:05:41so that would make me really happy if
5:05:43you achieved all these if these were
5:05:45really truly really truly the the
5:05:47results so that would be cool um but uh
5:05:50that's what I'm hoping we get out of
5:05:52this and if you practice critiques and
5:05:56you really think broadly about some of
5:05:58the concerns then I think this will be
5:06:00very valuable and uh might be a lot of
5:06:03fun to also discuss with your peer
5:06:04Learners here so please we'll have
5:06:06forums available and we can chat
5:06:10so I said the expedition that I had in
5:06:12mind um the actual app is Google
5:06:14Expeditions so now Google Expeditions
5:06:16comes in VR and in AR and it's actually
5:06:19popular in classroom environments so
5:06:21Google is targeting education scenarios
5:06:24and I thought this is a good app for us
5:06:26but I say Expedition like um it could be
5:06:30really anything but I don't want it to
5:06:31be too complicated ideally it's publicly
5:06:35available so that others can actually
5:06:36also try it out so you should be clear
5:06:39um how you obtained uh the app if it
5:06:42wasn't Google
5:06:44Expeditions so here's an example of me
5:06:47running the world cities Expedition and
5:06:49I've covered that in the lecture and
5:06:52here is a video of me trying it out um
5:06:56actually here in the studio the solar
5:06:59system uh app all right I'm going to do
5:07:03a design crit an exra design crit of the
5:07:07Google Expeditions uh the exploring the
5:07:10solar
5:07:11system and we're going to start it in AR
5:07:14so initially um I think this is this
5:07:18works quite well detects the surface
5:07:20surface okay and then the the problem is
5:07:24that the planet is like really really
5:07:27big that we're getting and any attempt
5:07:31to resize doesn't work I can just raise
5:07:35it but I cannot I have not figured out a
5:07:39way to resize or
5:07:42rotate and that is actually something
5:07:45that I would normally expect from an
5:07:48from an app then you can learn more
5:07:52about the Sun and what we're going to do
5:07:56instead is we're going to look at Earth
5:07:58so Earth is obviously smaller than uh
5:08:00the Sun but it's still definitely
5:08:04definitely definitely quite big and um
5:08:07same issue but I do like the inner
5:08:09planet so that's like kind of cool when
5:08:12they then appear above the keyboard here
5:08:15so there's
5:08:17Earth and we can look at some of the
5:08:22detail maybe even
5:08:26find parts
5:08:29um well maybe you can find where you
5:08:33live and um that could be
5:08:37interesting and tracking uh works it's
5:08:40tracking is really robust obviously so
5:08:42that's kind of nice and then the moon
5:08:44again a Monster
5:08:47boom and it's just
5:08:51massive and we have the outer
5:08:55planets so that's also pretty
5:08:59cool and so it would be nice to have
5:09:02labels on
5:09:04these I really like the look of the
5:09:06planets but like having labels would be
5:09:08very useful
5:09:11and so that you're actually learning
5:09:12something I mean right now it's kind of
5:09:14cool the asteroid belt it kind of looks
5:09:18cool and nice the dwarf planets but then
5:09:21you would really want to learn a little
5:09:22bit more
5:09:24and um the interactions here are
5:09:28definitely pretty
5:09:29limited um it does look cool though um I
5:09:33must admit that so I would say I like
5:09:37the look of the planets uh that's pretty
5:09:41cool
5:09:43and I like
5:09:45how um I especially like the inner
5:09:49planets and I like how they have breaken
5:09:52down the solar system um although it's
5:09:56harder for me to build the relationships
5:09:58between the different kinds of views
5:10:00that I have on the different planets and
5:10:02the asteroid belt and the Moon is
5:10:04normally obviously going around Earth so
5:10:07would have been nice to see that
5:10:10um I wished I were able to rescale
5:10:15resize um so this I just don't
5:10:18understand why I would want to raise it
5:10:22like crazy it does make sense when I put
5:10:24it
5:10:25on when I relocate it but I also can't
5:10:28relocate it so I don't know how to do
5:10:32that can send feedback
5:10:37um yeah so what
5:10:40if you
5:10:42were allowing me to scale it
5:10:46once and then also like I think the
5:10:48other thing is the distance when I then
5:10:51switch I'm sure it's going to just be
5:10:54again oh it does it does keep in mind
5:10:57what I wanted in terms of distance oh it
5:11:01does it does not rotate and
5:11:06scale um so that's really my biggest
5:11:09complaint
5:11:11here and so here is what you just saw um
5:11:15at least a subset of it and uh these are
5:11:19all the screens so this is the download
5:11:21screen obviously then you enter it you
5:11:23place it I didn't have a screenshot for
5:11:24the placing part and then once you have
5:11:26placed it you have like this card
5:11:28interface down there and uh this Edge is
5:11:31really really key it suggests that there
5:11:33is more so you swipe to the left and
5:11:35then you get to see Earth you can swipe
5:11:37up and you get to see some of the
5:11:39information about Earth and then you
5:11:42swipe to the left again you bring up
5:11:44this screen and so forth and uh this is
5:11:47how the Expedition works and I want you
5:11:50to critique uh the Expedition app the
5:11:53Expedition like app that you chose I
5:11:54want you to use the I like I wish what
5:11:56if template so what you like about it
5:11:59important to start with something
5:12:00positive and there should be two
5:12:02positives okay uh what could be improved
5:12:05and here really try to be specific try
5:12:07to be objective uh not subjective and uh
5:12:12what could be improved is improved is
5:12:13one thing but how would you improve it
5:12:15how could it be improved this is really
5:12:17the hard part this is what we're after
5:12:19here this will help us develop some new
5:12:21ideas as we are going to improve upon
5:12:24the
5:12:25app so here are some examples the look
5:12:29of the planets that's pretty cool I
5:12:30think that's something I said in the
5:12:31video how they have broken down the
5:12:33solar system is something that I liked I
5:12:35could resize and rotate the planets I
5:12:37really wished I could and I could better
5:12:39understand the relationships between
5:12:40planets it was really just not enough uh
5:12:43like the scale was really unclear I
5:12:46could adjust the scale and rotation
5:12:47using pinch to zoom yeah that was very
5:12:48confusing to me I remember um the going
5:12:51up and down part um that was weird like
5:12:55why does it only change around the Y and
5:12:58um so how would I improve it um well
5:13:00using pinch to zoom for example which is
5:13:02a very common gesture and how could I
5:13:05adjust the distance to the planets and
5:13:07then the other idea is to adjust the
5:13:10distance to the plane using drag and
5:13:12drop so that was referring to the fact
5:13:14that once the expedition was placed I
5:13:16basically had to restart the app I
5:13:18couldn't figure out how to move it to a
5:13:20different plane maybe it is somewhere in
5:13:23the app and I just didn't notice but
5:13:24then we need to improve upon that aspect
5:13:27that is called bad usability if people
5:13:30cannot discover what you can do with the
5:13:31app then yeah you have a problem so this
5:13:35was a review based on the design
5:13:37guidelines using the I like I but if
5:13:39template the next thing I want you to do
5:13:41is also to perform an Ethics review and
5:13:44I was talking about design ethics review
5:13:46as part of the XR design ethics lecture
5:13:49and I was describing these four main
5:13:52steps or considerations as part of the
5:13:54review so the context and situation the
5:13:57sensory information and data the
5:13:59processing pipeline as much as you
5:14:00understand it I mean clearly you will
5:14:02not have developed the app but um maybe
5:14:04you are developing intuition for what
5:14:06goes on behind the scenes or the app
5:14:07informs you and this informing part in
5:14:10terms of uh terms and what you consent
5:14:12to um this is also really key that we
5:14:15want to look at for data ownership and
5:14:17governance so in my example that would
5:14:20be that I was using the AR version uh in
5:14:23the studio here I was actually the only
5:14:26user in the device vicinity I mean my
5:14:29bunny was probably in the background but
5:14:31uh we don't know uh the sensory
5:14:34information well it sees mostly me it
5:14:36sees the furniture it does see expensive
5:14:38equipment here uh keep in mind that this
5:14:40equipment is on loan from the University
5:14:42of Michigan so this is not my own
5:14:44equipment um the surface area of the
5:14:48table and the floor um so alt together
5:14:52um you can obviously uh you can actually
5:14:55make some sense of this information both
5:14:58for tracking and registration but then
5:15:00also U maybe in a negative way and so
5:15:03that goes more under the ethics part
5:15:05obviously um the processing pipelines I
5:15:07think it's a pretty if application it
5:15:10does standard surface detection doesn't
5:15:11do a lot with spatial mesh in fact it
5:15:14doesn't have a spatial mesh there's no
5:15:15scene understanding uh it may build
5:15:17Anonymous usage statistics I don't know
5:15:19about that but uh it keeps Library it
5:15:21keeps track of the the different kinds
5:15:24of expeditions I was downloading there
5:15:26is a teacher mode which I think is a bit
5:15:29more advanced and multi-user and that
5:15:32has potentially a more complex
5:15:34processing Pipeline and actually will
5:15:35raise additional
5:15:37concerns uh expeditions app manages the
5:15:39downloads as I was just talking about
5:15:41there the SL tab so that's the kind of
5:15:43information that I know um I don't know
5:15:46what they collect and I also don't know
5:15:48because these Expeditions are actually
5:15:49developed independently so there are
5:15:51some Google ones but there are also ones
5:15:53developed by others H it would be
5:15:55interesting to understand the
5:15:56Expeditions framework a little bit
5:15:57better and understand what kind of data
5:16:00is actually being collected and shared
5:16:01with those uh third party
5:16:05vendors that's it right so we're going
5:16:08to learn a lot about ethics um by
5:16:11actually thinking it through but we're
5:16:13also going to do a really a design
5:16:15critique uh more broadly thinking
5:16:18obviously also about the social and
5:16:19ethical concerns as I was just talking
5:16:21about but also thinking a lot about the
5:16:23design and design guidelines and so pay
5:16:26attention to whether you're doing an AR
5:16:27or VR Expedition or something in between
5:16:30like 360 share that with us and fellow
5:16:33Learners so that we actually understand
5:16:34your critique better uh let us know how
5:16:37and where you used it and what you found
5:16:39confusing and uh well we should get a
5:16:42first indication of the kinds of things
5:16:44you think you would be improving as you
5:16:46keep working in this Honest track on the
5:16:49next steps which would then be
5:16:51storyboarding and wireframing we need
5:16:53brainstorming new ideas and flashing
5:16:56them out
5:17:10in this lecture we're going to talk
5:17:12about brainstorming and problem framing
5:17:16now in the context of extended reality
5:17:19so AR and VR this is really something
5:17:22that is dear to me I'm not really sure
5:17:25that we are getting it right yet and um
5:17:28I think the problem is that we are not
5:17:31always really addressing the problem now
5:17:34in this lecture we're going to talk
5:17:35about how to find good problems how to
5:17:39identify a problem so in this lecture
5:17:41I'm going to share seven techniques to
5:17:43help you innovate the first technique is
5:17:45the what I call ideate Implement inspect
5:17:48iterate design cycle everything starts
5:17:51from an idea and you really have to
5:17:53ideate part of ideation is actually
5:17:56implementing that thing creating a
5:17:58prototype and then inspecting it um and
5:18:01not just you as a designer although that
5:18:03I think that is the first important
5:18:05thing uh but you need to test and
5:18:07validate it with users and then you
5:18:09iterate I think that through prototyping
5:18:12we can really learn a lot about the
5:18:14actual problem and not just like about
5:18:17our solution I use ideation for problem
5:18:21Framing and I think that's a really
5:18:23important realization and so build
5:18:26prototypes to help you understand what
5:18:28the problem really is test it with users
5:18:31and then see how they respond and and
5:18:34iterate to improve your understanding of
5:18:38the the problem and then also like you
5:18:41grasping for that
5:18:43solution next technique this is more
5:18:46like a how to think about things
5:18:49technique the killer app problem is that
5:18:52you're trying to find that killer app
5:18:56and you're trying to take on too much so
5:18:59rather than trying to invent that whole
5:19:01new thing you can also look at something
5:19:04existing and all you need to do is be
5:19:08creative and novel in one
5:19:11aspect actually novelty and creativity
5:19:13is not the same thing so in my opinion
5:19:16and I always say this to my
5:19:18students um being novel is this idea of
5:19:21being the first at doing something and
5:19:25and that claim is is actually never a
5:19:28really strong claim because you could be
5:19:30the first person to do a really really
5:19:32bad thing to have a really really bad
5:19:34idea just because you're the first to to
5:19:37talk about it to write about it uh to
5:19:40implement it doesn't make it a good
5:19:42thing in itself and so being creative
5:19:44however means that you're like looking
5:19:46at something and you're trying to you
5:19:50maybe look at it differently or trying
5:19:52to innovate um by coming up with one
5:19:55small new thing and when I talked about
5:19:58design gems I already talked about this
5:20:00idea of trying to be creative in in how
5:20:04you approach the problem and how you
5:20:06execute uh the methods and how you
5:20:09prototype things uh how you implement
5:20:12things I think that is really cool that
5:20:14is where you want to be creative but you
5:20:15don't have to be novel in the whole idea
5:20:17that's very important so do this one
5:20:21thing and uh be disruptive
5:20:24there the next thing that I want to talk
5:20:26about is really how to uh frame a
5:20:30problem and how to kind of verify
5:20:33whether you're asking the right
5:20:34questions so I'm calling this the
5:20:37problem promise premise approach the
5:20:40first question I'm going to ask you
5:20:41about the thing that you're trying to do
5:20:43is what is the problem tackled by your
5:20:45exi solution so if there's no problem
5:20:47then you don't have a solution so if you
5:20:49have a hard time telling me what the
5:20:50problem is then you should rethink it
5:20:53also if you think you know what the
5:20:55problem is what is hard about it and why
5:20:58is it important and here I have a little
5:21:00denomination why is it important to you
5:21:03because this idea of trying to generally
5:21:04say why something is important I often
5:21:06struggle with this but I can tell you
5:21:09why some of the research that I'm
5:21:10working on is at least important to me
5:21:12cuz I observe how people struggle
5:21:14especially students how students
5:21:16struggle to learn about design because
5:21:18in my research on XR and specifically
5:21:20prototyping tools I have observed how
5:21:24novice designers really struggle with
5:21:26picking up XR Concepts and XR
5:21:28Technologies and that's what I at least
5:21:30I cared about and actually it turns out
5:21:33that this is really a problem and a lot
5:21:35of people are trying to innovate in that
5:21:37space not always have having good ideas
5:21:39but trying to at least innovate in that
5:21:41space so this agreement on then there is
5:21:43something that is a problem that is
5:21:46actually an indicator that you're onto
5:21:48something and so the next question I'm
5:21:49going to ask you what is the promise of
5:21:51your Exo solution so what are you
5:21:55offering uh where can XR actually help
5:21:59and uh what is your value
5:22:02proposition and so then I want to
5:22:04scrutinize more this idea of why XR so
5:22:07what is the parameter of your XR
5:22:09solution really ask yourself critically
5:22:11and carefully why can XR solve this and
5:22:14when I say XR you should actually be a
5:22:15little clearer and say whether augmented
5:22:18reality or virtual reality is the right
5:22:20approach and then what are your
5:22:23assumptions like with every solution
5:22:25there's always some kind of assumptions
5:22:27with every design we make assumptions
5:22:29write the inter first so if you think
5:22:31about an introduction uh now I'm talking
5:22:33about a paper what goes into that uh the
5:22:37problem and and that is at more like at
5:22:39a general level the obstacle now within
5:22:43that problem there is something more
5:22:44specific that you're trying to address
5:22:46so that is the obstacle and you should
5:22:48make it Crystal Clear what are you
5:22:49talking about in this bigger problem and
5:22:52then you should introduce the solution
5:22:54in essence not like the whole thing you
5:22:57have the rest of the paper to describe
5:22:58that but in essence the 10,000 ft view
5:23:01what is the solution here and then comes
5:23:04the hard part actually articulating your
5:23:07contribution science
5:23:09cifically um so this is where a lot of
5:23:12people struggle first year PhD
5:23:14students as well as old experienced
5:23:18professors really struggle to nail that
5:23:20contribution couple of things that I've
5:23:22observed over the years that really
5:23:23annoy me and so maybe you should try to
5:23:25stay away from those because I don't
5:23:26think there are really good arguments to
5:23:28make we are the first to do this as I
5:23:31said doesn't mean that this is a good
5:23:33idea and it's actually not a very strong
5:23:35argument it's now or never so like
5:23:38please take my paper now and yeah I've
5:23:41I've I've seen that a lot and timeliness
5:23:45is something that the scientific
5:23:46Community will really know you know if
5:23:50you're trying to solve an old problem I
5:23:52can tell you we are probably not making
5:23:54the right progress and this last thing
5:23:56is uh this is probably really speaking
5:23:59just to the scientific community that
5:24:01the fact that you evaluated Your
5:24:04solution that is not a contribution it's
5:24:06just expected We Are Scientists so
5:24:10obviously we evaluate our solution so
5:24:13don't claim it as a contribution now I'm
5:24:15done with my rant and we continue uh
5:24:17looking at some of the More Design
5:24:19oriented
5:24:32techniques so in this lecture I'm going
5:24:34to share seven techniques to help you
5:24:36innovate one technique that I really
5:24:38like is this idea of the idea hexagon
5:24:41introduced by ramish resca in a lot of
5:24:43his motivational talks and I really like
5:24:45it and I want to give some examples from
5:24:47the EXO space so the idea hexagon is
5:24:50basically given an idea x what is the
5:24:54next good idea from that idea one way to
5:24:57innovate is to try to extend to the next
5:25:00Dimension examples he gives is from
5:25:02flicker which is images to YouTube which
5:25:05is videos the next way to innovate is to
5:25:08to fuse the deimar so x + y and I'm
5:25:14going to have an example here in a
5:25:15second then the other way to innovate
5:25:19which is the one that I really like is
5:25:21the doing the exact opposite of what's
5:25:23been done before and we're going to see
5:25:24in a second whether you like my exra
5:25:26example here or not and then comes these
5:25:30given a hammer find all the nails and
5:25:32given a nail find all the Hammers it's a
5:25:35very popular approach especially in the
5:25:38technical human computer interaction
5:25:40design community and research Community
5:25:43um this idea of like coming up with a
5:25:45new system and then seeing what this new
5:25:49system or a solution what kinds of
5:25:51problems can it actually solve so having
5:25:53this idea because you think it's cool
5:25:56because you think it's creative because
5:25:58you think we kind of need this and then
5:26:01see what we can do with it in my opinion
5:26:03is actually very good way to innovate
5:26:05and ideate and do research but not
5:26:08everybody agrees so that's fine the last
5:26:11one that's missing here and this is
5:26:13actually the one that's done all the
5:26:15time when you look at all the new
5:26:16product announcements Etc is just adding
5:26:19your favorite adjective faster lighter
5:26:23better so these are some examples and
5:26:25now I'm going to translate this to the
5:26:27XO space taada here we go so taking it
5:26:31to the next Dimension my example here is
5:26:33hollow lens we've seen a lot of VR
5:26:35things so for example Oculus came out a
5:26:38cool really a cool headset and uh that
5:26:42was just virtual reality and all the AR
5:26:44stuff up to this point was relatively
5:26:46limited and then the Holland came out
5:26:48and then the Holland ported us into
5:26:50augmented reality but this idea that a
5:26:53CEO go walks up on stage and says this
5:26:55is a 5year mission we're trying to
5:26:57figure it out and we're using and we
5:26:59need you to help us figure it out that
5:27:01to me was really cool and I think it's
5:27:03the right approach and now we're going
5:27:04to look up there so here for the X+ y so
5:27:09Fusion of the disiner my example is the
5:27:12Oculus link so for this example I
5:27:14quickly stole you see from the back um
5:27:17the Oculus actually an Oculus the
5:27:19original Oculus which is Tethered so
5:27:22right so then Quest came out cool now we
5:27:27have it all on device not tethered
5:27:29Standalone dedicated headset pretty cool
5:27:32and then we're introducing Oculus link
5:27:34right we're going to give you a USB
5:27:37cable initially you had buy a specific
5:27:39one a 3.1 USB um so C cable and what it
5:27:44gives you is it turns your well
5:27:47Standalone dedicated into a tea device
5:27:50and turns your Oculus well Quest into a
5:27:54hang on essentially a rift s which had
5:27:57just come out but this idea of turning
5:27:59the quest into an Oculus Rift s to
5:28:03because for this one you really have to
5:28:04optimize your stuff to get it to work on
5:28:06mobile and for this one you you kind of
5:28:08can throw whatever because you just buy
5:28:10a better computer if it doesn't work I
5:28:12think that was quite disruptive and it
5:28:14was actually quite upsetting that this
5:28:17Quest can now be turned into an oadus
5:28:20Rift s because people just bought the
5:28:22rift s and now this one seems to be the
5:28:25one that Oculus is pushing forward which
5:28:28well we'll see where that goes and now
5:28:30we're going to jump over here so
5:28:33cardboard doing the exactly the opposite
5:28:36so the example that I have here is this
5:28:39kind of plastic version of cardboard but
5:28:41it's still cardboard okay um this idea
5:28:45of just doing something like this a
5:28:48device wrapper or actually giving you
5:28:50cardboard and putting that on your face
5:28:54and putting your phone inside
5:28:58um well that was like the opposite from
5:29:01everything else we had explored at the
5:29:02time so if you have a hammer like a
5:29:05smartphone what are all the things all
5:29:07the nails you can hit with it and AR and
5:29:10VR are actually things you can hit with
5:29:12it well because we have a hammer and
5:29:15turning this into uh an AR or VR display
5:29:18just by wrapping something around it I
5:29:21thought this was pretty cool now my
5:29:23other example is web XR down here so
5:29:29given a nail find all the Hammers so if
5:29:33you let's say if we now do AR and VR
5:29:37what are all the hammers that could
5:29:39benefit from it and we the web is a
5:29:42hammer that gives access as a platform
5:29:44to a lot of different content and so uh
5:29:47bringing arvr into the web and
5:29:50empowering the web is just going to take
5:29:52it to the next level and I think that's
5:29:54why I put web xar here as an example and
5:29:57finally coming back to the Oculus Quest
5:30:00well here the idea with the Oculus Quest
5:30:02was just like okay we've always had it
5:30:05somehow tethered and initially the first
5:30:08VR headsets were like really heavy and
5:30:10had to be mounted so the one adjective
5:30:13maybe to be associated with the headset
5:30:15is um mobile and portable and um
5:30:19pervasive if you want so now I want to
5:30:23jump gears a little bit and look more at
5:30:26innovating in interaction design so the
5:30:28object control design Matrix works as
5:30:30follows so you're putting out a 10x10 I
5:30:33don't actually draw a 10x10 Matrix here
5:30:35but the idea is you do a 10x10 Matrix
5:30:37and then you're thinking about um winter
5:30:41and cold and then you're putting down
5:30:44all the objects you associated with
5:30:46winter like snowshoes gloves hats boots
5:30:49ice
5:30:50scraper so you're putting those um here
5:30:55in the rows and then in the columns
5:30:58you're putting all kinds of controls
5:31:00that you can imagine like slider voice
5:31:03pressure control squeeze Shake dial
5:31:07whatever com comes to mind and then
5:31:09you're looking at the combinations of
5:31:11these and you're sketching out in each
5:31:14of these little quadrants you're
5:31:15sketching out all the different little
5:31:17funny combinations that you can come up
5:31:18with like an ice scraper that can be uh
5:31:21shaken now that would be an interesting
5:31:23device now we have a combination of
5:31:26objects that we can control in funny
5:31:28ways and if you sketch those ideas out
5:31:30this is really an activity you should do
5:31:31in class I've done this in a lot of
5:31:33interaction design classes um it's
5:31:35actually quite fun this idea of put
5:31:37pushing it out quickly like doing really
5:31:39a 10x10 Matrix like aund of these ideas
5:31:43pushes you to be quick sketch them out
5:31:46quickly ideate quickly don't criticize
5:31:51don't spend too much time on shaping
5:31:53this perfect idea no just go about it
5:31:55try out all these different combinations
5:31:58that is really cool like going all the
5:32:00way through line by line
5:32:03diagonally and I think that is why it's
5:32:06really a valuable technique
5:32:08it will not always work like it works
5:32:10really well for like warming up
5:32:13brainstorming so translating this
5:32:15directly to coming up with Innovative XR
5:32:19interface and interaction design ideas
5:32:21probably not going to work so well but
5:32:24as a warmup uh in a class or in a
5:32:27workshop I think this is actually really
5:32:30helpful
5:32:44one last thing that I want to share here
5:32:46is the threshold and seating
5:32:48considerations in tool design now this
5:32:50is something that uh comes from a paper
5:32:52from 2000 by Brad Meers and colleagues
5:32:56on the past present and future of user
5:32:58interface software tools and I think
5:32:59it's a really cool way to look add tool
5:33:03design now whether you have a new tool
5:33:05now this could be a user interface
5:33:07design tool or some kind of tool that
5:33:10helps you solve a problem let's just
5:33:12call it tool and then you look at the
5:33:14whole design space of all the things you
5:33:16could do in XR and then there is your
5:33:19tool and then your tool is defined by a
5:33:22threshold now that is the barrier of
5:33:25Entry that you have to overcome in order
5:33:27to use this tool so basically all the
5:33:29stuff you have to learn to use the tool
5:33:31and then the ceiling which is all the
5:33:34things you can do with this tool but no
5:33:36more it's like the upper bounds it's
5:33:37like that's it all right learn learn all
5:33:41this to get in here use the tool and
5:33:43then all you can do with the tool is
5:33:44here and then there could be so many
5:33:46other things for which you probably need
5:33:47a different tool or some combination of
5:33:49tools and now that is an interesting
5:33:52consideration given that there's so many
5:33:53different kinds of tools out there
5:33:55especially all these kinds of
5:33:56prototyping tools that essentially all
5:33:58do the same but it actually allows you
5:34:00to compare some of the solutions out
5:34:02there some of the tools out there and I
5:34:05think it's a really good way of
5:34:08examining whether the effort that you
5:34:10have to put in in learning something
5:34:12justifies the kinds of things you can do
5:34:14with it so the seaing that you will
5:34:15reach now one of the really popular
5:34:18tools out there in the arvr space is
5:34:20Unity Unity has actually a quite a high
5:34:24threshold like a really like high
5:34:27learning curve um okay depending on your
5:34:29background you may have an easier time
5:34:31but the first time I open up Unity I was
5:34:33overwhelmed uh but the cool thing about
5:34:35Unity is once you have crossed the
5:34:37threshold you're in this land and you
5:34:39can start to learn all the things you
5:34:41can do with unity and then I would say I
5:34:43wasn't really sure where to put this
5:34:45ceiling the seating is probably
5:34:46relatively high but nobody knows the
5:34:49whole design space of XR and I'm not
5:34:51sure whether I can do everything in XR
5:34:54now in my research I really try to have
5:34:57a much lower threshold so I really want
5:34:59more people to be able to pick this tool
5:35:02up or my tools up more quickly and I'll
5:35:05do a trade-off because obviously I can
5:35:07really not reach the same seiling as
5:35:10unity and I don't even have to because
5:35:13I'm not even trying to replace Unity
5:35:15Unity has its place in the arvr space
5:35:20and my tools are just helping more
5:35:23people transition into tools like Unity
5:35:25so empowering more designers and that is
5:35:27really the idea of my research so for me
5:35:29it's okay to make that
5:35:31tradeoff uh so basically the ceiling is
5:35:34lower of my tools but that's fine but
5:35:36there are also Lots of little problems
5:35:38in the design space for which Unity
5:35:41isn't really a good solution because it
5:35:43has so much overhead you really have to
5:35:45learn so much and then solve that thing
5:35:49no you just use a different tool and so
5:35:52that's why this graphic is probably also
5:35:53useful because there are lots of
5:35:56prototyping tools out there similar to
5:35:58mine although my emphasis has been on
5:36:00physical to digital transition which I
5:36:03don't think a lot of tools do very well
5:36:06um and so and so that's my claim here
5:36:09okay just saying as a researcher and
5:36:11that's fine it's been peer reviewed so I
5:36:13can say that now this is an interesting
5:36:15area here so there is this overlap
5:36:18between some of the things you can do
5:36:20with my tools and some of the things you
5:36:21can do with unity and and and then
5:36:24that's probably is the same thing for a
5:36:26lot of tools out there you can use a lot
5:36:29of tools to do the same things now
5:36:31obviously we not trying to be like Unity
5:36:34I would never try to do that so I don't
5:36:36want my goal is not to be congruent to
5:36:39actually be like Unity no you don't want
5:36:42to be like that tool you want to make it
5:36:45simpler obviously the coolest thing we
5:36:47could do is like lower the barrier to
5:36:50entry and then have an even higher
5:36:52seating as unity and um that is really
5:36:57unrealistic um just like how if it were
5:37:00possible Unity would do this now I think
5:37:03unity set that threshold in order to
5:37:06enable you to reach this really high
5:37:09seiling but that threshold is
5:37:10unfortunately a little high but by the
5:37:13time you want to solve all these upper
5:37:14problems you're already quite committed
5:37:17so it's an okay cost for them um and so
5:37:21these problems here well you can you can
5:37:24do them with whichever tool I feel like
5:37:26if you already have the knowledge of my
5:37:28tools you could just continue using my
5:37:29tools and solve the problems in the
5:37:31space if you already know unity and this
5:37:35is the stuff that you want to solve
5:37:36there's no need to learn my tools so
5:37:38it's that simple you can just look at it
5:37:40that way and I think it's a really cool
5:37:42way to illustrate the the the power and
5:37:46the use of tools and really also
5:37:50estimating how much learning is required
5:37:52in order to pick up a tool and then
5:37:54again what you can actually do with a
5:37:56tool and this knowing what you can
5:37:58actually do with a tool that is the
5:38:01really hard part because unfortunately
5:38:04the tool developers the vendors are not
5:38:07always very honest about that and so
5:38:10that's something we need to improve so
5:38:12this was a little bit of a fun lecture
5:38:14but it's actually really really
5:38:16important one and I wanted to give it
5:38:18this a little bit of a fun spin so that
5:38:22you don't see that I'm actually dead
5:38:24serious but I am so um this lecture
5:38:27comes is really heartfelt and comes from
5:38:30years of experience as a PhD student
5:38:32myself really trying to learn uh how
5:38:35this scientific Community works how to
5:38:38come up with good research problems um
5:38:41which especially in the XR space where
5:38:42so many things are going on is really
5:38:45really hard and um and then also from my
5:38:48teaching experience both in traditional
5:38:51relatively boring mobile and web design
5:38:53interaction design classes and to some
5:38:56of the more advanced courses that I'm
5:38:57teaching now that I'm fortunate to teach
5:38:59now uh to both non-technical and really
5:39:03technical highly skilled designers and
5:39:06Developers because I want everybody to
5:39:08be able to participate in this XR design
5:39:24space so I'm going to introduce you to
5:39:26these two piles of good ideas and bad
5:39:28ideas and um you can use them as a
5:39:32checklist like the things that I say if
5:39:35that is true about your idea then maybe
5:39:37it's a good or a bad idea um well I
5:39:41don't want to offend you I could still
5:39:42be wrong it's just an intuition that
5:39:45I've developed over the years so if you
5:39:46can say this about your idea you're
5:39:48adding a significant new feature to an
5:39:50existing Solution that's a cool idea do
5:39:54it you're doing a complete redesign of a
5:39:56solution uh well actually that's not a
5:39:59good idea let me fix this so I'm going
5:40:01to go in here you're doing a
5:40:05redesign of part of an existing
5:40:11solution now that is a good idea okay
5:40:15let's look at it so you're doing a
5:40:17redesign of part of an existing solution
5:40:20don't do the full redesign of something
5:40:22existing don't redo Amazon there
5:40:25probably something good about it's it's
5:40:26quite popular and stuff so don't so if
5:40:29you can say this about your idea then
5:40:30you probably have a good idea you're
5:40:32picking one particular aspect of a
5:40:34problem and try a different solution now
5:40:37you there's a big problem and maybe lots
5:40:39of solutions out there but then you're
5:40:41trying to focus on this narrow thing and
5:40:43you're trying it slightly differently
5:40:45that I think is a really good way and
5:40:47this is similar to what I'm saying here
5:40:48you're looking at an existing problem
5:40:50through a new lens now you're actually
5:40:52like looking at this whole problem and
5:40:54you're recasting it from this different
5:40:56lens and now you're not coming up with
5:40:58the same solution but you're coming up
5:41:00with a slightly different solution and
5:41:02that solution might be novel or at least
5:41:05Innovative in some way especially if
5:41:07that new lens is really a new lens like
5:41:09nobody has looked at it this way yet I
5:41:11mean that lens is not new but nobody has
5:41:14actually applied that lens to the
5:41:15problem then I think it's a very good
5:41:17way to innovate and this is actually
5:41:18what a lot of researchers do and it's
5:41:20actually very good way to make progress
5:41:22so looking at the candidates for bad
5:41:25ideas so if you can say something like
5:41:27this about your problem or your project
5:41:30then may not be the best idea you're
5:41:33just adding a small new feature to an
5:41:35existing
5:41:36solution you probably agree with this
5:41:38unless you are the designer of that
5:41:41existing solution in which case you're
5:41:42creating a new version of it which is
5:41:45fine you should really not be saying
5:41:47anything like this about your idea
5:41:49you're trying to make an existing
5:41:52solution much better now I've heard this
5:41:55all the time so the problem with the
5:41:57much better part well in theory it's a
5:41:59cool idea but the problem is if you
5:42:02can't say the much better if you can't
5:42:04quantify if you can't specify in which
5:42:07way you're making it much better then
5:42:10it's really a problem so a good way to
5:42:12specify something like this would be
5:42:13like I'm making it easier to learn it or
5:42:17I'm uh automating this part which makes
5:42:19it faster to do that part and then
5:42:22you're trying to do what multiple
5:42:24existing Solutions do in one new
5:42:26solution oh yeah you're doing the
5:42:30dashboard um you know trying to wrap
5:42:32around things and then coming up with
5:42:34this whole new thing that brings
5:42:35everything together and makes it better
5:42:37not a good idea first of all if that is
5:42:40your project and you're like working in
5:42:41a class or something like this you have
5:42:43only so and so much time actually every
5:42:45project is time limited so abandon that
5:42:47and refocus and reshape finally you're
5:42:51trying to do what nobody else has done
5:42:53before really now I've heard that so
5:42:56many times and it's probably never ever
5:43:00true the problem might be that you don't
5:43:02know whether somebody has done it before
5:43:04so you haven't done your homework you
5:43:05haven't read the literature you have you
5:43:07haven't tried other the tools you
5:43:08haven't played around with this existing
5:43:11technology uh maybe you didn't know
5:43:12about it well that's fine so run your
5:43:15ideas by people could be professors
5:43:18could be other designers in your team
5:43:20could be your users your your future
5:43:21users or any kind of stakeholders and
5:43:23that's a good way to shape and
5:43:25reformulate ideas now if your ideas fall
5:43:29into these calories like the good pile
5:43:32or the bad pile there's always Ro room
5:43:35for improvement doesn't mean that if
5:43:36you're here that it's safe and you know
5:43:39you'll be fine doesn't mean that you
5:43:41could still have uh introduce issues as
5:43:44you're executing the ideas yeah you're
5:43:46not really involving users for example
5:43:48when you're designing a new product now
5:43:50you maybe you had a really cool idea in
5:43:52the beginning but you could still mess
5:43:53it up as you're trying to implement that
5:43:55idea H and so now these are not uh fail
5:44:00saves or guarantees that your product
5:44:03will be successful but really if you're
5:44:07in this pile you should really
5:44:10reframe your problem a big problem is
5:44:13really about scope most of the time
5:44:15you're taking on something too big like
5:44:16the dashboard thing I just talked about
5:44:19res scope make it smaller now that's a
5:44:22good way to innovate as well because
5:44:25it's easier to come up with new things
5:44:27in a much more constrained and small
5:44:29problem don't make it too small
5:44:30shouldn't be trivial that's the other
5:44:32thing that I sometimes complain about
5:44:34when I work with students on projects so
5:44:36don't make it get too small there should
5:44:38still be room for exploration creativity
5:44:41and in some sense obviously Innovation
5:44:44and that's what this whole lecture was
5:44:45about so I hope you enjoyed some of
5:44:47these lessons uh in this case I Tred to
5:44:49be a little bit more fun than usual and
5:44:51we cut this slightly differently it's uh
5:44:54supposed to be stimulating and inspiring
5:44:57and not as serious but there are
5:44:59actually really valuable lessons in here
5:45:00I think these are uh cool techniques
5:45:02that help me at least try to innovate
5:45:05and come up with the next research
5:45:06problem and maybe they'll also help you
5:45:08as you're thinking about the next
5:45:10solution for a real problem and a real
5:45:13XR solution in that space now that would
5:45:16be
5:45:30amazing storyboarding and wireframing uh
5:45:34that's what I'm going to talk about in
5:45:35this segment since these are really
5:45:36important ear stage design steps so I
5:45:39laid out the larger XR prototyping
5:45:42process um storyboarding and wireframing
5:45:44physical prototyping and digital
5:45:46prototyping we go through each of these
5:45:47steps but in this segment I'm just going
5:45:49to focus on the first one storyboarding
5:45:51and wiof framing so storyboarding and
5:45:55wireframing here's what what we mean by
5:45:57this so you basically sketch out the
5:46:00interface ideas that you have and um
5:46:03organize those sketches into what we
5:46:04call a storyboard and the cool element
5:46:08of the sketches is usually when they are
5:46:11a little bit more precise uh than usual
5:46:13we call them wireframes they're actually
5:46:15blueprints of the screens and
5:46:17potentially 3D scenes that we'll see
5:46:20later in our arvr
5:46:23interfaces all right and what I want to
5:46:25show you here is basically this is from
5:46:27one of our design jams where students
5:46:30were explaining to me how they have been
5:46:33thinking through this prototype of an
5:46:37furniture placement app it was inspired
5:46:40by some of the products that Ikea and
5:46:42Wayfair have out there now so you could
5:46:44choose uh between different types of
5:46:46furniture and then rotate for example
5:46:48the chair here and um also color and get
5:46:52like real previews of the furniture that
5:46:56you pick with specific choices like
5:46:59color and size through a menu and so
5:47:04this is how the app would work what I
5:47:05find interesting about the storyboard is
5:47:08obviously contains a lot of detail um
5:47:10you can really see how the the AR
5:47:13interface blends with the 2D menus that
5:47:15you see on the left and at the bottom
5:47:18and the students did really do a good
5:47:19job thinking this
5:47:21through so what we just saw is actually
5:47:23just one example of the types of
5:47:25storyboarding techniques that I'd like
5:47:27to cover so the way I'm going to do this
5:47:29is I'm going to think these different
5:47:31techniques I'm going to introduce you to
5:47:32a few techniques here and I'm going to
5:47:33think them through from low Fidelity to
5:47:36high fidelity
5:47:37and I'm also going to think them through
5:47:39in terms of like what I think is easier
5:47:41to do and what I think is harder
5:47:42requires more practice and potentially
5:47:44technical skill so the four techniques
5:47:47we're going to cover is storyboarding
5:47:48which you just saw so traditional
5:47:50storyboarding which I think is good for
5:47:52organizing sketches into a story so and
5:47:54into a sequence we're going to talk
5:47:56about 360 photo stories um where you
5:47:58would essentially use a 360 camera um to
5:48:01capture physical space around you and
5:48:04you treat these physical uh environments
5:48:06that you capture as like prototypes as
5:48:08mockups of your virtual reality
5:48:09environment so you actually prototype in
5:48:11the physical space come in with the
5:48:13camera capture and create content that
5:48:15you can L use in VR pretty
5:48:17powerful and then I'm going to talk
5:48:19about 360 storyboarding this is actually
5:48:21something that I've worked on and looked
5:48:23at in detail in a lot of my own research
5:48:26uh I believe in some of these techniques
5:48:27a little bit harder to do but I think
5:48:29they're actually very powerful
5:48:31especially if combined with some of the
5:48:33tools that I've working been working on
5:48:35and I'm going to show you why this is a
5:48:36good technique for storyboarding and why
5:48:38it allows you to think around the user
5:48:40which really spacially thinking around
5:48:41the user is really important for AR and
5:48:43VR
5:48:44interfaces finally I'm going to cover 3D
5:48:47storyboarding where like inside a VR
5:48:49headset and you're sketching out
5:48:51essentially the 3D world around you
5:48:53which I think is a very good way of
5:48:55visualizing uh in 3D but it also
5:48:57requires quite a bit of practice it's
5:48:59the closest to the final prototype if
5:49:02you want uh but it's still a storyboard
5:49:05so keep that in mind so now I'm going to
5:49:07walk you over each of these techniques
5:49:09in a little bit more detail so 360 photo
5:49:11remember we're going to use a 360 camera
5:49:14we're going to capture an environment
5:49:15like for example my lab in this example
5:49:18um and uh the way it looks initially
5:49:20very distorted uh different types this
5:49:22is accurate angular projection so
5:49:25different formats in which these cameras
5:49:26actually capture this but what you can
5:49:28do with it you can actually map it into
5:49:30a 3D sphere and that 3D sphere is
5:49:33essentially around the user and this is
5:49:35how 360 photos and videos that you may
5:49:38have seen on social media this is how
5:49:40they actually work you map it back
5:49:42around the
5:49:43user okay so here we are like in my lab
5:49:47and uh you've probably seen these
5:49:48experiences before so this is 360 photo
5:49:51and um it was obviously captured at this
5:49:54very position in this case it was
5:49:56captured by my uh former postto Max in
5:50:00the lab and it actually shows um it's a
5:50:03capture of when I remember just starting
5:50:06out in my lab we didn't have as much
5:50:07equipment as you can see um you may see
5:50:10later portions in this uh moo
5:50:12specialization where we have a lot more
5:50:14stuff in the lab and you will not even
5:50:15recognize this anymore but as you can
5:50:17tell um we could use this 360 photo now
5:50:21and actually prototype a virtual reality
5:50:24application for the lab and we have
5:50:26relatively good understanding of the
5:50:29spatial layout of this room and um we
5:50:31may also even use this as a placeholder
5:50:33for the environment and prototype AR
5:50:36experi experiences without actually
5:50:37being in that physical
5:50:39location okay and if you now put several
5:50:42of these 360 photos together you can
5:50:44actually build a story and that way
5:50:47prototype for example what it would be
5:50:49like uh to walk through campus like at
5:50:52the University of Michigan or like
5:50:54finding my lab finding my office and
5:50:56those kinds of things and this is a
5:50:59traditional way of thinking of photo
5:51:00stories but the way I'd like us to think
5:51:02about it is like as a way of
5:51:03storyboarding interactive experiences uh
5:51:06some of them may involve navigation in
5:51:08this traditional sense um they could
5:51:10also be different steps in a sequence
5:51:13and an interaction technique uh so we we
5:51:16will we will spend more time with some
5:51:18of these techniques in the activities
5:51:20that I have designed for you um but uh
5:51:22this would be 360 Photo Story
5:51:25boarding okay so here you can see this
5:51:28360 Photo Story in sequence basically U
5:51:31Max took these photos and then he was
5:51:34going down the stairs um um and you can
5:51:38like experience that from these
5:51:40different perspectives and you get a
5:51:42sense for oh he's actually walking
5:51:45towards the lab and um this is a pretty
5:51:49standard Photo Story the way you could
5:51:51think of it but each of these 360 photos
5:51:54would give us a an environment that we
5:51:57can actually prototype for right we have
5:51:59relatively good 360d overview we could
5:52:01do something when people exit the
5:52:02elevators and come to the school of
5:52:04information or when they for for example
5:52:06come over to my lab and we have a lot of
5:52:09visitors in my lab and um this is the
5:52:11photo that I had earlier and um this is
5:52:15I show you this in a prototyping
5:52:19software that I'll make available to
5:52:21Learners and um it's a very good way of
5:52:24creating 360 photo stories and actually
5:52:27prototyping out different kinds of
5:52:28interaction sequences for example here
5:52:31standard navigation tasks but there
5:52:33could be different steps in uh multi St
5:52:36interface with different changes to the
5:52:39interface let's come back to the idea of
5:52:41360 storyboarding what you see here is a
5:52:43template and the way this works and it's
5:52:46described in more detail is this is what
5:52:48I would see right in front of me if I
5:52:50follow some of these grid lines I'm
5:52:51basically sketching content that appears
5:52:53above the user similarly this would be
5:52:55content that appears 90° to my left 90°
5:52:59to my right content that appears at the
5:53:01bottom it's actually really powerful
5:53:02template and um in some of our
5:53:05activities you can explore the template
5:53:06in more depth I want to show you a
5:53:08couple of examples of what we can
5:53:10actually do with this template so what
5:53:13we have done is for example creating a
5:53:15scenery where we put butterflies in the
5:53:17middle um so the user will be seeing
5:53:19these butterflies and they have a doggy
5:53:21and a car and then we add a second layer
5:53:24to it where we actually then show trees
5:53:26and we're starting to prototype on paper
5:53:28storyboarding still um out the scene
5:53:31where you're suddenly in this
5:53:32environment so now we have three layers
5:53:34mountains in the background the doggy
5:53:36and the butterflies in the foreground
5:53:38and the trees in the
5:53:41midr so how do we do this um yeah I just
5:53:44explained all these layers to you and
5:53:46then what will happen is if we add these
5:53:48ideas from 360 photos that I talked
5:53:51about earlier we can map them onto a
5:53:53sphere and now the user will be
5:53:56navigating inside the sphere as they put
5:53:58on their VR headset and the way this
5:54:00works then is that they actually just
5:54:02see this experience prototype it will be
5:54:04actually flatten out in some sense and
5:54:06I've just have this experience doggy
5:54:09butterflies trees and
5:54:11mountains okay so now I've composed the
5:54:13scene and you can see a little bit of
5:54:15Parallax so we're basically spreading
5:54:17out these layers that we have between
5:54:19the foreground the mid-ground and the
5:54:21background and this gives this effect
5:54:22that now we can actually walk towards
5:54:24these butterflies and we can turn around
5:54:26and can see the doggy and the car and
5:54:29the trees are there and so we just
5:54:32composed this from three layers of a 360
5:54:35storyboard it's actually really quite
5:54:37cool and you can play around with these
5:54:39parameters like how far do you want each
5:54:40layer to be separated from the other and
5:54:43so to create a more realistic experience
5:54:46and
5:54:59preview I've talked about foreground
5:55:02mid-ground and background and now what
5:55:04we can also do is actually move out some
5:55:06of these spheres um that we are mapping
5:55:08these 360 story boards onto and so we
5:55:11can create an experience like this Star
5:55:13Trek Experience where it's possible for
5:55:15the user to actually walk into the
5:55:18observation deck that is just next door
5:55:20the bridge and I'm going to show you
5:55:21this as well and a quick example here
5:55:24what I'm going to do now is walk around
5:55:26and I'm going to walk into this room
5:55:27over there so we have to walk around the
5:55:29bridge a little bit physically
5:55:31constrained here but the beautiful thing
5:55:33is that this room nicely the physical
5:55:35room nicely Maps into our AR version of
5:55:38it and I in the room and I have some
5:55:42seethrough um effects here and we can go
5:55:46back into the other room hop hop
5:55:49hop and there we have
5:55:52space this is actually the film studio
5:55:55where we're working and we're back at
5:55:59the captain's
5:56:00chair all right I talked a lot about
5:56:03storyboarding and so in this case I want
5:56:05to specifically focus on 360
5:56:07storyboarding now 360 storyboarding I
5:56:10brought like basically three examples
5:56:13here and I want to briefly show what is
5:56:16required to create those examples so one
5:56:18of the things you see throughout the
5:56:19course is this scene with uh the doggy
5:56:23and the
5:56:24butterflies and then we have a little
5:56:26bit of background like the trees here
5:56:30and we have mountains in the background
5:56:32and I explained how these are different
5:56:34layers um that we can then and kind of
5:56:37like remove the white and layer them and
5:56:39actually have a really cool experience
5:56:40that way now the way we created these is
5:56:43through a 360 grid that I kind of like
5:56:46try the line here and you can see that
5:56:50the butterflies are in the center and I
5:56:52talked about this GD in a lot more
5:56:53detail so next I briefly want to talk
5:56:56about what it's like to prototype an
5:56:58existing experience so if you are
5:57:00familiar with the Star Trek this is the
5:57:02bridge with the captain's chair and
5:57:04that's the bridge behind and then what
5:57:05we also did is we actually created a
5:57:08room um that is the observation deck and
5:57:11we when we create this experience in VR
5:57:14we can actually move these spheres um so
5:57:17that they're intersecting you can
5:57:18actually walk over into this other room
5:57:20now what is very interesting when we
5:57:21created this experience this was
5:57:23actually an 8h hour prototyping
5:57:25experiment um with KD a very talented
5:57:27designer in my lab and we also used uh
5:57:31additional tools so one thing that we
5:57:32were actually really a little concerned
5:57:34about was the amount of movement that is
5:57:37required to really explore the Star Trek
5:57:40and what users would see initially when
5:57:42they start this experience so this is
5:57:44giving us the range of motion of a user
5:57:46when they are seated in VR and um to be
5:57:49honest this experience uh the Star Trek
5:57:51we prototype for both VR and for AR so
5:57:54the more interesting thing for us was
5:57:56the field of view so what would a user
5:57:59especially when they are starting out I
5:58:01will show you later the final experience
5:58:03to the user but when they are starting
5:58:05out what would would what would be the
5:58:07initial field of view and how much of
5:58:08the bridge would they actually see and
5:58:11so these are these are some useful
5:58:13templates and tools that I share with
5:58:15you so you can actually prototype some
5:58:16of these experiences and learn about um
5:58:19how to do this actually well on paper
5:58:22one of the issues with paper is that
5:58:23things often appear uh different in
5:58:25terms of scale and so you will actually
5:58:27have to experiment a little bit with
5:58:29this so um a good way of prototyping
5:58:32this is actually putting a blank sheet
5:58:34of paper on top of our templates and
5:58:36then see the pattern shine through and
5:58:39so you can Orient yourself around the
5:58:41center and actually start drawing
5:58:42content in the middle content that would
5:58:46appear and then content that would
5:58:49appear to the left of the user would
5:58:51would appear over here content that
5:58:53would appear to the right to the user um
5:58:55so 90° to the right you draw in there
5:58:57and you can also uh draw the top and
5:59:00bottom so you can actually draw 360
5:59:02around the user so it's a really
5:59:03powerful
5:59:04template uh finally
5:59:06one way we often use this is actually if
5:59:08you want to prototype experiences for my
5:59:11lab at Michigan we also have a template
5:59:13um but this essentially a 360 sketch of
5:59:16of my lab now we wanted to try out how
5:59:19difficult it is to prototype W angular
5:59:22rooms using a 360 template you can see
5:59:24lots of distortions here just the way it
5:59:27is but when you actually view it in 360
5:59:30on a VR headset it will be evened out
5:59:32it's right it's like it goes around the
5:59:34user it's like wrapping it around their
5:59:35head and so these what appears to be
5:59:38distorted are actually rectangular
5:59:39shapes and then suddenly they become
5:59:41windows and walls and this is a very
5:59:44good way of using VR techniques to
5:59:46prototype AR
5:59:49experiences finally once we have created
5:59:53uh some of these prototypes we often use
5:59:56cardboard to rapidly preview these kinds
5:59:58of things and I'll introduce it to tools
6:00:01where all it takes is like put your
6:00:02phone in there and take a picture this
6:00:04way and then start previewing the
6:00:06experience in VR it's something I'll
6:00:08talk about and then we'll also talk
6:00:11about some of the AR um ways of
6:00:14prototyping things we make use of
6:00:16markers uh fiducial markers that are
6:00:18visible to the camera and so when we
6:00:20prototype these experiences we not only
6:00:22work on paper we often want to actually
6:00:25quickly preview things in VR and AR and
6:00:27so the way we often do this is at least
6:00:29in my classes we use software that I've
6:00:31developed that can run on our phone and
6:00:33all we need to do is actually put the
6:00:35phone inside car
6:00:38cardboard run the software take a photo
6:00:42of the experience click and then preview
6:00:45it in VR and these are things that I'll
6:00:48actually show you um throughout the Muk
6:00:52and uh we can do similar stuff for AR so
6:00:55we actually I also brought a few paper
6:00:56markers here that are popular and there
6:00:59is a cardboard version of um AR which is
6:01:02called holoid and so same idea phone
6:01:05goes in there the experience looks a
6:01:08little bit different you have to make
6:01:10sure that the camera is exposed
6:01:12otherwise it will not work so it has to
6:01:14go inside uh this way
6:01:17and we run the software we put it on our
6:01:21heads and we actually use these
6:01:23markers to tell the phone where the
6:01:27coordinate system of our um prototype
6:01:31and then we can actually preview fairly
6:01:33quickly any of the things we have
6:01:35prototyped here actually preview that n
6:01:38AR so I've talked about storyboarding
6:01:41I've talked about 360 photo stories and
6:01:43I've talked about 360 storyboarding none
6:01:45of them are actually 3D and so this last
6:01:48technique is storyboarding in 3D all
6:01:51right I want to show you 3D sketching
6:01:54and the way I'm going to do this is I'm
6:01:56actually also explaining to you how we
6:02:00actually created this experience so what
6:02:02I'm doing right now is actually
6:02:03sketching out the room that I'm in okay
6:02:07and it is as you can tell it's already a
6:02:103D kind of uh room but there's nothing
6:02:13interesting in it so where is Michael in
6:02:16this room okay so this is pretty easy
6:02:18I'm kind of like here standing and
6:02:22waving at you so that's me okay and you
6:02:25guys are at the other side of the room
6:02:27obviously um the camera is roughly here
6:02:30and it's pointing towards me and you
6:02:33can see it I'm never sure which is the
6:02:37best perspective for you guys cuz it's
6:02:38like 3D but you know this is kind of
6:02:41like the experience that you get and how
6:02:44do we do this actually how can you
6:02:45actually be part of uh the experience so
6:02:47the way we record this is there's
6:02:48actually a green screen behind me okay
6:02:53now this is what it looks like and
6:02:57um well you don't see the green right
6:03:00you have this really cool interesting
6:03:01background behind me from tiltbrush and
6:03:04the way we do this is we're using
6:03:05software that is essentially removing
6:03:08the green screen uh through chroma
6:03:10keying so it's gone and we can fill in
6:03:14some really cool stuff in the background
6:03:16we could actually do anything we could
6:03:18have a rainbow or other things behind me
6:03:21and create a really interesting
6:03:22experience for you that way and
6:03:25obviously while this sounds like movie
6:03:28production uh I mean as you can tell
6:03:30from how I talk about some of these
6:03:31Concepts when it comes to XR it actually
6:03:34seems very related there a lot of
6:03:35technologies that blend and blur between
6:03:37movie making virtual production and
6:03:39actually arvr content creation now this
6:03:42is a 3D sketch and it's very useful in
6:03:44the early stages of design but it also
6:03:47is actually could be one of your first
6:03:49assets you could actually export this
6:03:50into other kind of arvr software and
6:03:53process it further so that it can appear
6:03:55in AR or and VR which it is right now
6:03:58but we can process it further as I said
6:03:59and could really do a really cool
6:04:01experience that
6:04:03way so I've introduced you to four
6:04:06powerful storyboarding techniques and uh
6:04:09the extent of wireframing really built
6:04:10into into these techniques really
6:04:12depends on the type of storyboard that
6:04:14you are creating so if you're paying
6:04:15attention to the detail the traditional
6:04:17storyboards are usually very rich in
6:04:18terms of 2D wireframes but traditional
6:04:21storyboards are quite limited when it
6:04:22comes to arvr scenes which are in 3D so
6:04:25I've introduced you two additional
6:04:26techniques like the 360 storyboarding
6:04:28technique which allows you to plan
6:04:31layout around the user uh and 360° but
6:04:34but you don't have depth it's it's not
6:04:35really 3D so the last technique I was
6:04:38just talking about 3D sketches really
6:04:40allow you to detail the content around
6:04:42the user plan it out in 3D space in fact
6:04:45you know draw it in 3D and then you can
6:04:48actually create assets that way and
6:04:49download them and process them further
6:04:52so that they can actually become
6:04:53elements in your final arvr
6:04:56experiences I want to wrap up with a few
6:04:58tips here so I would say start with the
6:05:01high level sketches first add gradually
6:05:03more detail but only what matter to the
6:05:06experience for example don't Focus too
6:05:08much on the physical world around it
6:05:10only on specific elements that dictate
6:05:12the AR or VR experience for example if
6:05:14the room that you're prototyping for has
6:05:16a very particular shape then you should
6:05:18really pay attention to this otherwise
6:05:20it's just a rect angular environment and
6:05:22you don't have to draw out every wall um
6:05:24what I've noticed a lot is really when I
6:05:27work with my students and there's this
6:05:29implicit notation uh that uh comes up
6:05:33where that people use to distinguish
6:05:35between drawing out the physical
6:05:37elements and drawing out the virtual
6:05:38elements really the overlays and so what
6:05:40I often see is like for example color
6:05:43being used so red is virtual and green
6:05:45is physical and if that is used
6:05:47consistently it really becomes a good
6:05:49notation that allows you to
6:05:51communicate W these storyboards actually
6:05:53translate to design and later uh design
6:05:56specifications when you work with
6:05:58developers finally I would communicate
6:06:01the type of devices being supported I
6:06:03would pay a lot of attention to what
6:06:05kind of experience you have in mind so
6:06:06keep in mind uh we have lots of
6:06:08different types of arvr it could be
6:06:10handheld it could be headworn and you
6:06:12should really keep that in mind even at
6:06:14the storyboarding stage however at that
6:06:17stage we are still brainstorming a lot
6:06:19and storyboarding is a very powerful
6:06:22tool um for brainstorming new kinds of
6:06:24experiences you want to create for arvr
6:06:41physical prototyping that is really one
6:06:44of the most important lectures to me and
6:06:46I've actually split it up into two
6:06:47lectures here and I want to introduce
6:06:49you to a few physical prototyping
6:06:51techniques So Physical prototyping is
6:06:53really in the larger XR prototyping
6:06:55process it is actually between
6:06:57storyboarding and digital prototyping
6:07:00and I think personally physical
6:07:01prototyping is very powerful and it's
6:07:04actually worth spending quite some time
6:07:06on it because it'll save you a lot of
6:07:08time later if you get the physical
6:07:09prototypes right you learn a lot about
6:07:11the digital prototypes and you cut cut
6:07:13some Corners there iterate faster and
6:07:15that's why I think this is really
6:07:16important in the larger prototyping
6:07:19process so I'm going to show you a few
6:07:21examples of physical prototypes created
6:07:23in some of my classes and in some of our
6:07:25research okay jumping right in this is
6:07:28from one of our design jams where I was
6:07:31prototyping an AR furniture placement
6:07:33app with students you can see that the
6:07:35final experience is supposed to be on a
6:07:37smartphone there's a menu at the bottom
6:07:39and the user can choose different
6:07:40options to switch
6:07:43Furniture so in this segment I'm
6:07:45actually going to talk about a few
6:07:46different kinds of techniques I'm going
6:07:47to use the same analysis we did before
6:07:50how do we move from low fi to high-fi
6:07:52and what other techniques in terms of
6:07:54their relative difficulty along the easy
6:07:56to hard Spectrum so we just saw this
6:07:58example of paper prototyping uh which I
6:08:01think is a very good way of thinking
6:08:03through at least the basic interactions
6:08:05might be limited for some specific arvr
6:08:07interaction techniques that you need to
6:08:08prototype and so I'll also introduce you
6:08:11to 360 paper prototyping which is
6:08:14basically taking the 360 storyboards
6:08:17that I showed you earlier taking these
6:08:19ideas further to the next level so that
6:08:21it's actually possible for users to
6:08:23quickly prototype on paper and then
6:08:25experience those prototypes in AR and
6:08:29VR I'm also going to talk about diaramas
6:08:31diaramas are Miniature 3D physical
6:08:35prototype of the arvr scenes we want to
6:08:37create which I think is a very good way
6:08:39of thinking through prics and by prics I
6:08:42mean the spatial relationships between
6:08:44the people so the users of our arvr
6:08:46experience which could be more than one
6:08:49and the devices that they use as well as
6:08:52any kind of physical and virtual objects
6:08:54in our arvr experience and you can
6:08:55Prototype all of this physically without
6:08:58having to do anything with digital tools
6:09:01finally we're going to talk about
6:09:02physical prototyping and by actually
6:09:04using modeling clay which I think is a
6:09:07very fast way of modeling quickly in 3D
6:09:11personally I'm not very good with 3D
6:09:12modeling tools if you have an
6:09:14architecture background maybe you want
6:09:15to skip this step but in general I think
6:09:17in the larger physical prototyping
6:09:19workflow 3D modeling with Clay is really
6:09:21good for creating um the main 3D
6:09:24characters and acting out the stories
6:09:26physically before you implement them so
6:09:29starting from paper prototyping there is
6:09:32this popular paper from 1994 by Mark
6:09:35retic prototyping for tiny fingers which
6:09:38introduced this idea of using paper
6:09:40prototypes for testing so you can
6:09:42actually have multiple screens available
6:09:44you need to use a few people in
6:09:46different rols there's obviously a user
6:09:47trying out that experience there's a
6:09:49facilitator running through the test and
6:09:51then there is a person called computer
6:09:54that's a person a human that actually
6:09:56produces the screens by choosing from
6:09:58the different assets that people have
6:10:00designers have previously prototyped and
6:10:02then there's a notaker as well so how
6:10:04does this idea actually translate to
6:10:06arvr prototyping well we have worked on
6:10:09this and in some of my research um which
6:10:12I'm talking about here uh a project that
6:10:14we call 360 Proto we've taken these
6:10:16ideas further so now we have our user in
6:10:19VR or in AR we still have a facilitator
6:10:22that facilitator also has an additional
6:10:24task of actually capturing some of these
6:10:26mockups that we produced on paper we
6:10:29still have the computer person the
6:10:30computer is still producing the assets
6:10:33but we're using specific templates that
6:10:35I'll go in a little bit more depth here
6:10:37these 360 storyboarding templates I
6:10:39talked about earlier to really create
6:10:41360 paper prototypes and I'll show you a
6:10:43few examples throughout you can bring
6:10:45them to life and actually have test
6:10:48users experience paper prototypes
6:10:50essentially in VR or ar really quite
6:10:52powerful and I'm excited about this so
6:10:54I'm going to show you some of these
6:10:55techniques um throughout the
6:10:57segment if we think of paper prototypes
6:11:00you can distinguish just traditional
6:11:02paper prototypes which don't use any
6:11:03specific templates and 360 paper
6:11:06prototypes which make both the
6:11:08prototyping experience a little
6:11:09different and also the final user
6:11:11experience as we will see in some of the
6:11:13videos I'm going to show you
6:11:14now so if you stick just with paper
6:11:17prototypes what are they really good for
6:11:19well they bring storyboards to life and
6:11:21that's actually what we do as we take a
6:11:23storyboard and explain it to other
6:11:25designers using a paper prototype you
6:11:27can do the same with users you can
6:11:30actually demonstrate interactions in
6:11:32front of the user the user can perceive
6:11:35um this prototype with some imagination
6:11:37obviously to understand how these
6:11:39translate to arvr
6:11:41interactions what you can see in the
6:11:43image is how we use for example
6:11:45cardboard sticks or toothpicks in some
6:11:48later examples you will see how we put
6:11:49ploh figures on on on toothpicks to
6:11:51really animate the objects relatively
6:11:53seamlessly in front of the user so that
6:11:55they give they get the sense that this
6:11:57is actually an interactive prototype
6:11:59even though it's still on
6:12:00paper so here's an example where one
6:12:03user with a VR controller is pointing at
6:12:05forest animals and some of the designers
6:12:08around the user are actually animating
6:12:10these animals and respond to the user's
6:12:12interactions and it's quite an
6:12:14interactive storyboard in the sense that
6:12:16you could have new elements appear like
6:12:18in the second there will be a froggy
6:12:20triggered by the interaction uh with the
6:12:22VR controller and this really helps the
6:12:25user understand how this VR experience
6:12:29is going to unfold as they interact with
6:12:31different elements like these forest
6:12:33animals now now if you prototype that
6:12:36same experience with 360 U things look a
6:12:39little bit different you actually have
6:12:40to pay attention to the template and
6:12:42move elements like these animals uh
6:12:46accordingly so you can actually now lay
6:12:48out content in 360° around the users but
6:12:51pay attention that the grid actually
6:12:54means that and so you can't just move
6:12:57objects too quickly you really have to
6:12:58follow the grid lines requires a little
6:13:00bit of practice but it's actually a very
6:13:02powerful way of prototyping uh 360
6:13:04experience
6:13:05you can demonstrate interactions along
6:13:07these grid lines as I said and that will
6:13:09actually make for example something fly
6:13:11around the user and um you can also use
6:13:15objects of different size to simulate
6:13:17depth remember 360 photo stories and 360
6:13:21photos in as I mentioned earlier don't
6:13:23actually have depth it's not really 3D
6:13:25and so one way to simulate depth is
6:13:27actually using objects of different size
6:13:30the way this works is um a designer is
6:13:33prototyping uh bird flying down and we
6:13:36are now actually this owl has now landed
6:13:39on the tree and we've just replaced
6:13:41essentially the model oura is coming in
6:13:44and uh this was an experience of
6:13:46prototype with students uh we are not
6:13:48particularly careful with paying
6:13:50attention to the grid lines at this
6:13:51stage we were still getting familiar
6:13:52with the template but um I'll show you
6:13:55um how we can actually take this
6:13:57storyboard to the next level because the
6:13:59idea is that you can actually capture
6:14:01this and uh this video and actually play
6:14:05it back in 360 remember it follows the
6:14:08360 format and we can actually then
6:14:10visualize this paper prototype for VR
6:14:14users so this is what I'm going to show
6:14:16here quickly we can compose a scene uh
6:14:19putting the 360 grid template on top and
6:14:22then adding the elements that we
6:14:24actually want users to see now you're
6:14:26spacing them out you can cut them out
6:14:27you could sketch them you could draw
6:14:29them as you did with the storyboards but
6:14:31I think for paper prototyping I would
6:14:32usually work with paper cutouts you can
6:14:35animate uh more easily you could re swap
6:14:37out models it's actually really powerful
6:14:40and then you can compose a scene like
6:14:42this way okay and because this is
6:14:45actually still following the 360
6:14:47template what we can do now is we can
6:14:48actually take a photo close up of that
6:14:50template wrap the photo in cardboard and
6:14:53then experience that in VR I'm going to
6:14:56illustrate this quickly okay this is a
6:14:58closeup of the paper prototype we take
6:15:01the picture we go into VR and we see it
6:15:04with steros cop views and I'll show you
6:15:06the same in a video so this is me taking
6:15:09a picture of a previously prototyped
6:15:11experience I click the button the phone
6:15:14is already inside cardboard and I can
6:15:16start previewing in some sense prev
6:15:18visualizing the 3D scene you can see the
6:15:21grid lines uh we could remove this layer
6:15:23of grid lines if you use transparency or
6:15:25other kinds of means but you have a very
6:15:27quick way of visualizing a paper
6:15:29prototype now you can actually have a
6:15:32facilitator or a computer um a human
6:15:35computer make adjustments to this uh
6:15:38scene composition and now as we can see
6:15:41the owl has moved a lot closer if you
6:15:43combine this with live streaming you can
6:15:45actually simulate interactive 360
6:15:48storyboards and this is something you
6:15:49can read about later where I share some
6:15:51of my work on 360 Proto what is a little
6:15:54bit tricky is actually creating AR
6:15:56previews this way um um and what I'm
6:15:59going to show you now is an idea I've
6:16:00developed in 360 burough the research
6:16:03project um we're going to take the 360
6:16:06template we're going to map it we take a
6:16:08photo of it we're going to map it into a
6:16:103D sphere and our goal is to actually
6:16:12animate a butterfly around the user or
6:16:15at least have a butterfly stuck in the
6:16:18air and uh so that the user can actually
6:16:21walk around the AR user can walk around
6:16:23and actually experience that butterfly
6:16:25what I don't do this in this example is
6:16:27actually moving the butterfly but you
6:16:28still see a really cool effect here so
6:16:31how do we prototype this well we can use
6:16:33additional AR Tech like markers like a
6:16:36physical marker here we can use a phone
6:16:39to capture that and live stream that and
6:16:43we can then actually take that picture
6:16:45remove the background white becomes
6:16:47green and then actually becomes
6:16:49transparent we extract the butterfly and
6:16:51we show it at the very location the way
6:16:54we do this is we're using the 360 grid
6:16:55again to keep uh attention to where the
6:16:59object is around the user and we're
6:17:02replacing that marker with actually our
6:17:05virtual overlay which in this case is
6:17:07the
6:17:08butterfly okay let's experience this so
6:17:10This is actually this is actually me
6:17:13working with my student Katie and we are
6:17:16actually now uh having this butterfly
6:17:19kind of like move in front of our camera
6:17:22and be live streaming this uh to the
6:17:24other phone that you see in the
6:17:25background and what I'm going to show
6:17:27you in the next step is actually how I'm
6:17:31going to take up the phone and then we
6:17:35have the butterfly there and because
6:17:36we're using AR techniques with the
6:17:38camera the smartphone camera needs to be
6:17:40exposed for this to work we can actually
6:17:42walk around this butterfly at this stage
6:17:44it is uh still um again if you were live
6:17:47streaming this um to the VR or ar user
6:17:52we would have a really cool experience
6:17:53that way so this was just the first part
6:17:56of physical prototyping but I wanted to
6:17:58illustrate to you how you can actually
6:18:00take paper and make it digital a lot of
6:18:03this is obviously described in more
6:18:04detail in in some of my research I and
6:18:07others have worked on this and I hope um
6:18:09that we will have better tool support in
6:18:11the future and the meantime I will give
6:18:14you access to some of the tools that
6:18:15I've created in my
6:18:29research physical prototyping 2 this is
6:18:32the second part of my physical
6:18:34prototyping lecture because physical
6:18:35prototyping is so important to me I made
6:18:37two lectures out of it and in the
6:18:39previous segments I've talked about
6:18:41paper prototyping and 360 paper
6:18:43prototyping and now there are two things
6:18:45that I really want to talk about a
6:18:46little bit more which is working in
6:18:48physical space and with physical
6:18:50modeling tools so we're going to talk
6:18:52about the diarama which is good for
6:18:54thinking through proxemics really the
6:18:56spatial relationships between all the
6:18:58elements of our a experience and then if
6:19:00you're not good in 3D modeling uh I
6:19:03don't blame you I'm not good at it
6:19:04either but I can actually do it with
6:19:05Playdoh so I'm going to talk to you a
6:19:07little bit about 3D modeling with Clay
6:19:09which I think is very good for creating
6:19:10the main 3D
6:19:12characters so the diarama is essentially
6:19:15a 3D physical often miniature version of
6:19:19a set like a 3D scene or an arvr scene
6:19:22that we run a prototype um it actually
6:19:25allows you to enact that 3D scene in the
6:19:28physical world you can use multiple
6:19:30designers to help each other and then
6:19:32give that experience to a user and
6:19:35as before you can actually demonstrate
6:19:36interactions to users in this case what
6:19:38often happens when I do this with my
6:19:40students in class really the students
6:19:43become actors they really play out the
6:19:453D characters in response to user
6:19:47interactions so it's really cool to see
6:19:49and side effect and actually very useful
6:19:52for learning about um this prototype as
6:19:55we move it digitally you can actually
6:19:57gauge the spatial requirements and
6:19:59understand the relationships between the
6:20:00different physical objects that you're
6:20:02using to prototype that arvr experienc
6:20:06I'm going to show you an example from
6:20:08something we've prototyped in my lab
6:20:10it's again this forest scene that we
6:20:12like so there we have the owl fly in and
6:20:16then somebody interacting and now this
6:20:19alll was replaced by different object um
6:20:22now this looks a little weird because uh
6:20:25we chose this upper angle and we have
6:20:26played around with different types of
6:20:28stages but the main takeaway here is
6:20:31that it's actually a very good physical
6:20:33representation all the pieces you had to
6:20:36fill in previously when thinking just on
6:20:38Plain paper in 2D or thinking around 360
6:20:41and using our tempate from before
6:20:44actually working in physical space makes
6:20:46makes things so much easier so at least
6:20:49that's what my students tell me and
6:20:50that's what I also personally feel when
6:20:51I translate some of these paper
6:20:54prototypes into
6:20:57diaramas obviously I would like to share
6:20:59a lot of the work I've done with my
6:21:01students in my classes and in my
6:21:02research so here is a quick selection we
6:21:04have for example prototyped VR gardening
6:21:07experiences using diaramas or we have
6:21:10talked about this public speaking
6:21:12experience which actually creates a lot
6:21:14of anxiety and you can prototype out a
6:21:16TED talk this way if you want and some
6:21:18of my students have also prototyped an
6:21:20immersive theater uh where you can
6:21:23actually um here you see a 3D version of
6:21:25the stage and later it'll be in VR and
6:21:28so diaramas are really powerful to
6:21:30actually prototype out these techniques
6:21:32often in a matter of 45 minutes you can
6:21:34do this in class with designers of any
6:21:37kind of background and you don't
6:21:39actually need a lot of technical skills
6:21:40and any digital tools which I think um
6:21:43is actually the cheapest and fastest 3D
6:21:46tool that is out there physical space
6:21:49and a diarama might be a very good way
6:21:50to do
6:21:52this so the last technique I want to add
6:21:55to our portfolio and you've seen this to
6:21:57some extent in previous videos is we
6:21:59often use 3D modeling with Clay here I
6:22:02show a few of the examples created by my
6:22:04my students coming back to this idea of
6:22:06prototyping an AR furniture placement
6:22:08tool so the way we often use 3D modeling
6:22:12with Clay is we actually focus on the
6:22:14main characters of our scene and then um
6:22:17it be these storyboards we used
6:22:20previously are now becoming physical and
6:22:22you can actually capture nicely what
6:22:24these 3D characters do they actually
6:22:25become they come to life and they
6:22:28actually are in 3D they can be moved in
6:22:303D they can be rotated and you can
6:22:33actually get a sense of the complexity
6:22:34of the 3D models you need to create
6:22:36later on and you also can demonstrate
6:22:39interactions again in real 3D it really
6:22:42makes a lot of things easier however
6:22:44sometimes with some of the interactions
6:22:46you really need to coordinate better
6:22:48between designers and users so this
6:22:50requires a little bit of practice as we
6:22:52can see in one of the videos I'll show
6:22:54you in a second so here's an example
6:22:57where we are uh trying to coordinate
6:23:00between user and designer the user is
6:23:01performing a rotation gesture and and
6:23:04the designer in this case the computer
6:23:07the human computer is moving the chair
6:23:09accordingly so the students have just
6:23:11prototyped this experience uh quickly
6:23:14and got a sense of the interactions they
6:23:16want to support in their AR furniture
6:23:18placement app and this is a really cool
6:23:22uh way of getting a sense for the
6:23:24physical space and actually creating 3D
6:23:26assets that way as well great so I've
6:23:29brought a few physical prototypes here
6:23:32and uh what I want to illustrate is how
6:23:34you can can actually prototype
6:23:35interesting concepts for both virtual
6:23:38reality and augmented reality um using
6:23:40just physical prototypes so I've
6:23:42prepared three scenes here and there are
6:23:45different ideas behind each of these
6:23:46scenes I'm going to start with this one
6:23:48so this one was actually a project that
6:23:49happened in one of my classes the
6:23:51students were interested in using
6:23:52virtual reality to prototype um
6:23:56standardized test scenarios so lot of
6:23:58students actually are afraid of
6:24:00standardized tests usually really big
6:24:02audiences really big rooms and so what
6:24:04this prototype allowed them to Think
6:24:06Through were like the basic interactions
6:24:08who are the characters and what would be
6:24:10interesting perspectives so you could
6:24:12the experience if you are in the front
6:24:14where you can always see the door might
6:24:16be very different uh to like sitting
6:24:19behind a bunch of other people in this
6:24:21kind of like standardized test scenario
6:24:24might create more anxiety and so VR is a
6:24:26very powerful tool for building empathy
6:24:28and so this is actually was was one of
6:24:30my uh favorite projects for from some of
6:24:33my students in class this was also a
6:24:35really really cool idea here the idea
6:24:37was to bring language and culture closer
6:24:40to the audience in fact this was about
6:24:43Chinese food and all kinds of really
6:24:45good-looking food in fact this looks so
6:24:47good I would really like to eat it and
6:24:49but it is actually made of clay and so
6:24:51you shouldn't and so there were
6:24:53additional elements that I don't show
6:24:54here right now I just focus on Playdoh
6:24:57and the physical aspects and but there
6:24:58was paper that was actually Chinese
6:25:00language around it the idea was to
6:25:02really prototype an experience
6:25:04so that Learners can learn about
6:25:06different language and culture now these
6:25:09were two examples of prototypes that
6:25:11allow us that are new ideas in some
6:25:14sense that allow us to Think Through
6:25:16interactions and these are some of the
6:25:18things we practice in our physical
6:25:19prototyping activities now this is a
6:25:22last scene that obviously if you are
6:25:24familiar with some of the movies out
6:25:26there this should remind you of The Lion
6:25:28King now what we've also tried in a lot
6:25:30of our research and also in some of our
6:25:33student projects um we have actually
6:25:35taken existing scenes and recreated them
6:25:37um here you see Rafiki and we actually
6:25:39had to do two shots of Rafiki and you
6:25:41may see this in other portions and other
6:25:43segments of this Muk specialization
6:25:45where we actually animated raiki
6:25:47especially this one difficult scene
6:25:49where he was like moving the hands apart
6:25:52and breaking this is a really a really
6:25:55nice element in the teaser and this
6:25:57allowed us to think about character
6:25:58movement and complexity of 3D models and
6:26:02that's what you learn from doing
6:26:04everything physically when you start
6:26:06analyzing these physical prototypes I
6:26:08believe you get a lot of knowledge in
6:26:09terms of requirements uh what what kinds
6:26:12of user requirements do we have what
6:26:14kind of display requirements tracking
6:26:15requirements what does this mean for our
6:26:173D models how complex is this stuff can
6:26:20we download existing ones do we have to
6:26:22create our own ones and so doing this
6:26:24with Playdoh in the matter of an hour
6:26:26you can do this in a classroom setting
6:26:28you can have students think through this
6:26:30um very quickly and I'm I'm a big
6:26:32proponent of physical prototyping so in
6:26:34the last two segments physical
6:26:36prototyping 1 and physical prototyping 2
6:26:38I've introduced you to a variety of
6:26:40physical prototyping methods for example
6:26:42just using paper then adding the 360
6:26:45grid onto paper and then using clay and
6:26:47the diaramas and what is most
6:26:50fascinating to me is that these physical
6:26:53prototyping methods are largely arv
6:26:55agnostic unlike digital tools which are
6:26:57often just really good at AR or VR if at
6:27:00all and physical prototyping is really
6:27:02fast and cheap and it's really allowing
6:27:03you quickly to iterate on designs
6:27:06however keep in mind that when you
6:27:08prototype physically for VR it's really
6:27:11about the immersive experience and this
6:27:13feeling of creating presence so you
6:27:15should really pay attention in your
6:27:17prototypes to creating this virtual
6:27:19environment for example using the 360
6:27:22grid because this is the best method
6:27:24that allows you to think spacially
6:27:27around the user however for AR it's
6:27:30really about augmenting experiences and
6:27:33context so context here is really
6:27:35referring to the physical environment
6:27:37that the user is in so your AR device
6:27:40would read this environment and then s
6:27:42send signals to your interface so what
6:27:45you need to pay attention to is when you
6:27:46prototype AR experiences the tiny little
6:27:49things in the physical environment
6:27:50around the user that could trigger
6:27:52certain aspects of your experience
6:28:07in this video I'm going to introduce you
6:28:09to the XR storyboard exercise is part of
6:28:12our XR design project and uh so we are
6:28:16now in the storyboarding uh and
6:28:19wireframing Stage so that means we are
6:28:22brainstorming essentially we want to
6:28:23focus on new features um as we think
6:28:27about uh the Expeditions like XR app uh
6:28:31that you have chosen and so part of the
6:28:34exercise is really understand the
6:28:36existing flow without thinking about new
6:28:37features and then also introducing
6:28:40developing one new feature so that's all
6:28:43I'm asking and it's important uh don't
6:28:45try to come up with your own complete
6:28:47app that's really hard for for other
6:28:50Learners than to provide feedback and uh
6:28:53help you improve uh your storyboards and
6:28:56wireframes so it would be good to ground
6:28:59it in something that is not too
6:29:03complicated we don't need forever to
6:29:05understand what the problem is you're
6:29:06trying to solve now we want to focus on
6:29:09the well the the things that matter when
6:29:11we do storyboarding and uh wireframing
6:29:14so really focusing on the key design
6:29:17ingredients if you will so here is my
6:29:21exercise cheat sheet this is what I have
6:29:24in mind so the task will be to create a
6:29:27storyboard for in Expeditions like XR
6:29:29app so one storyboard one type of
6:29:32storyboard is fine you should create
6:29:34probably a few different uh kinds of
6:29:37story boards ideally but let's say you
6:29:39do one storyboard you should start with
6:29:41two to three high level sketches so
6:29:43capturing the flow of the app really
6:29:45want to understand how does this
6:29:47actually work how do you transition
6:29:48between states if not screens and uh how
6:29:52does it work and then you actually start
6:29:54sketching out you can just sketch out
6:29:56some of the existing screens as well or
6:29:58scenes or whatever we should call it and
6:30:01you should add more detail and
6:30:03distinguish obviously in your sketches
6:30:05the physical and the virtual objects so
6:30:07for example using color coding H you
6:30:10should annotate your sketches with
6:30:11arrows and labels anything that helps us
6:30:13understand what you've been sketching
6:30:14usually a short label is very helpful uh
6:30:17for any kind of key elements in there
6:30:19this really drives the attention now um
6:30:23my sketches
6:30:25look ugly most of the time but that's
6:30:28just because I only sketch what I want
6:30:32you to focus on that's not true um yes
6:30:36so the uh we don't judge each other
6:30:39sketches uh in terms of Aesthetics uh
6:30:42the whole purpose of a sketch is to
6:30:43convey an idea and so you want to only
6:30:46sketch what is necessary only the
6:30:48resolution that is required uh so that
6:30:51you can uh you know get the other
6:30:53person's attention on your idea and help
6:30:56you refine it and improve it and so it
6:30:59doesn't have to look beautiful in fact
6:31:01it's great that my stuff doesn't look
6:31:03good then you don't feel intimidated by
6:31:05it and I think that is H nice would it
6:31:08be worse if I were like a really good
6:31:10artist anyway so you annotate your
6:31:12sketches I just covered that and you
6:31:15should actually then create two to three
6:31:16alternative wireframes of your new idea
6:31:19with content I would like to see what
6:31:21we're talking about both in content but
6:31:23I really care also about the flow and
6:31:26the interactions how do I get to these
6:31:28new uh screens and scenes uh that you
6:31:30have prototyped and uh optionally and by
6:31:34I think this would be a really good task
6:31:35and I say optional because well to
6:31:38manage your workload but also normally
6:31:40in a project you wouldn't actually try
6:31:42out all kinds of different uh types of
6:31:44storyboards but I would like you to try
6:31:46out a 360 or a 3D storyboard um so the
6:31:49expected results of this activity and
6:31:51exercise are that you have a better
6:31:54understanding of the overall flow of the
6:31:55app and you also have more concrete
6:31:58ideas of how to change and improve the
6:31:59design including a new feature that you
6:32:02think could be cool and you have a good
6:32:04sense of what your app may feel like in
6:32:05arvr especially if you use a 360 or 3D
6:32:09storyboarding approach so again we are
6:32:12going to stick with a Expeditions like
6:32:15AR or VR app so an XR app um can also do
6:32:19uh 360 and it should work on smartphones
6:32:24uh like this one does and um that way
6:32:26other people can also try it out so
6:32:29here's the one that I was trying out
6:32:31this is the solar system you've seen me
6:32:33uh do it you've seen me critique it and
6:32:36um this is I'm going to show you a few a
6:32:39few videos of me brainstorming uh and
6:32:42storyboarding around it and also then
6:32:44creating a few wireframes so so here you
6:32:47see uh screenshots from my design
6:32:51critique as I was playing around with
6:32:52the app and that is really the input
6:32:54here the function is to create a
6:32:57storyboard and the input parameter is
6:32:59your uh existing critique and so you
6:33:03experience of trying out that existing
6:33:06app and so I've uh covered all the kinds
6:33:09of uh screens and states of the app that
6:33:12I could reach and that were in the scope
6:33:15of my critique and here I'll show you a
6:33:19this is from the lecture so we have a
6:33:21few different kinds of storyboards they
6:33:24vary both in terms of the Fidelity as I
6:33:26said so lower to higher Fidelity and
6:33:29easy to hard and I'll talk about this or
6:33:31talked about this in detail in the
6:33:33lecture and so I'm going to show you
6:33:35examples of each of these types of
6:33:37storyboards first a traditional
6:33:40storyboard
6:33:41so here you see a time lapse of me doing
6:33:46the exercise and um I was you know I had
6:33:50already sketched out the basic flow so
6:33:52this was just the flow and then the
6:33:54start and then I was like developing my
6:33:56own new idea and while I was actually
6:33:59developing my own new idea I was also
6:34:01playing around with the existing app
6:34:03from time to time just to to to get to
6:34:06uh really to flesh out some of my ideas
6:34:09get a better understanding and some of
6:34:11the things I really wanted to do was
6:34:13actually uh see multiple planets in
6:34:16relationship to each other and then I
6:34:18had even more ideas of doing different
6:34:21kinds of day and night and wouldn't it
6:34:23be cool if Earth how how would earth
6:34:26look at night and uh really was
6:34:28interested in the relationship of the
6:34:30Moon around Earth and how it fits into
6:34:32this picture and how far it
6:34:34Earth away from the Sun anyway and then
6:34:36other planets I didn't even care so much
6:34:38about I was just focused on those three
6:34:40planets in the original app they had
6:34:42more uh planets but they didn't actually
6:34:45do much about Earth and Moon and Sun
6:34:47like no relationships and so I was
6:34:49really figuring out what could I do to
6:34:51improve upon uh some of the things that
6:34:55I think were missing from the app from
6:34:56the existing
6:34:58Expedition and uh I understand that this
6:35:01Google Expeditions framework might be a
6:35:03little Limited so all these Expeditions
6:35:05work in a very similar fashion and so
6:35:08maybe there you could see hey that was a
6:35:11break uh me thinking about what I could
6:35:14do and um then I had this idea of doing
6:35:18a different night times and Day times
6:35:21and seeing really the zones like on an
6:35:23airplane uh when they show you this
6:35:25world map and show you where this day
6:35:27and where it is night right now I
6:35:29thought that could be cool and um yeah
6:35:33so then usually I do kind of like a
6:35:35walkthr and illustrate some of the ideas
6:35:37I had in mind and this is also obviously
6:35:40in not 500 times speed would be a good
6:35:44way for you to share your
6:35:48results and I have each of these steps
6:35:51recorded so you can watch me do a
6:35:53storyboard I'll share that video with
6:35:56you you can watch me uh actually review
6:36:00my own storyboard that gives you some
6:36:02ideas of maybe what to pay attention to
6:36:05um and also what you can learn from
6:36:07these storyboards so the reading between
6:36:10the lines is really important when it
6:36:11comes to these activities and exercises
6:36:14so I'll I'll share those uh steps and
6:36:18videos with you so that may help you
6:36:20guide through each of the steps even
6:36:23more and uh then I was doing a 360
6:36:28storyboard so now we're down here right
6:36:30a 360 storyboard I think is harder cuz
6:36:33you're using a template that you may
6:36:34have to well spend some time with and
6:36:37actually understand and get a feel for
6:36:39it and then I would say it on the
6:36:41Fidelity scale it actually moves you
6:36:44more like here rather than here or here
6:36:48I do think that Fidelity is actually
6:36:49really quite High it is not as high as a
6:36:523D storyboard but I'll show you and I
6:36:54actually share an app with you so you
6:36:56can actually take pictures of your
6:36:58storyboards and then preview them in
6:37:00arvr and I think that helps you cross
6:37:03this line and brings you into this
6:37:04quadrant the one that we want to be in
6:37:07ideally you want to be here right super
6:37:09high fidelity and super easy and uh well
6:37:12I can't claim that for the 360
6:37:14storyboarding approach obviously there
6:37:16is my my research on 360 Proto and we've
6:37:18really worked quite a bit on this but um
6:37:21I think it can get you here maybe maybe
6:37:25but not not much further than that so
6:37:29this is the template and uh this is what
6:37:32you see right in front of you this is
6:37:3390° to your left so over there 90° to
6:37:36your right over there and um this is
6:37:40right above you this is oops down there
6:37:42would be right below you and uh and then
6:37:46it actually also like because it
6:37:47stitches on the sides and the top it
6:37:49really wraps around your head like
6:37:52that's how the 360 template works right
6:37:53think of 360 photos this is AC
6:37:55rectangular format so I was talking
6:37:57about this in more depth and Detail in
6:37:59the lecture but um you know sketching
6:38:02like a a rectangle you would actually
6:38:04have to follow these grid lines okay you
6:38:07follow these grid lines you go down here
6:38:09and you follow this grid line back and
6:38:11you just sketched a really big rectangle
6:38:14it didn't look like a rect angle but it
6:38:15will look like a rectangle when you then
6:38:18uh do this
6:38:20transformation and this is a projection
6:38:22from accurate angular to spherical and
6:38:25uh I was just talking about this so
6:38:27these are the things that are right in
6:38:28front of you would be here and left and
6:38:32right uh this Blue Cross keep that in
6:38:34mind we even entered all the degrees and
6:38:36there are templates and you see me use
6:38:39some of these templates to um help me
6:38:42check for the field of view and the
6:38:44range of motion this is something that I
6:38:45would do in VR and you see the templates
6:38:49here so they are like lying up there the
6:38:51transparencies that we printed out I'll
6:38:53share those with you as well I think
6:38:54they're a little bit more advanced and
6:38:56um not even necessary but um it's still
6:39:00kind of cool
6:39:04and so here you see me sketch it out and
6:39:07um I'm doing a storyboard this is the
6:39:09more interesting one I didn't really
6:39:11play the other one and I was playing
6:39:14around and preview that's really
6:39:15important and then I was checking for
6:39:17the field of view and the range of
6:39:18motion really how much would the user
6:39:20need to move initially in order to keep
6:39:23view things in View and this assumes uh
6:39:26a standard VR headset now since I was
6:39:30actually more thinking about AR um this
6:39:32was a little
6:39:34bit or more like an often a
6:39:35demonstration just to show you that it
6:39:37exists and um this app is really quite
6:39:40cool that I created so I'm going to
6:39:42share that with you it should work on
6:39:43your phone hopefully um it is research
6:39:46so um don't don't hate me if it doesn't
6:39:50work on your phone um it works well on
6:39:53my phone and on Android phones on a
6:39:55variety of phones and uh we can probably
6:39:58configure it for your phone too and then
6:40:01in between I sketched out just using the
6:40:03the normal uh well traditional paper and
6:40:05now I'm moving this sketch to the 360
6:40:08template and um I should
6:40:11share uh I could share that entire video
6:40:13with you I do think that I was sketching
6:40:15for like 20 or 30 minutes it's probably
6:40:18a bit long um I was thinking we just
6:40:21look at the time lapse and uh well and
6:40:25then I was using color so that color is
6:40:27actually quite important because as I
6:40:28was talking about in the lecture you
6:40:30should really Mark the virtual content
6:40:33and tell me what's virtual and tell me
6:40:35what's physical and here I used red as
6:40:38annotations so red wouldn't be displayed
6:40:41blue is the virtual content and black is
6:40:43just the reality uh that's the notation
6:40:45that I have established you for myself
6:40:47like you don't have to use it exactly
6:40:48like that but or do it exactly like that
6:40:51but you have to be clear how you were
6:40:52thinking about
6:40:54it here is the
6:40:56result um I'll share that video with you
6:40:59um definitely because rather than
6:41:02watching me sketch forever this is the
6:41:04one where I'll talk you through some of
6:41:06the results and that will give you some
6:41:07inspiration so we're going to share that
6:41:09360 storyboard uh video and you can
6:41:12watch it in full length was playing
6:41:16around with a template and really
6:41:17getting a sense for how things work and
6:41:21uh then I was applying that template so
6:41:24here for example I show um how the
6:41:26template really works and how I applied
6:41:29it to paper and how I was then sketching
6:41:32out the planets and and uh then also
6:41:34previewing the planets in uh VR I also
6:41:38have a video for you where I'll actually
6:41:40preview the story boards that I have
6:41:42sketched and um so you can see what this
6:41:46might look like I hope uh that you will
6:41:48be able to try out this app and it runs
6:41:51on your phone um this is not a product
6:41:54this is a research prototype uh coming
6:41:57out of uh some research from my lab it's
6:41:59a simplified version not as advanced as
6:42:01the one that we actually had in the
6:42:02paper and 360 Proto uh where you could
6:42:05live stream and use multiple phones and
6:42:07all that kind of stuff but even this
6:42:09simplified one is uh already good enough
6:42:11for us to get an overview and or a
6:42:14preview and actually
6:42:16transform paper into a VR interface
6:42:20especially when you then combine it with
6:42:22cardboard um that is then um really cool
6:42:26because you really get a VR uh preview
6:42:28of VR experience and here is that
6:42:31example where I then actually put put it
6:42:33in cardboard and uh then put obviously
6:42:37the actual cardboard um use that with a
6:42:41device so Okay so we've seen the
6:42:43storyboard we've seen a 360 storyboard I
6:42:45don't do a 360 photo story but again the
6:42:47idea would be I just like do it in the
6:42:49physical world and then I take 360
6:42:51photos of it or any kind of standard
6:42:54Photo Story you could do as well but I
6:42:55think you would learn more from a 360
6:42:56Photo Story and I describe that in the
6:42:58lecture what I've done for my own
6:43:01exercise I have done the 3D story board
6:43:03so I'm going to focus on that so I have
6:43:05a 11 minute video of me actually going
6:43:09through that uh storyboarding activity
6:43:12in tilt brush and really I was using
6:43:15tilt brush in a way that is it's
6:43:17probably not really being used for as a
6:43:19storyboarding tool um keep in mind that
6:43:21tiltbrush is not a tool that is for free
6:43:23so um there are other tools out there
6:43:26I've played around with a few you'll see
6:43:27me use other few other tools later quill
6:43:30for example is one that you could use on
6:43:32the arus
6:43:34and there are a number of sketching
6:43:36tools out there and then it is on you to
6:43:39use them as a storyboarding tool the way
6:43:41I've done it here is like sketch out
6:43:43different stages or scenes of the app
6:43:45and I was thinking I was doing it in VR
6:43:47but I'm actually sketching an AR
6:43:49experience and I'm sketching a little
6:43:51crazy here this was really supposed to
6:43:52be a glowing sun and uh First Step was
6:43:55like place the
6:43:58app the Second Step was uh to then see
6:44:02the sun
6:44:03and then I also wanted to really
6:44:06understand the transition between these
6:44:08different steps and later in the video I
6:44:10then review some of my work and then I
6:44:12use it like a user and think it through
6:44:16and Think Loud and Mark like the
6:44:17interactions that users might do and
6:44:20then uh I was picky with the color for
6:44:23some reason but that's just being me um
6:44:25anyway this was supposed to be blue I
6:44:27guess and I was looking for the right
6:44:28blue but come on Michael focus on the
6:44:30task and the task at hand was to to
6:44:33really illustrate to you how this might
6:44:34work and then you see I put a little
6:44:37Moon next to Earth there and then put
6:44:38both around the Sun and I was sketching
6:44:41how they would go around the Sun and uh
6:44:43I'm quite sure you would be learning
6:44:44quite a bit from doing a 3D storyboard
6:44:48again it doesn't have to be tiltbrush uh
6:44:50use something that is for free and
6:44:52available to you um or you make that
6:44:55investment um you could also go pro do
6:44:58gravity sketch and there are some other
6:45:00tools out there as well um anyway we can
6:45:04discuss uh good tools the 3D
6:45:07storyboarding is probably one that you
6:45:09would find very exciting personally I
6:45:12think I get a lot out of the 360
6:45:14storyboarding just like because I find
6:45:16that template so cool but also
6:45:18challenging and um but every time I do
6:45:20it I learn something both about me and
6:45:23my limited sketching uh skills and also
6:45:27then obviously about the project that
6:45:29I'm working on and that is actually what
6:45:31matters so this is going to be a fun
6:45:35exercise uh you can blow it up a little
6:45:37bit by trying out these different kinds
6:45:39of storyboards or keep it simple and
6:45:40just work on paper but work through the
6:45:43flow and really figure out your new
6:45:45feature
6:46:03hey there fellow Honest track Learners
6:46:06I'm excited to welcome you to this next
6:46:09step you know on our path to creating
6:46:12this really cool XR project physical
6:46:14prototyping I don't know in every course
6:46:17I teach I always get so excited when it
6:46:19comes to the physical prototyping stage
6:46:21I mean we were already quite physical
6:46:24when we were doing the storyboarding uh
6:46:26previously but now we're really doing
6:46:29physical prototyping so that includes
6:46:31paper prototyping but also goes beyond
6:46:34paper and this is what I want you to
6:46:36learn and want you to practice in
6:46:37exercise so I'll walk you through the
6:46:40steps of this exercise and I'll come
6:46:42back at the end with a few ideas for how
6:46:45to make use of this activity so let me
6:46:48walk you through the exercise and I'll
6:46:50also share a few insights and tips along
6:46:52the way since keep in mind I've actually
6:46:54been doing this myself uh in preparation
6:46:57for the moo and so I spend quite some
6:47:00time prototyping and I'll be happy to
6:47:03share a lot of these uh videos where it
6:47:06makes sense with you as well so this is
6:47:08our task your task is to create a
6:47:10physical prototype of an Expeditions
6:47:13like XR app same rules as before I say
6:47:16Expeditions like it doesn't have to be
6:47:18Google Expeditions I say XR it can be VR
6:47:20it can be ar it doesn't have to be both
6:47:23it can also be just 360 the key point of
6:47:26this activity is really to create a
6:47:27paper prototype of one of the key
6:47:30interactions right I mean there's a lot
6:47:32of stuff you you probably want a paper
6:47:33prototype and that includes the content
6:47:35and all that and I understand that but
6:47:37we really want to focus our attention on
6:47:39the interactions we really need to make
6:47:40more progress when it comes to that but
6:47:43we also going to do content so we model
6:47:45two to three main 3D characters with
6:47:47Clay this will be fun and uh we want to
6:47:50create a diarama with the 3D characters
6:47:53to define the story now diarama doesn't
6:47:56have to be like the one that I'm going
6:47:57to show you it can be just like in 3D on
6:48:01it can still be on paper but then think
6:48:02pop out book so it has to come out uh
6:48:05really into the physical world we don't
6:48:07just want to paper prototype paper
6:48:08prototype we actually want to be more
6:48:10physical than that and so bring some
6:48:12cardboard and uh yeah do that record a
6:48:16video and narrate the interactions that
6:48:18I think is a really really really
6:48:20important part that brings your physical
6:48:22prototype to life and also allows you to
6:48:24share it with other Learners optionally
6:48:27you can create a 360 paper prototype to
6:48:29define the context um so 360 paper
6:48:32prototype would mean to use my template
6:48:34and follow uh some of the instructions
6:48:36this is makes a lot of sense especially
6:48:38when you've done a 360 storyboard like
6:48:40to keep refining that storyboard and
6:48:42turn it into a 360 pepp prototype really
6:48:44wireframing it out and thinking about
6:48:47some of the Transitions and interactions
6:48:48and then actually start really playing
6:48:49with it so moving the characters around
6:48:51on the grid that is really cool but as
6:48:54before you don't have to do each and
6:48:56every type of uh physical prototype uh
6:48:58you can just do a paper prototype
6:49:00although I would like you to really play
6:49:02around with Play-Doh and create a
6:49:04diarama the expected results from this
6:49:07activity um because these different
6:49:09types of prototypes also increase
6:49:11Fidelity I mean you're still kind of low
6:49:12to medium Fidelity but um you can
6:49:14increase Fidelity just by changing the
6:49:17uh tools that they using to create the
6:49:19XA prototype so you will get a better
6:49:21sense of the problem the context and the
6:49:23story and that's really really important
6:49:25uh these are the key ingredients that I
6:49:26introduced you to and this really
6:49:28matters for XR actually for any kind of
6:49:30user interface but h story is
6:49:33particularly important uh in XR we want
6:49:36to identify the key elements so the
6:49:38environment the 3D characters and this
6:49:40also helps you drive requirements in
6:49:42terms of assets for digital prototyping
6:49:45and development later on we really want
6:49:47to learn about the interactions these
6:49:50physical prototypes will allow you to
6:49:51play around with the materials try out
6:49:54different compositions different scene
6:49:55compositions different transitions
6:49:57different animations just by moving
6:49:59things around all the things you would
6:50:00have to program digitally you can now in
6:50:02the physical world and so that's so
6:50:04exciting get a feel for technical
6:50:07requirements and then also digital
6:50:08prototyping needs I would say these are
6:50:10the expected results this is why we do
6:50:13physical prototypes and when you share
6:50:15your physical prototype video that's
6:50:17really cool that would be a major
6:50:19accomplishment what would be even better
6:50:21is you reflect upon that experience just
6:50:23you going through it in your mind again
6:50:26reflecting I usually have that activity
6:50:28with my students in the residential
6:50:30courses we do the physical prototypes
6:50:32and and then we write half a page just
6:50:34thinking it through like a little diary
6:50:36entry thinking it through what did I
6:50:37actually learn and we critique both the
6:50:40method like what was limited what was
6:50:41hard to do I was doing VR but I was just
6:50:44using paper and I was using a 360 but
6:50:47then how does 360 translate to AR and
6:50:50and that's something um that is
6:50:52something that I've been thinking a lot
6:50:53about as part of my research and uh
6:50:56would be cool if you raise similar
6:50:57questions and maybe we can engage in
6:50:58some interesting discussions around it
6:51:00don't hate the 360 template it will just
6:51:03require some time um there are easier
6:51:05tools out there you don't have to jump
6:51:06right in uh you can also do an immersive
6:51:10prototype so keep in mind this is
6:51:12supposed to be physical so don't jump
6:51:14into the digital tool yet that is kind
6:51:17of not allowed and ideally you would uh
6:51:20do with physical materials I want you to
6:51:24get that experience I think it's it's a
6:51:26pretty important one and it allows you
6:51:29to think without all the technical
6:51:30constraints that you usually have to
6:51:34well phase and solve and work around
6:51:36when you do digital prototyping or later
6:51:39on development so that's why physical
6:51:41prototyping is so
6:51:43important so as before we were I'm doing
6:51:47the solar system so I'll share a few of
6:51:49the things I had obviously the knowledge
6:51:51from the design critique I also had my
6:51:53storyboard I did a standard storyboard I
6:51:55was playing around with the app I was
6:51:57going through the flow and I was
6:51:58developing my own new idea on storyboard
6:52:01I was actually then uh on paper and and
6:52:03doing a normal storyboard a traditional
6:52:05paper uh storyboard and then I was doing
6:52:08this 360 storyboard like sketching out
6:52:10got few new ideas about light and
6:52:12shadows and what are the interactive
6:52:14objects so that's something that I paid
6:52:17attention to I previewed it so I have
6:52:19all that experience and that experience
6:52:20is important we are taking that
6:52:22experience forward in our physical
6:52:24prototype here and then I was doing a 3D
6:52:27storyboard a 3D storyboard using
6:52:29tiltbrush you can use other tools I said
6:52:32that's fine in some sense um so this is
6:52:36a immersive authoring tool so it's
6:52:38clearly digital and we were doing that
6:52:40for the storyboarding so now we're
6:52:42moving back to physical Michael is that
6:52:44really what you want yes yes that's what
6:52:46I want we're not moving back to physical
6:52:48don't think of it like in that way um
6:52:52think of it in terms of a tool the
6:52:54physical tools are the ones that we have
6:52:56around us in the physical world so
6:52:58you're quite familiar with them we can
6:52:59do so many things with paper and that's
6:53:02something I want you to uh think about
6:53:04so rather than you thinking well this is
6:53:06a step back from til brush you think
6:53:09this is a step forward it's paper it's
6:53:11so much more versatile and flexible
6:53:14obviously um yeah you won't have that
6:53:17steroscopic view and this really really
6:53:19key sense of a preview but I'll also
6:53:22show you again how you can preview some
6:53:24of your prototypes uh using some of the
6:53:26tools that I have developed in my
6:53:27research and you get versions of that
6:53:29you get access to versions of that so
6:53:31here is our landscape from the lecture
6:53:34on physical prototyping I was talking
6:53:36about paper prototyping and 360 paper
6:53:39prototyping a lot so that's like one
6:53:41thing that we're going to try and then
6:53:42along the way we're also going to do 3D
6:53:44models that we can then put in a diarama
6:53:47and uh that will be cool as well so
6:53:50let's get started with the paper
6:53:52prototype I'll show you what I have done
6:53:53as I was working on the exercise so here
6:53:55is uh me sketching out well screens
6:53:59first so H clearly the traditional 2D UI
6:54:02artist in me um that I am not to be
6:54:05honest but anyway I was just like
6:54:07thinking this might be how you also may
6:54:09want to start on paper and I was
6:54:11starting from the smartphone it was an
6:54:13AR experience I was sketching out a few
6:54:14of the um screens and um this paper
6:54:18prototype was really just like the first
6:54:20step all right sketching out uh the
6:54:23different kinds of transitions now it
6:54:25goes beyond wireframes I start coloring
6:54:27things I'll actually turn it into
6:54:28screens um
6:54:30so um this is still relatively wif framy
6:54:34but then I would actually cut it out I
6:54:36could think of doing device stencils I
6:54:39think what I did is actually I took
6:54:40screenshots of these and then just used
6:54:42my phone to um play with the different
6:54:45transitions between these screens and
6:54:47States that's something one can do and
6:54:51it's very easy so you don't have to be
6:54:53so I say physical prototype doesn't mean
6:54:56that you're not allowed to use anything
6:54:57digital in fact you should capture it at
6:54:59some stage and so uh the paper uh uh is
6:55:02fine but then using your phone as a tool
6:55:06um is even better here I was then
6:55:10actually highlighting the interactive
6:55:11areas I was thinking hey maybe one can
6:55:14just like uh the user can just click all
6:55:17these things that are kind of like blue
6:55:19and this highlighted blue and then I was
6:55:21working on the moon I think and uh so
6:55:25also the rings on which the uh planets
6:55:28orbit around the sun I was thinking
6:55:30would be cool if you could just click
6:55:31them and then go back
6:55:32uh to or transition from Earth to Sun
6:55:35and vice versa so I'm going to show you
6:55:37my 360 paper prototype here's another
6:55:39time lapse and so then I was you know
6:55:42you you probably thought wow Michael
6:55:45this didn't look so bad and why are you
6:55:47destroying it now well you know you got
6:55:49to move on
6:55:50so uh learn about the work by doing the
6:55:54next step so I learned about the project
6:55:56and I was like okay this is me just
6:55:58trying to make it rounder this believe
6:56:00me this sun no matter what I tried it
6:56:02never became really round in any case I
6:56:05was then using toothpicks and I just
6:56:08needed more flexibility putting up a
6:56:10little bit of a phone stand here and
6:56:12then this is my plane where the AR
6:56:14objects will appear and now I assimilate
6:56:15these AR objects with physical
6:56:17properties on toothpicks uh physical
6:56:19props on toothpicks and then I can
6:56:22actually try out different Transitions
6:56:24and how how does the relationship
6:56:26between these planets work and what's
6:56:28the right scale so I have like two sizes
6:56:30of the earth I have a third Earth
6:56:31somewhere this is the moon that looks
6:56:34like a mushroom one of my students said
6:56:36and so she's not wrong and um yeah I was
6:56:40just uh playing with this then I was
6:56:42moving to the 360 template uh so you can
6:56:44also use the 360 template but you don't
6:56:46have depth so I was talking a lot about
6:56:48how you don't have depth and how you can
6:56:51simulate depth with different sized
6:56:52objects and how you can do different
6:56:55kinds of uh well orbiting and different
6:56:58kinds of transitions just using the
6:56:59template the template is really quite
6:57:01cool especially if you keep taking
6:57:04pictures and previewing things in AR or
6:57:07VR um with the two versions of the
6:57:10capture tool that I'll share with you
6:57:12hope it works again hope it works on
6:57:14your phones capturing at different
6:57:16stages and uh this is kind of like what
6:57:19I wanted to show you I have a more
6:57:22detailed video that I'll share with you
6:57:23in full length if you want where I
6:57:25really try it out um if you don't go
6:57:28through the whole step of uh creating
6:57:31this 360 uh storyboard maybe uh by
6:57:36watching this video uh you can learn a
6:57:39little bit more about how this may work
6:57:41so I'll share this video with you I'm
6:57:43not going to comment on it now and but
6:57:46it will be
6:57:47available and then I can preview right
6:57:50that's the whole point this template
6:57:52allows us this an acir angular format so
6:57:54we can actually convert it to spherical
6:57:56and then put it on a sphere in a 3D
6:57:58scene and then preview it in VR and also
6:58:01in AR
6:58:02uh this uh implementation that I share
6:58:04with you right now at the stage um when
6:58:07I was still developing the moo uh only
6:58:09has support for VR and 360 so 360 you
6:58:12can do and then virtual reality this is
6:58:14the 360 one and uh at some stage I
6:58:17should be also showing how to do the
6:58:20virtual reality you just click the
6:58:21virtual reality button and put it in the
6:58:23cardboard and um well also here I'll
6:58:26share a longer uh video with you I
6:58:31actually could put cut all this together
6:58:33and uh I think you'll learn quite a bit
6:58:35from just watching these videos as well
6:58:38I wish more people used this template I
6:58:40do think it helps me think a lot through
6:58:43environment and uh also then a little
6:58:46bit of the interactions and you get such
6:58:47an interesting sense uh or actually so
6:58:51many surprises when it comes to
6:58:54scale then I was um I wasn't really
6:58:57happy with how I was commenting on my uh
6:59:01objects I was drawing like my son looked
6:59:04like a pizza or ice cream and I felt
6:59:05like I wanted to eat it so then I was
6:59:08like hey what else could I use I could
6:59:09use a little bit higher Fidelity if you
6:59:11will higher quality cutouts so I got
6:59:14cutouts of Earth Sun and the Moon and I
6:59:16was playing around with them at
6:59:18different kinds of resolutions this is
6:59:19totally allowed you can print out uh you
6:59:22don't have to sketch everything this
6:59:23template is very flexible um you can put
6:59:26things on it and you just have to pay
6:59:28attention to the scale and where you
6:59:29place things but this is the point so
6:59:31the 360 60 paper prototype can be really
6:59:34cool and I'll play around with that as
6:59:36well I'll share that video with you so
6:59:37you'll have the full overview I was then
6:59:40also doing Play-Doh uh or some kind of
6:59:44uh clay modeling here and um yeah I look
6:59:49like I'm making sandwiches or baking and
6:59:52but what I'm actually producing here is
6:59:54Earth so yes no not what you're thinking
6:59:58no I'm not making Earth and no I'm just
7:00:01doing my own little Earth here with um
7:00:04clay and I'm doing two different sizes
7:00:06of Earth cuz I know that I'll need them
7:00:09for close-ups I need a bigger one and
7:00:12then in relationship to the sun I really
7:00:14need to do a smaller one but then I
7:00:16can't make it too small because how tiny
7:00:18is the moon going to be and so that's
7:00:20the thing right in the physical world we
7:00:22we have certain limitations when it
7:00:23comes to resolution and the scale and
7:00:25size of objects and it is so interesting
7:00:28when you then learn about how these
7:00:29translate to VR and AR uh the scale is
7:00:33probably the one thing that we never get
7:00:35right um here I'm making a sandwich no
7:00:38it is actually a night uh version
7:00:41supposed to be a night version of uh
7:00:43Earth and then I was playing around with
7:00:45these texture she texture sheets that I
7:00:47bought really cool so these texture
7:00:50sheets actually give a pattern to the uh
7:00:54models the props and you will see in
7:00:56later videos they actually really stand
7:00:58out especially when you use a digital
7:00:59camera you can see every one of these
7:01:03little uh you know uh
7:01:07little dents that I actually essentially
7:01:09made into the
7:01:12objects was playing around with
7:01:14toothpicks the toothpicks is my way but
7:01:16you can also uh use other ways to
7:01:19stabilize and mount the objects and uh
7:01:22there's a number of different things you
7:01:24can do here's the Sun that I'm making
7:01:27and shaping and I'm trying to mix I'm
7:01:30not sure I always got the colors right
7:01:32at some stage I feel like I had to check
7:01:35it actually doesn't have to look like
7:01:36the paper version that was just me
7:01:39trying to uh match it because I knew I
7:01:42was going to produce videos for you but
7:01:44really I mean it it could look different
7:01:47it should look like a sun but it looks
7:01:48more like a pizza ball or something and
7:01:53then once you have the clay you can
7:01:55obviously really explore interesting
7:01:57things so this is the video that I'm
7:01:58going to share with you again because it
7:02:00is a major step here here and I think it
7:02:03nicely illustrates the advantages of
7:02:05using clay and physical prototyping and
7:02:09having these as well the 3D models
7:02:11obious scenes and I was playing around
7:02:14with different kinds of compositions and
7:02:16uh animations because I could then
7:02:19really move these objects I show a later
7:02:22portion at the end I was playing around
7:02:25with um different kinds of compositions
7:02:29I remember I also put together like
7:02:32uh the Earth the Sun and um the moon
7:02:37yeah I mean this probably looks weird
7:02:39but this is what you will look like when
7:02:40you then are supposed to take your video
7:02:43of your physical prototype and so it's
7:02:45really important I was just running it
7:02:47all the time and um I was using a little
7:02:50bit of basic video editing later on to
7:02:52cut it so we didn't have to share this
7:02:55is now bringing in all the objects and I
7:02:57was going through but the the portion
7:03:00that I was looking for is this one
7:03:01because it does really show nicely how
7:03:03you can then uh bring the camera wasn't
7:03:06always playing nicely with me it was the
7:03:09autofocus was a bit weird but this depth
7:03:11of field effect is something that I fell
7:03:12in love with and um if I knew better how
7:03:15to do shaders in in uh in 3D then I
7:03:19would probably uh come up with that uh I
7:03:23would probably implement a solution uh
7:03:25with the depth of field effect in any
7:03:27case I was then doing the diarama so
7:03:30this is a 6 minute video that I'm happy
7:03:33to share with you I really set it up
7:03:35quite nicely I was proud um to put this
7:03:38all together and I will just go very
7:03:41briefly over what I did so I was
7:03:43bringing back all my materials this
7:03:46diarama is in some way the culmination
7:03:48of these initial uh physical prototyping
7:03:52steps I actually did this last and as I
7:03:56was doing the diarama I was then also
7:03:57bringing in some of my tools again to
7:03:59have some AR uh previews so that's what
7:04:05I'll show later I'm just going to jump
7:04:07ahead a little bit here for example is
7:04:10what you see in the video is how I take
7:04:12some of my objects uh that were placed
7:04:14on paper and how I can then actually
7:04:18preview that in AR now they really
7:04:20floating above my uh little table here
7:04:23and I thought that was so cool so I'm
7:04:25going to share all this with
7:04:28you I was also playing around with
7:04:30different backgrounds and and there is a
7:04:32part in the video that I find even more
7:04:34interesting now that is cool but at the
7:04:36end I actually show this part is really
7:04:39cool because I'll mix the physical and
7:04:41the digital here so I have like a two
7:04:45virtual copies in AR and then I was
7:04:47bringing in a physical Moon and uh was
7:04:50commenting on how obviously occlusion
7:04:52doesn't work because the camera has a
7:04:54hard time seeing these objects that are
7:04:56so close to the camera and it doesn't
7:04:58actually have uh body segmentation
7:05:00implemented AR C at this stage at least
7:05:03and in any case it we just see a
7:05:04toothpick and a little moon but um yeah
7:05:08so you'll see you'll see me go through
7:05:10all these uh exercises and uh steps of
7:05:14the exercise I actually had fun I hope
7:05:17you have fun I hope you do a subset of
7:05:20this and uh you feel like you learned
7:05:22something this is the physical part and
7:05:25um with a few tools to allow us to
7:05:27preview things in AR and VR and then
7:05:29we're going to jump into the digital uh
7:05:31into the digital world after this we're
7:05:34still using authoring tools no
7:05:36programming yet because that's what
7:05:37we're going to leave for the next course
7:05:39but we are going to get to our final
7:05:41prototype that we can then actually also
7:05:43peer review and um yeah we're going to
7:05:46talk about peer review as well
7:06:01so we're still talking about designing
7:06:03XR experiences but we're moving into
7:06:05digital prototyping space and what I'm
7:06:08going to do in this segment is really
7:06:09focus on well how do we take for example
7:06:12our story boards that we created on
7:06:14paper or our physical prototypes how do
7:06:16we take them to the next level in terms
7:06:18of fidelity how do we take it into
7:06:20digital and I'm going to talk about a
7:06:23few techniques in digital prototyping up
7:06:25to the point of actually doing it uh in
7:06:27immersive authoring using AR and VR
7:06:30devices so let's watch a a quick video
7:06:32of what I'm talking about he's now
7:06:35moving around the object and capturing
7:06:37the Play-Doh
7:06:41figure the information will be live
7:06:43streamed back to the editor where the
7:06:46video is available
7:06:50immediately he has now access to an
7:06:53additional resource in the collect pan
7:06:55which you can preview in the capture pan
7:06:57on the right and could actually
7:06:58recapture if necessary in this case the
7:07:02designer is however satisfied and all he
7:07:04needs to do is actually Drag and Drop
7:07:06The Playdoh model capture into his AR
7:07:11scene
7:07:16Ena he's enabling the marker
7:07:21tracking and he can immediately preview
7:07:23the object above the marker he disabled
7:07:26the digital marker representation and
7:07:28now he's showcasing how Pro renders
7:07:32perspective frames depending on camera
7:07:35movement this gives him a first sense of
7:07:37the a interface he wanted to create with
7:07:39prot in
7:07:41this
7:07:48minutes now this is just one way of
7:07:51prototyping things and it's something
7:07:52that I've come up in research and it's
7:07:54not exactly the kinds of tools that you
7:07:56have available um as a product so more
7:08:00generally speaking what does the XR
7:08:02tools landscape look like as a little
7:08:04bit of a reminder we obviously have
7:08:07digital and immersive authoring so we
7:08:09have existing tools here and I just give
7:08:11uh some examples um keep in mind that
7:08:14some of these examples are current
7:08:17examples and we don't know yet whether
7:08:19the tool that I list here is still
7:08:21around tomorrow so we have digital
7:08:23immersive authoring we have webbased
7:08:25development so this is this whole idea
7:08:27of building on the web XR specification
7:08:30and then we have tools like a frame that
7:08:31I've talked about earlier we can do a
7:08:35kind of cross-platform development
7:08:36approach uh where popular choices are
7:08:38unity and unreal these are these game
7:08:40engines that I talked about earlier that
7:08:42are usually combined with some kind of
7:08:44AR VR software development kit or SDK
7:08:47from popular vendors like Ulus Vive
7:08:50Microsoft you can also specifically
7:08:52Target devices and do Native development
7:08:55so this is the fourth approach um the
7:08:57fourth big category of tools uh you
7:08:59could choose a particular SDK like the
7:09:01cardboard Oculus or W SDK to develop for
7:09:04these specific types of platforms and
7:09:07devices so this is the overall landscape
7:09:11and um as you can tell some of them
7:09:13actually require programming some of
7:09:15these tools like Unity unreal with these
7:09:18software development kits or native
7:09:19development actually do require uh quite
7:09:22a lot of experience and programming
7:09:24background so what are these approaches
7:09:28good for um I think the digital
7:09:30immersive authoring tools that I want to
7:09:31introduce you to are good for
7:09:33storyboarding but then there are limited
7:09:36uh there's limited support for
7:09:37interactions so you can do a few things
7:09:39but you will very quickly realize
7:09:41there's not so much you can do after
7:09:43initial prototyping so after that you
7:09:45need to choose one of these three
7:09:47possible paths through the web
7:09:49crossplatform or native so the web based
7:09:52approach I think is very good for XR
7:09:54apps um that have some basic
7:09:57interactions um sometimes it still feels
7:10:00a little unfinished finished with
7:10:01current web XR approaches if you compare
7:10:04it to a a Unity or unreal experience so
7:10:07these would be in the crossplatform
7:10:08development approach um these are
7:10:11actually good for full-fledged XR apps
7:10:13they they're really visually stunning
7:10:15sometime sometimes you can really create
7:10:16really nicely looking experiences there
7:10:19um but I also think they the cost is
7:10:20pretty high the learning these tools
7:10:22usually comes with a very high learning
7:10:24curve depending on the path and the
7:10:27background remember I talked about uh
7:10:29paths to being an XR creator if you have
7:10:31a Game Dev background unity and unre
7:10:33feel very natural to you but if you're
7:10:35coming if you're like me and coming more
7:10:37like from a web dev background it'll
7:10:38it'll be a little harder for you to pick
7:10:40up those tools quickly finally um the
7:10:43native development approach is actually
7:10:45good for again full-fledged XR apps and
7:10:47exploiting platform specific or device
7:10:49specific features even so in some sense
7:10:52it's like the best experience you can
7:10:53create but the issue is that you kind of
7:10:55like locked in with a particular device
7:10:57platform or vendor and that is limits
7:11:00the experience it limits the the the
7:11:02numbers of users that you can
7:11:04reach so now let's really talk about
7:11:06just digital and immersive authoring so
7:11:08digital prototyping um I want to do it
7:11:11the same way as I did it for physical
7:11:12prototyping again we're thinking about
7:11:14low Fidelity to High Fidelity and keep
7:11:17in mind that digital prototyping should
7:11:18be closer to High Fidelity it's like the
7:11:21The Experience should be closer to the
7:11:22final thing that we imagine and then
7:11:24again there are methods that I want to
7:11:26tell you about that are some of them are
7:11:28easier and some of them are harder so
7:11:29how does it how does it work for digital
7:11:32prototyping so digital authoring which
7:11:34is a specific set of tools that allow
7:11:36you to create things through Visual
7:11:38authoring it's like using an editor
7:11:40click and play in some sense and then
7:11:43you can run these experience often
7:11:44directly on an AR uh device for example
7:11:47I'll show you some of these
7:11:49tools uh immersive authoring is actually
7:11:52it really means that you're working in
7:11:53AR or in VR so the way you interact with
7:11:57the content um at this level is is quite
7:11:59different you're really immersed uh in
7:12:01VR or in AR and the tools need to
7:12:04translate a little bit so imagine that
7:12:06with immersive authoring you're really
7:12:08going inside VR or ARS you're putting on
7:12:10your headset and then obviously the
7:12:12interactions change right you need to
7:12:14think about oh the user doesn't actually
7:12:16see the real world anymore how do they
7:12:17interact with the objects in in in
7:12:19physical space and so immersive
7:12:22authoring is really but you're really
7:12:24testing as you're designing it so it's
7:12:26actually quite an interesting way um of
7:12:28prototyping and I'll show you some of
7:12:30the tools in that space in both cases
7:12:32with digital authoring and immersive
7:12:34authoring you really subject to the
7:12:36limitations of what you can prototype
7:12:37with these tools so if you're moving
7:12:39into the other ones that I talked about
7:12:41the other approaches web-based
7:12:42development crossplatform development
7:12:44with unity or unreal or native
7:12:46development specifically for some of the
7:12:48software development kits for some of
7:12:50the arvr devices that we have out there
7:12:53then you're moving slowly but um
7:12:56definitely into that space where it
7:12:58becomes a lot harder to do but you also
7:13:00can can create a lot of Rich experiences
7:13:02so you're moving towards High Fidelity
7:13:04but also it is hard now I wouldn't
7:13:07actually say that all of this is digital
7:13:09prototyping when I use the word digital
7:13:11prototyping I want to draw I want to
7:13:13draw the boundary here around from
7:13:15digital authoring tools these click
7:13:17andplay tools through immersive
7:13:18authoring doing things in AR or VR as
7:13:21opposed to just 3D on a laptop or
7:13:23desktop and then webbased development
7:13:25and just scratching a little bit of
7:13:27crossplatform here because I think the
7:13:29unity and the unreal editor
7:13:31as long as you do not have to write your
7:13:33own scripts for interactions I still
7:13:35think of them as a tool for prototyping
7:13:38so I still and that's what I wanted to
7:13:40that's what I wanted to illustrate by
7:13:42drawing the circle and the boundaries
7:13:44that way okay now let's zoom into this
7:13:46circle of uh prototyping tools and I
7:13:49want to start with digital authoring
7:13:51tools for VR and unlike what you might
7:13:54expect I've have selected Amazon Samaran
7:13:57and unity but just the unity editor part
7:13:59of it I don't refer to Unity as the
7:14:01bigger game engine and the whole
7:14:02platform with c and scripting and all
7:14:05that just as a visual authoring tool I
7:14:08think it is a very popular choice a lot
7:14:10of people prototype directly in unity so
7:14:14um what does what does it mean what can
7:14:15you do with these digital authoring
7:14:17tools where most of them support visual
7:14:19authoring so using a mouse and a
7:14:21keyboard and navigating with you know
7:14:25first person shooter kinds of
7:14:27experiences with W ASD and those kinds
7:14:29of things I give you an intro to that
7:14:30later
7:14:31but they support visual authoring of 3D
7:14:34scene graphs so essentially hierarchies
7:14:36of 3D objects and then they also do
7:14:39support um VR previews so often it's
7:14:42just a matter of quickly connecting your
7:14:44VR device or going through some
7:14:46deployment method to actually quickly
7:14:48deploy it to this Quest here and then
7:14:50previewing that thing that you just
7:14:52created and being happy if you did it
7:14:55right but the workflow is really like
7:14:58using the desktop putting on a VR or ar
7:15:02device and and testing it that way I'm
7:15:05alluding to other tools here immersive
7:15:07authoring uh would give you the
7:15:09advantage that you do everything in
7:15:10arvr but just staying with digital
7:15:13authoring tools um so you can do basic
7:15:15interactions uh without programming you
7:15:17can actually you have usually access to
7:15:19basic interactions like the user looking
7:15:21at things or the user clicking things
7:15:24with a VR controller or touching things
7:15:26on their phone those things are usually
7:15:28uh available in these tools you can also
7:15:31do some more advanced directions um but
7:15:34this is hard uh sometimes you may add
7:15:36specific tool kits to it like for
7:15:39example Unity has the XR interactions
7:15:42tool kit or Microsoft is working on the
7:15:44mixed reality tool kit there is also a
7:15:46virtual reality toolkit vrtk so if you
7:15:48add these kinds of libraries to it you
7:15:50may get away without having to write a
7:15:52lot of your code but um yeah to do
7:15:55Advanced interactions there's usually No
7:15:57Way Around writing code so the next
7:15:59group of tools is digital authoring
7:16:01tools for AR so how does AR differ well
7:16:03there's a different set of tools out
7:16:05here um and they differ in that they
7:16:07actually support visual authoring of
7:16:10either marker based or markerless ar
7:16:12apps we learn more about what that means
7:16:14but in the marker based scenario all you
7:16:16would do is actually use one of the
7:16:18markers that they usually have and um
7:16:21hold it up uh in in front of the camera
7:16:24and um then use this for tracking so the
7:16:27way I would interact with it is using my
7:16:30my phone against the marker and then
7:16:33prototyping uh this way so you can do
7:16:36this with the default markers that are
7:16:38provided like the astronaut I just
7:16:39showed you or you can also create custom
7:16:41markers um in later stages of the
7:16:43prototyping experience I would often
7:16:45just use default markers to begin with
7:16:47because they track really well the sdks
7:16:49work well with them and then custom
7:16:51markers is something when you want to
7:16:53advance into the prototyping stage and
7:16:55maybe building something more like a
7:16:57minimum viable product that gives it a
7:16:59specific theme the last characteristic
7:17:02of digital authoring tools for AR is
7:17:04that they typically enable augmented
7:17:06reality previews either through some
7:17:08kind of software um that you download
7:17:11and onto your computer or have like
7:17:13inbuilt into the prototyping tool like
7:17:15in in the case of lens Studio here so we
7:17:17call this an emulator or some kind of um
7:17:21sending it to device usually requires uh
7:17:24to connect the AR device maybe a
7:17:26smartphone via USB to the computer and
7:17:29quickly deploy to it um for testing so
7:17:32as I was talking about these digital
7:17:34authoring tools I was also pointing out
7:17:36some of their limitations and uh what
7:17:38makes it cumbersome when it comes to
7:17:41actually working with arvr you often
7:17:43have to deploy to device to actually get
7:17:46a preview because existing emulators or
7:17:49existing previews are often not good
7:17:51enough so this is where immersive
7:17:54authoring tools really have an advantage
7:17:56and play an important role again I would
7:17:59distinguish between immersive author
7:18:00tools for VR and AR here I show you two
7:18:03for VR so the two examples I wanted to
7:18:05give here is Google blocks and aradus
7:18:08quill so there are both immersive
7:18:10authoring tools for VR um imagine that
7:18:13I've done these um prototypes on these
7:18:16digital prototypes that you see here
7:18:18I've done them while I was in the VR
7:18:21headset and uh in one case I was working
7:18:25with uh primitive shapes and I was
7:18:28working on a chair in Google blocks you
7:18:30may not be able to see it but that was
7:18:32the idea I try to create a 3D model um
7:18:34because I needed it for my furniture uh
7:18:36placement application and I wanted to do
7:18:39it immersively in VR even though the
7:18:41final experience may be in AR these
7:18:43immersive authoring tools for VR are a
7:18:46good way of prototyping 3D objects that
7:18:48finally end up in an AR or a VR
7:18:51experience it really doesn't matter but
7:18:52you're using VR to prototype and at the
7:18:54bottom you see Oculus quill which is
7:18:56more like a 3D sketching tool for VR
7:18:59it's it's similar to tiltbrush
7:19:00which you have seen in other segments of
7:19:02this specialization and what is
7:19:05interesting about quill is that they
7:19:06actually move the entire authoring
7:19:08environment um kind of like reminds you
7:19:11of flash or macromedia director if
7:19:13you've ever worked with these tools
7:19:14where you can do key frame based
7:19:16animations and you can actually do the
7:19:18same in VR and that's something I still
7:19:21have to learn a little bit more it's
7:19:22actually quite tedious to use the tools
7:19:24in VR but in principle these immersive
7:19:26authoring tools for VR enable visual
7:19:28authoring but in three right you do it
7:19:31in VR so that's quite different from
7:19:33digital authoring in itself how is it
7:19:36different well they make it possible to
7:19:38edit while at the same time you're
7:19:40previewing it in VR so the whole time
7:19:42you are in VR and that is quite a
7:19:44different experience I would say that
7:19:46the overall tool support of existing
7:19:48immersive authoring tools is really with
7:19:49a focus on 3D modeling there is support
7:19:52for animation and uh some kind of
7:19:56scripting in some of these tools but
7:19:58really since you're doing it in VR you
7:20:00really don't want to have a keyboard and
7:20:01write code and the stepwise animation
7:20:04even if it's key frame based and
7:20:07intelligent in some sense is still
7:20:08relatively tedious so I think these
7:20:10tools are best used for doing some kind
7:20:12of 3D modeling and prev visualization
7:20:15for example of the 3D models you have
7:20:17later or the environment that the user
7:20:19will be using and that's where these
7:20:21tools are really
7:20:23good they typically actually support
7:20:25export to Common 3D model formats so
7:20:28this is actually quite useful in fact
7:20:30Google Google blocks here that you can
7:20:31see actually you can share directly to
7:20:33Google poly and in that case um you can
7:20:36find lots of 3D models online that other
7:20:38people have created with these immersive
7:20:40authoring tools you can see them as
7:20:42inspiration or like to push yourselves a
7:20:44little bit there are really people who
7:20:45are really skilled with it sketchfab is
7:20:47an online platform where people actually
7:20:50share the 3D models for free some of
7:20:52which may have been created with some of
7:20:54these IM immersive authoring tools often
7:20:57however they actually use professional
7:20:583D modeling software and uh and Google
7:21:03sketch or 3D Warehouse um is the other
7:21:06kind of um platform that is often used
7:21:09for sharing these kinds of
7:21:10models okay let's look at immersive
7:21:13authoring tools for AR so there is a set
7:21:15of tools emerging for example Apple's
7:21:18reality composer that you can run
7:21:19directly on your iPad or adobe Arrow so
7:21:23what you do with many of them is you use
7:21:25your tablet as your viewfinder a tablet
7:21:28that has AR um cap abilities and then
7:21:31you can also do immersive authoring
7:21:34you're not as immersed as in VR but you
7:21:36can actually make use of physical space
7:21:38and walk around and zoom these objects
7:21:40here that I show you on the left um I
7:21:42did the same example in both of these I
7:21:45just actually placed a 3D model in uh my
7:21:48office and um so Apple's reality
7:21:51composer is one example Adobe arrow is
7:21:55another so what do these tools actually
7:21:57do well they enable visual authoring of
7:21:593D content in augmented
7:22:02reality they make it possible for you to
7:22:05edit these um experiences as you are
7:22:07creating them and you're previewing them
7:22:09at the same time so you don't actually
7:22:11have to do something on a laptop or
7:22:13desktop and then deploy to your device
7:22:15nor you're working directly on these AR
7:22:17devices and in this case you also design
7:22:20specifically usually for the environment
7:22:22that you're in so the examples I showed
7:22:24here on the left why not particularly
7:22:27tied to the specific environment I just
7:22:29place in there often when you do
7:22:32immersive authoring with these kinds of
7:22:33tools you really want to do it in the
7:22:36environment um that you're designing for
7:22:38when you're using these tools for
7:22:40AR the existing tools provide relatively
7:22:44basic support for interactive behaviors
7:22:47any kind of advanced fancy gestures or
7:22:50any kind of oh let's just use this
7:22:51speech command or whatnot those kinds of
7:22:54things would still require programming
7:22:55there's often not support for that but
7:22:58you do have location based trigger so
7:23:01for example as you're going towards an
7:23:02object or clicking a virtual object or
7:23:06bringing some kind of marker into the
7:23:08view all of these kinds of things would
7:23:10be supported usually without
7:23:13programming some of these tools also
7:23:16actually support and exporting to web XR
7:23:19so that you can actually then continue
7:23:21editing uh the experiences you have
7:23:24prototype this way in a tool like
7:23:26A-frame which we're using in this Muk
7:23:29specializations
7:23:31so here I'm showing Apple reality
7:23:33composer I have a very simple scene and
7:23:35just have an object a cube in there um
7:23:39that I can author digitally so I can
7:23:41rotate and uh scale actually both of
7:23:45world but also the object um so rotation
7:23:51and basically have access to all the uh
7:23:53transform tools here skating as well but
7:23:56so this is just digitally but what I
7:23:58actually want to do is immersive auth so
7:24:00I'm actually going to go into AR so
7:24:03you're scanning the environment very
7:24:04quickly that places the object um in the
7:24:08world and now I can look at it and
7:24:11author it um immersively so here uh
7:24:18basically going to change the way it
7:24:21looks to mat to maybe some kind of
7:24:24interesting Reflections um as the
7:24:28properties going to go down quickly to
7:24:31let you see this a little bit better how
7:24:33this might look or we're going to go for
7:24:36this kind of car paint look um or
7:24:39aluminium and can basically play around
7:24:42until we are happy with it not too much
7:24:44we can do with this simple 3D object
7:24:46there are other ways of actually
7:24:47bringing in other kinds of 3D models but
7:24:50um I just wanted to show very quickly
7:24:52how we can do immersive authoring on an
7:24:54iPad and AR as well so this was a quick
7:24:58overview of the first set of tools in
7:25:00digital prototyping specifically as it
7:25:02relates to digital authoring tools and
7:25:04immersive authoring tools and in another
7:25:07video I look into more detail into
7:25:09A-frame and unity as some of the other
7:25:11tools you may be using for digital
7:25:13prototyping as
7:25:26well digital prototyping to we are still
7:25:30still in the designing XR experiences
7:25:32segment and I have uh talked about
7:25:35digital prototyping space already where
7:25:37we're starting from digital authoring
7:25:39tools into immersive authoring tools
7:25:41where you work directly in AR or VR
7:25:43rather than on a desktop computer and
7:25:45then I said there is also web-based
7:25:47development crossplatform development
7:25:49and Native
7:25:51development which usually require
7:25:54programming and um but people who are
7:25:57experienced with these tools would still
7:25:59consider um at least portions of it as
7:26:02digital prototyping so I want to talk
7:26:04about two specific tools here uh that we
7:26:08actually using in this Muk
7:26:10specialization A-frame is a example of
7:26:12web- based um development which we are
7:26:15using it's based on the web XR standard
7:26:18and A-frame is really a tool that I want
7:26:20you to play a little bit around with you
7:26:22may actually like it if you have a webd
7:26:25or web design
7:26:26background Unity is like unreal a very
7:26:30powerful tool usually requires some time
7:26:33to actually learn it Unity comes with an
7:26:36editor um where you can do a lot of
7:26:38things just like clicking and playing
7:26:41and they actually do have good
7:26:42documentation you can do quite a few
7:26:44things in unity without actually having
7:26:47to program and so that's why I've
7:26:49selected these two tools as examples uh
7:26:52that still fall into what I would call
7:26:54digital prototyping for AR and VR so
7:26:57let's start with A-frame A-frame is
7:26:59based on 3GS and webgl this is the
7:27:03technology stack uh that actually is
7:27:05important information to some people out
7:27:07there what it means is like within the
7:27:09HTML 5 web standard we actually have a
7:27:12new element that is called the canvas
7:27:14that canvas is essentially like a
7:27:17graphics context that you can render 2D
7:27:20content into or 3D content and when we
7:27:23talk about webgl we actually usually
7:27:25mean 3D content rendering 3D content
7:27:28into the web a frame also gives you a
7:27:31new set of HTML tags all of these tags
7:27:34actually start with a dash like a scene
7:27:37a box a cylinder lots of tags that are
7:27:40then translated to specific entities we
7:27:43call them entities in 3D so you can
7:27:46actually create 3D scenes but you're
7:27:48writing HTML
7:27:50code aame has an inspector this is not
7:27:53an editor keep that in mind so it allows
7:27:55you to inspect the scene and look under
7:27:57the hood and find out where the light is
7:27:59and why this animation isn't working for
7:28:01example a has Asset Management um in
7:28:04this case you bring in typical uh
7:28:06resources from the web like images
7:28:08videos and then also sounds but then
7:28:10A-frame also has a set for asset 3D
7:28:14model loaders and this is a good way of
7:28:16bringing in common 3D model file formats
7:28:20and I list out two more features of
7:28:22A-frame one is the ECS architecture ECS
7:28:25stands for entity component system and
7:28:27that's something that we'll look into
7:28:28more detail later it's really relevant
7:28:30for programming um just keep in mind for
7:28:33now that you will find lots of
7:28:35components out there the ECS The Entity
7:28:38component systems lots of Open Source
7:28:40components out there that you can plug
7:28:41into your A-frame code they give you
7:28:44access to VR controllers they give you
7:28:46access to kinds of certain kinds of 3D
7:28:48models and certain behaviors so it's a
7:28:51very plug-and playay kind of
7:28:52architecture and that's what I'd like
7:28:54you to think of A-frame at the moment
7:28:57and it actually supports crossplatform
7:28:58both AR and VR so crossplatform
7:29:02XR here I have an example of a 360 photo
7:29:06that my former post dog Max actually
7:29:08took in the lab and I can just like zoom
7:29:10into different areas of this photo um so
7:29:13it looks a little more
7:29:16normal um there we have for example the
7:29:18clock and there we actually have some
7:29:20sketches we were working on Proto XD and
7:29:23360 Mr at the time and 360 Mr then was
7:29:26called 360 anywhere and these all became
7:29:28papers so
7:29:30um if you look at this photo it is
7:29:32actually a 360 photo of the lab so you
7:29:35can see some of these distortions um
7:29:38it's basically an accurate angular uh
7:29:40map an accurate angular format this
7:29:42photo was taken with a 360 camera a
7:29:45Theta V from Rico and uh it actually has
7:29:49two fishee lenses and then it stitches
7:29:51uh at the end it stitches basically the
7:29:54photo together and I'm going to show you
7:29:57how we can use this photo for
7:29:59prototyping digitally with A-frame and
7:30:01with unity all right here we have the
7:30:05A-frame scene so I'm using the 360 photo
7:30:08from before and I'm mapping it onto the
7:30:11sky that allows me to basically look
7:30:13around already and uh then I'm also
7:30:15mapping it onto the sphere here it's
7:30:17part of my example so I'm trying to
7:30:19create this virtual reality where
7:30:20there's this interesting Cousy looking
7:30:22sphere here and so the way I've done it
7:30:24is I've basically uploaded this photo
7:30:26that we took this 360 photo and I linked
7:30:29it as an said so I'm using Code pen here
7:30:32and it shows the code on the left
7:30:34basically the body uh of our HTML
7:30:36document and then I'm using A-frame so
7:30:39Define my a scene link the asset Define
7:30:42the sphere and Define the sky in both
7:30:45cases I'm linking the lab uh as a
7:30:48texture so basically my 360 photo
7:30:49becomes a texture of both the sky and
7:30:51the background and then our 3D object
7:30:53here in the
7:30:54foreground the last thing I did was
7:30:56actually playing around with the
7:30:57material so really making sure that this
7:31:00looks like interesting by playing a
7:31:02little around with the metalness and the
7:31:03roughness so I can adjust actually some
7:31:05of these here and actually preview it
7:31:08immediately in there and metalness and
7:31:11roughness is always something that I
7:31:13think as a concept it's clear but what
7:31:16so basically they determine how
7:31:19metalness how metallic an object is and
7:31:23how rough it is and that then actually
7:31:25gives you a different kind of reflection
7:31:27here so the light will be reflected
7:31:29differently
7:31:30so I understand that as a concept but
7:31:32it's really hard to really get it right
7:31:35and um so before we so let's say I'm my
7:31:39cool with this so I'm going to preview
7:31:41this in VR quickly so I have my Oculus
7:31:43Rift s here I'm just going to put it on
7:31:45my head and uh so I'm working in Firefox
7:31:49and I can see a preview so I'm kind of
7:31:51happy with the composition of the scene
7:31:54maybe that sphere could be a little bit
7:31:55more at a different location but that's
7:31:58something we can fix later
7:32:00and I can also adjust other parameters
7:32:02in the inspector so I can just go in
7:32:05here into the scene edit my sphere a
7:32:09little bit maybe give it a bit more
7:32:11space so the code becomes less relevant
7:32:13right now because I'm using the
7:32:14inspector and I'm going to go to the
7:32:17material and play a little bit just with
7:32:19the metalness and then also with the
7:32:23roughness there's a lot of things we can
7:32:25play around with maybe I'm like happy
7:32:28with it and I'll leave it like that
7:32:30and go back to the scene and I could
7:32:34copy paste these values now into the
7:32:37document here on the left if I'm truly
7:32:39happy but I can continue iterating
7:32:41previe in VR going through that whole
7:32:44process until I'm like really happy with
7:32:46how things look in VR and so I have a
7:32:49combination of digital and immersive
7:32:50prototyping here using
7:32:53A-frame so Unity um has been out there
7:32:57for a while now it's actually growing
7:32:59out of the game development Community
7:33:01Unity was originally a game engine and
7:33:04it still is but it has grown as a
7:33:07platform with significant support for XR
7:33:10both AR and VR so as a result Unity
7:33:14nowadays has integrated support for many
7:33:17XR platforms for example voria which is
7:33:20a marker based AR library is now
7:33:22actually part of unity and ships with
7:33:24unity but you can also add lots of other
7:33:27sdks into unity and um there is
7:33:30increasing support inside Unity built
7:33:32into Unity so that you have to install
7:33:35Less on top of
7:33:37unity one of the things that I
7:33:39particularly like about Unity is its
7:33:41powerful 3D scene editor there's
7:33:44actually a lot of things you can do in
7:33:45this editor you get immediate previews
7:33:47and I'll show you just the basics in a
7:33:49few seconds but imagine um that you can
7:33:53actually create real experiences and
7:33:55edit them in 3D on a laptop or on a
7:33:58desktop so you often get a sense of what
7:34:00it may look like in AR or VR although
7:34:04most often you still need to deploy to
7:34:06these devices to actually figure it out
7:34:08or use like an emulator so Unity also
7:34:10provides increasing support for kinds of
7:34:12emulators and has asset management and
7:34:15actually an asset store which is
7:34:18different from other tools I've talked
7:34:19about so far where you have to like one
7:34:22is the tool and the other is like you
7:34:23try to find 3D models on the web with
7:34:26unity you actually do a lot through
7:34:28their assets a store and you can
7:34:31actually download libraries and add
7:34:33packages to Unity or find 3D models and
7:34:35like I said with A-frame Unity basically
7:34:38supports all kinds of arvr devices out
7:34:41there U many vendors if not all the
7:34:44vendors provide a Unity SDK that you can
7:34:48download so you can develop against
7:34:50their platform and Hardware using
7:34:53unity all right and here we have the
7:34:55same scene in unity basically working
7:34:58with a few object
7:35:00uh I have the started out with the 360
7:35:02photo imported it and created a sphere
7:35:05and then mapped that material uh map
7:35:09that photo onto the sphere and that
7:35:11generates this material now this
7:35:13material I can play around with this a
7:35:15little bit more um as I adjust some of
7:35:18the parameters like uh the metalness or
7:35:22this is U more interesting the
7:35:24smoothness the smoothness uh gives this
7:35:27interesting effect of the light source
7:35:30being reflected so that's that light and
7:35:32we can actually rotate that to change a
7:35:35little bit where the light essentially
7:35:38hits the sphere
7:35:41and can create a more interesting
7:35:43looking effect this way now uh one thing
7:35:48that I'm not exactly happy yet is uh the
7:35:50position relative to the camera so I can
7:35:52either move the camera a little bit to
7:35:54the right or move the sphere a little
7:35:56bit to the left which I think that's
7:35:57what I'm going to do going to move it a
7:35:59little bit to the to the left so now if
7:36:02I press play we're using an EXO rig
7:36:05already so the camera has already been
7:36:06converted to the XR rig using unity's XR
7:36:10plugin so I just press play and I have
7:36:12the Oculus on my head the rift s I'm
7:36:15using here looking at the sphere and I
7:36:19kind of like the position and it gives a
7:36:23relatively good reflection of Max and
7:36:28it's just interesting in where it's
7:36:30floating so I'm kind of happy with the
7:36:32result and we also mapped that 360 photo
7:36:37also into the environment so we
7:36:40basically changed uh the rendering
7:36:41settings here so a couple of ways I
7:36:43could do this but I chose to do it this
7:36:45way I basically replaced um the normal
7:36:51Skybox which is this one with our 360
7:36:55photo
7:36:56and uh that gave me that
7:37:00effect and um well not a big effect but
7:37:04one thing we could do is we could
7:37:05further edit this material as well uh so
7:37:08for example if I'm not exactly happy
7:37:10with the current rotation I could rotate
7:37:12this a little bit differently so I can
7:37:16change the default rotation but then I
7:37:18would also need to adjust the settings
7:37:20on the sphere so basically also rotate
7:37:22the sphere I could make it a little bit
7:37:24brighter if I wanted so maybe that's
7:37:27maybe that's cool and I'm going to try
7:37:30it one last time in VR and start
7:37:32prototyping this way and I'm fairly
7:37:34happy with what it looks like now it's
7:37:36maybe a little Overexposed so but that's
7:37:40fine we could continue play with this um
7:37:42the point of this last part is that we
7:37:46can actually while we are prototyping
7:37:49for VR here using VR techniques um and
7:37:54um basically you could be anywhere and
7:37:56then we Port you into the lab you might
7:37:58as well be in the lab lab and we we're
7:38:01actually prototyping effectively a an
7:38:04arc and um well you can do this while
7:38:09being in the
7:38:10lab or you can do this anywhere based on
7:38:13the 360 photo that you previously took
7:38:15of the location and I think that is a
7:38:18very powerful way of prototyping XR
7:38:21applications uh very quickly and if
7:38:23you're good and skilled in unity which
7:38:26will dedicate a larger part of of the
7:38:30development oriented course as part of
7:38:32this exam work we're going to dedicate
7:38:34that uh to being better at Unity
7:38:36essentially then you can create those
7:38:38prototypes very quickly and I consider
7:38:40the unity editor a powerful tool for
7:38:43prototyping as
7:38:57well when I say evaluation I really
7:39:01think in terms of three big things and
7:39:04depending on who you are it may just be
7:39:06one of them so I'll give you a little
7:39:08bit of an overview of what evaluation
7:39:10can actually mean usability test user
7:39:13study or play test and so how do these
7:39:17differ well they differ in terms of
7:39:19metrics uh that you're looking at so for
7:39:22usability testing you think in terms of
7:39:24usability metrics so efficiency
7:39:26Effectiveness satisfaction for user
7:39:28studies you may actually have this idea
7:39:31of showing an effect uh so maybe
7:39:34determining whether the participants so
7:39:37your subjects in a user study whether
7:39:40they feel or get this feeling of
7:39:42presence and whether they see whether
7:39:44they feel immersed and know so there are
7:39:47presence and immersion questionnaires
7:39:49for example that might be the
7:39:50instruments that you would be using in
7:39:52those types of studies or you might have
7:39:55a hypothesis somehow embodied in your
7:39:57prototype and now you are are testing uh
7:40:00whether you can find support some kind
7:40:02of evidence support for your hypothesis
7:40:05or you might just be interested in doing
7:40:07some kind of benchmarking maybe you've
7:40:09come up with a cool new interaction
7:40:10technique in the XR space and so now
7:40:12you're doing some kind of performance or
7:40:14user performance
7:40:17benchmarking as a game designer and
7:40:19developer you approach this probably a
7:40:22little bit differently one of the things
7:40:23you really really care about is gam
7:40:25playay yeah do the participants well
7:40:28actually do your players have fun in
7:40:30fact you don't talk about users you
7:40:32probably just talk about players and but
7:40:34they're all the same so if I use the
7:40:35term user participant subject or player
7:40:38this all the same the people that are
7:40:40trying out our XR experience you pay a
7:40:42lot of attention to fun you want to find
7:40:45out weaknesses in gamepl whether
7:40:47something gets boring after some time
7:40:50yeah the the questions that you ask are
7:40:51a little bit different uh as opposed to
7:40:53the other two which are really detail in
7:40:56this video I'm going to talk more about
7:40:58usability testing and us studies finally
7:41:00and this may also be depending on
7:41:03whether you're approaching this more
7:41:04from a marketing perspective user
7:41:05retention yeah whether the users would
7:41:08be willing to continue using your
7:41:10experience your product your title or
7:41:12whatever you call it your application
7:41:14whether they would come back whether
7:41:16they would continue to use it and that
7:41:18is usually a function of whether they
7:41:19can see some kind of value in it and
7:41:22whether it's engaging and fun uh so this
7:41:26value might be productive or might be
7:41:28entertaining
7:41:30so these are the things that we are
7:41:32trying to figure out how well are we
7:41:34doing along all these different kinds of
7:41:36metrics and criteria So speaking of
7:41:39criteria I think at the highest level we
7:41:42can distinguish four different sets of
7:41:44criteria and then Associated questions
7:41:46one big one is usability as I said so
7:41:49how well can our users use that
7:41:52interface really really important
7:41:54question and then we're talking about
7:41:55efficiency Effectiveness and
7:41:57satisfaction usually and there are other
7:41:59usability metrics that I'll talk about
7:42:01later as
7:42:02well usefulness it's a very different
7:42:05question whether or not something is
7:42:06usable is very different from whether
7:42:08it's actually useful do our users find
7:42:11it helpful now there could be this most
7:42:13helpful most useful tool but it's just
7:42:16really difficult to use so usability is
7:42:18poor but it's really high on the
7:42:20usefulness and then if you get the
7:42:22usefulness right you will probably still
7:42:24have users but if you don't have
7:42:26anything useful but it's super design in
7:42:29terms of usability it mostly becomes
7:42:32something that us as designers would
7:42:34look at and study it's not necessarily
7:42:36something that speaks to the users
7:42:38utility whether the users see real value
7:42:42does it actually solve their problem
7:42:44that is really really an important
7:42:46question and that question you cannot
7:42:48figure out an answer with usability
7:42:51testing there has to be more to it
7:42:53usually this is a question of some kind
7:42:55of longer term use and there are
7:42:57longitudinal studies to explore exp this
7:42:59in more depth but um utility and
7:43:02usefulness are really Higher Goals
7:43:05higher Ambitions and that are harder to
7:43:08evaluate and then finally user
7:43:10acceptance yeah will the users actually
7:43:13continue to use the product will they
7:43:15come back will they see value do they
7:43:18use it on a daily basis how frequent
7:43:21does usage become a trend is usage just
7:43:25some kind of novelty effect and that
7:43:28novelty effect is something that
7:43:29especially researchers need to be very
7:43:32concerned about your users seeing XR for
7:43:35the first time might be like yeah I'm
7:43:37totally going to use this again this is
7:43:39great I mean I didn't know that you
7:43:40could do this but now that they know
7:43:43this and let's say they buy the product
7:43:46and then they have it will they actually
7:43:48really continue to use it and that is a
7:43:50function of whether that product now
7:43:53provides utility is useful and is usable
7:43:57so ultimately user acceptance is really
7:44:00the combination of all those evalation
7:44:02criteria and
7:44:03questions now I'm going to talk about
7:44:05four selected evalation methods and
7:44:08these are more closely aligned towards
7:44:10user studies and usability testing and
7:44:13usability testing is actually one of
7:44:15them but we start with system
7:44:16walkthroughs these you can do based on
7:44:18storyboards so really in the early
7:44:20stages of design or digital prototypes
7:44:23so when you have something implemented
7:44:24in a digital kind of platform or tool
7:44:27paper prototype testing so this is
7:44:29really that can happen in the phase
7:44:31between storyboarding and digital
7:44:33prototypes still very early and I'll
7:44:35show you some cool things you can do
7:44:37this way and you need to improvise a
7:44:39little bit you need a facilitator and
7:44:41you need a person working as a
7:44:43functioning as a computer as we say here
7:44:46uh there's this idea of the wizard of
7:44:48ass this idea of human simulating uh
7:44:52system functionality that is not
7:44:53implemented yet and um we can simulate
7:44:57that to some extent with paper prod ypes
7:44:59but this may also play a role later when
7:45:01it comes to digital
7:45:03prototypes there is this idea of ristic
7:45:05evaluation and this is something based
7:45:07on the design guidelines that we talked
7:45:09about and then you becoming more and
7:45:11more knowledgeable in the field of XR
7:45:13design um euristic evaluation is
7:45:15something that is usually done by
7:45:17experts considering established design
7:45:19principles and guidelines and then also
7:45:21their own experience actually from
7:45:24designing lots of XR things and also
7:45:26testing and then usability testing uh is
7:45:29the one big field and Method well it is
7:45:33is both a method and it's a whole field
7:45:35because a lot of people have studied how
7:45:37to improve usability testing um
7:45:39basically the way I'd like to think
7:45:41about it here is like you have something
7:45:43digital or maybe even immersive so that
7:45:46means you can actually go into arvr you
7:45:49have that kind of prototype ready and
7:45:51now you're going to do some kind of
7:45:52usability testing it's a little bit
7:45:54different from paper prototype testing I
7:45:56mean you can figure out questions to do
7:45:59with usability even with a paper
7:46:01prototype especially when it comes to
7:46:03like terminology and user flow through
7:46:06an application but whether or not
7:46:07something is truly usable in XR so in AR
7:46:10or VR that is something you can only
7:46:12figure out when you actually have a
7:46:13digital or immersive Prototype all right
7:46:16so these are four big methods system
7:46:20walk through to usability testing so
7:46:23depending on how much we have and and
7:46:24the one thing that you noticed in my
7:46:26selection of the methods is that they
7:46:28really depend a little bit on the stage
7:46:30of design some of the things that make a
7:46:32lot of sense early on and then you would
7:46:34choose different methods if you have
7:46:36more design artifacts so if you have a
7:46:38complete system to test U and I'm going
7:46:41to show you one example here a system
7:46:43walkthrough is something you can do
7:46:45almost at any time you can do it based
7:46:47on the storyboard or you can do it based
7:46:49on the thing that you have implemented
7:46:51and you do this system walkth through to
7:46:53help users think through the system and
7:46:57um well not maybe us but other experts I
7:47:00got in critique and kind of opinion that
7:47:02way without them actually doing a user
7:47:05test a usability test okay it's a
7:47:08walkthrough and uh often this is done uh
7:47:12as a cognitive walkthrough so you're
7:47:14thinking in terms of the tasks that
7:47:16users might want to do and then how
7:47:19would they be able to use your system so
7:47:21let me show you an example based on my
7:47:23own work so here I'll walk you through
7:47:26some of the key features of Proto so you
7:47:29can create screen mockups on paper then
7:47:31you can do some annotations and prepare
7:47:33it for capture then you would just use
7:47:35your phones capture those screens but
7:47:39all the paper actually become screens uh
7:47:42and then also capture 3D objects using
7:47:45some kind of we call the 360 Capture
7:47:47technique and now what we've just done
7:47:49is digitizing a Playdoh object and
7:47:51previewing it in a pretty cool actually
7:47:55so this was a 30- second video summary
7:47:57of one of my uh research projects
7:48:00published in 2018 in the ACM Kai
7:48:03conference the project was called Pro to
7:48:05AR and I've mentioned it before and you
7:48:08may have read the paper already so
7:48:09that's pretty cool and uh yeah so in
7:48:13fact as part of one of the early uh
7:48:16evaluations in this project we actually
7:48:18did a system walkthrough um I didn't
7:48:21have the full thing implemented and
7:48:23ready it I invited students to design
7:48:25jams uh in my lab I often like to work
7:48:28with students obviously these are all
7:48:30master students focusing on HCI so human
7:48:35computer interaction and user experience
7:48:36and interaction design so pretty
7:48:38knowledgeable and so I had them critique
7:48:41uh what I had and the one thing that I
7:48:42don't always get right in my own
7:48:45research is the usability of the
7:48:48projects just because I have larger
7:48:49goals I'm interested in other issues
7:48:52than usability whether something is
7:48:53pixel perfectly aligned and makes sense
7:48:56whether it follows gal principles and
7:48:58all those kinds of things I I usually
7:49:00get that feedback and it's always
7:49:02fascinating to me how and actually
7:49:03really cool how the students destroy
7:49:05some of my own work and obviously in my
7:49:07mind I'm thinking yeah but that's not
7:49:09really the main concern here but okay
7:49:11that's fine um so you get this kind of
7:49:14feedback uh as part of a system walkth
7:49:16through maybe you had a few questions I
7:49:18mean in 30 seconds I went over a project
7:49:20that took half a year to build and then
7:49:23to do uh we used this to do significant
7:49:25research about it and and now I just
7:49:27walked you through it in 60 seconds well
7:49:32um that was the key idea and a system
7:49:34walk through can be can be very quick
7:49:36but can also be like task based and you
7:49:39show the system step by step by step and
7:49:43then you have
7:49:44your your critiques let's just watch it
7:49:47again and look at what I actually did so
7:49:50I selected a few key scenes from the
7:49:53project here um I show some of the key
7:49:56interactions actually one interaction a
7:49:58ows you to capture paper and then bring
7:50:00it into the system the second
7:50:02interaction allows you to capture a
7:50:05physical object and we don't do
7:50:06photogrammetry or all kinds of 3D
7:50:08reconstruction that you imagine no we're
7:50:10doing a much simpler technique here and
7:50:12then the third technique is we can
7:50:13actually
7:50:14preview that captured object in
7:50:17AR and that's really that's really key
7:50:20so I structure the system walk through
7:50:22around three key interactions these are
7:50:24really the important things and also
7:50:25actually the form the contributions of
7:50:27the paper and playdoh played a really
7:50:29important part this idea of quickly
7:50:32physically prototyping mobile AR
7:50:35experiences and then showing uh how to
7:50:37actually transition from the physical so
7:50:39from both from paper which become 2D
7:50:41screens and then paper sketches uh which
7:50:44become 2D screens and then Play-Doh
7:50:46which become actually 3D models and then
7:50:50Play-Doh props modeled physically which
7:50:53then actually become 3D models uh those
7:50:56those kinds of uh transition
7:50:59to make those really quickly uh
7:51:01digitization really quickly I think that
7:51:03was the key and that was also a little
7:51:05novel I would say if not at least uh
7:51:08creative so um that was the system
7:51:12walkthrough and the system walkth
7:51:14through helps you get initial feedback
7:51:17again in the early stages of design you
7:51:19can also use it later but let's look at
7:51:21another method another method that we
7:51:23can do before we have the final real
7:51:25thing and I really like paper prototype
7:51:28so let's look at paper prototype testing
7:51:30I'm going to play a second video here
7:51:32and let's look at it together so we
7:51:35talked about paper prototyping before
7:51:38and here I show basically how this uh
7:51:41technique works we using a specific type
7:51:45of uh 360 template here there's a 360
7:51:48grid and some kind of marker embedded in
7:51:51the paper that is then scanned by the
7:51:52phone and then we're streaming from that
7:51:54phone to a VR user we're capturing that
7:51:58content live
7:51:59to a VR user or to an AR user and we're
7:52:03removing the paper background and so
7:52:05this is basically the 360 Proto project
7:52:07described in my kipe so this is from the
7:52:11360 Proto project that my student KD
7:52:13media and I were working on and it also
7:52:17considers the role of a computer so a
7:52:19person that actually helps assemble uh
7:52:22content here on our little surface and
7:52:25keep in mind anything and everything we
7:52:26place on this grid here in 360 around
7:52:30the VR user or around the AR user and so
7:52:33that's pretty cool and so we can
7:52:35actually prototype quite a few things
7:52:37this way uh paper prototype testing now
7:52:39the first thing that I'm going to show
7:52:40you that doesn't actually involve a lot
7:52:42of live streaming and interactivity is
7:52:45I'm going to layer a interface that
7:52:47actually Katie created for me I really
7:52:49wanted the Star Trek and I'm going to
7:52:51show you the Star
7:52:52Trek so this is the 360 Star Trek and
7:52:56you may have seen this uh in other port
7:52:59of the
7:53:00Muk because it's really a cool prototype
7:53:03to me here I'm going to show it actually
7:53:06in uh our academic Innovation Studios at
7:53:09the University of Michigan and what
7:53:11really nicely worked here is how it
7:53:13mapped uh onto the physical world and
7:53:17how I was just basically leaving the
7:53:18Star Trek bridge now looking into outer
7:53:21space and now I'm going back into the
7:53:25bridge onto the bridge and looking at
7:53:27the captain chair actually looking at
7:53:28outer space again this is the screen on
7:53:31screen and then um here we look at the
7:53:34Cap's
7:53:35chair well kind of a walkth through uh
7:53:38in this case I could actually experience
7:53:40it myself um so this was also kind of a
7:53:44walk through but like quite literally I
7:53:47was walking through that paper prototype
7:53:49so I just made it interactive H it was
7:53:52just as sketches on a 360 paper template
7:53:56so in the physical prototyping as
7:53:58aspects of this course um we talk about
7:54:00more how to do this kind of stuff but um
7:54:04you know I just wanted to bring it back
7:54:06as an example of how powerful 360
7:54:09prototyping uh could be and uh these
7:54:12prototypes can be tested now I as I'm
7:54:15walking around with that thing in my
7:54:16hand with a tablet and I'm I could think
7:54:19aloud like what are the things that I
7:54:21notice um did I expect this or this
7:54:24behaves in an interesting way and I get
7:54:25a prev visualization maybe you don't
7:54:28want have that 360 thing the final thing
7:54:30maybe this is a prev visualization of
7:54:33what you are going to implement later
7:54:34and you get early feedback even before
7:54:37you have made that commitment to now
7:54:39Implement that in unity or in unreal
7:54:41which could be the next iteration and
7:54:44now I'm going to show you another
7:54:45example so this is the Amazon shopping
7:54:47AR view that we prototyped and we're
7:54:50just putting paper Furniture in front of
7:54:52the user basically showing it on the
7:54:54camera and then here is our VR record
7:54:57ball example KD actually implemented
7:54:59this using web XR and now we prototype
7:55:02this we we implemented this first to set
7:55:03a baseline but now we prototype this
7:55:05together with a toothpick and Playdoh as
7:55:08a tennis ball and we had a virtual
7:55:10record so the way we prototype this KD
7:55:13my student was actually go in the
7:55:16Daydream so this is a VR headset and she
7:55:19was controlling the tennis racket on uh
7:55:21using the three do controller so not too
7:55:24much you could do with a Daydream
7:55:25smartphone inside okay and I was the
7:55:28tennis ball and the way we did it was we
7:55:31used the 360 paper template and I had a
7:55:33toothpick well and um well I don't have
7:55:37the exact Play-Doh here so in this case
7:55:40we're going to have a little bit of an
7:55:42orange uh tennis ball but
7:55:45basically I'm just recreating that
7:55:47prototype very quickly here this is all
7:55:49we did where's my toothpick okay we got
7:55:52it taada and now we have our tennis ball
7:55:55and then I had to follow the specific
7:55:57paper template that was printed and I
7:56:00was moving up and down or I don't
7:56:02remember exactly yeah I was moving up
7:56:05and down I think yeah and I was moving
7:56:07up and
7:56:08down to let the ball bounce and then I
7:56:11was watching I had a live preview and
7:56:12then I was watching whether KD actually
7:56:13hit it or not and then I could like
7:56:15bounce it somewhere here or bounce it
7:56:17off somewhere there and that allowed
7:56:20Katie as the VR user to get a sense for
7:56:23oh yeah okay this could be interesting
7:56:26and obviously what I could do as because
7:56:29I'm simulating the the ball and
7:56:31therefore the physics I could we could
7:56:33do whatever we could also play with a
7:56:35really really big tennis ball some of
7:56:37the coolest games created by my students
7:56:39in my courses usually always there's
7:56:41always one or two tennis or table tennis
7:56:43games in every semester um which is
7:56:47pretty cool uh I like those that allow
7:56:49me to cheat because I'm not that good at
7:56:51tennis uh so give me a really big you
7:56:54know a really big tennis ball um and
7:56:56then we can play around with physics and
7:56:58actually have some fun as well and you
7:57:00can play around with physics by
7:57:02simulating that as I said wizard of our
7:57:04style even way before you implement it
7:57:07that's why I think these paper
7:57:08prototypes are so powerful so let me
7:57:11quickly go over the roads just I mean I
7:57:13just explained it basically there's a
7:57:14facilitator that guides the process
7:57:17introduces the tasks and helps the user
7:57:20if they get stuck uh usually the
7:57:23interfaces are not that complex on paper
7:57:25but if the user gets stuck the
7:57:26facilitator prompts then to ask a
7:57:28question it's really important that you
7:57:30don't just like go in there and like
7:57:32tell them what to do because you you
7:57:35you're losing your temper you don't have
7:57:36enough patience now you can see whether
7:57:38the user is struggling and as you see
7:57:40this you actually think about well you
7:57:43prompt them uh is is there a a question
7:57:47you have um could you formulate it as a
7:57:49question and then you jump in as a
7:57:51facilitator you give a short answer to
7:57:53the user um but you still keep it short
7:57:56so uh that they user's attention is
7:57:58still on the task okay and then if the
7:58:01user keeps asking well then something is
7:58:03wrong with the interface and the
7:58:05experience and so as the facilitator you
7:58:07can actually gradually provide more
7:58:10detail and then well as the facilitator
7:58:13and also you have an observer together
7:58:15you can determine the piece of
7:58:17information that really gets the user
7:58:18unstuck and that is key because that
7:58:20will potentially help you refine your
7:58:22interface in the future and then there's
7:58:25obviously this computer and quotes and
7:58:27this computer Sim system dialogue so
7:58:29both system input and output so I was
7:58:31talking about the tennis ball example I
7:58:33was the tennis ball and that was my
7:58:36computer example here and then there is
7:58:38the Observer uh the person that actually
7:58:40documents the study very important what
7:58:43did the user do and how did the
7:58:44facilitator actually help that's really
7:58:46important so in here again is a visual
7:58:48overview of our paper prototype testing
7:58:50method and again I'll invite you to read
7:58:53the 360 Proto paper if you thought this
7:58:55was cool and fun definitely quite a lot
7:58:58fun for us and these are techniques we
7:59:00can do even in the early stages next
7:59:02we're going to expand on some of the
7:59:03other techniques usability inspection
7:59:05usability testing and then also
7:59:07automated usability
7:59:25testing in this video I'm going to talk
7:59:28talk about different types of usability
7:59:30evaluation we have talked about
7:59:31different forms of evaluation before
7:59:33with different kinds of metrics and in
7:59:35this one we focus on usability so we're
7:59:38going to talk about usability inspection
7:59:40involving experts usability testing
7:59:43involving users and automated usability
7:59:45testing which we're going to keep
7:59:47relatively short because this is a new
7:59:49area and we don't have so much to talk
7:59:51about yet when it comes to using systems
7:59:54for automated not automatic but
7:59:56automated usability testing
7:59:58so let's talk about different types of
8:00:01usability evaluation so there are three
8:00:04main categories of usability evaluation
8:00:06I would say usability inspection with
8:00:09expert designers usability testing with
8:00:12actually with Target users and automated
8:00:15usability testing with computer
8:00:17algorithms that's what I say here so
8:00:19these are the three main categories that
8:00:21we can distinguish when it comes to
8:00:22usability evaluation so basically having
8:00:26experts do it for you in involving users
8:00:28and learning this way and trying to
8:00:30involve systems now in practice you may
8:00:33combine some of these methods uh doing
8:00:36the expert evaluation later doing the
8:00:38usability testing maybe earlier and then
8:00:41maybe at the very end as well and uh
8:00:44involving systems throughout you
8:00:46shouldn't think of any of these
8:00:47techniques to be basically perfect and
8:00:51um all of them could actually contribute
8:00:53unique knowledge and so you should think
8:00:56of ways of combining and incorporating
8:00:58some of these techniques so there are
8:01:00many different factors for the choice of
8:01:02evalation method the stage of design so
8:01:05again if you're early or mid or end
8:01:07stage of the project the novelty of the
8:01:09project depending on how much expertise
8:01:12might be there or how many users trained
8:01:14in this technology you may actually able
8:01:16to or how many users of the technology
8:01:19you may actually be able to find so
8:01:21think of an AR application for example
8:01:24that needs is designed for a kind of
8:01:27condition or environment that is very
8:01:28hard to access so NASA for example has
8:01:31several competitions going on about
8:01:33lunar and Mars exploration and so there
8:01:37involving mixed reality uh is a thing
8:01:40and so the nature of the project May
8:01:41dictate whether or not you can actually
8:01:45uh do this kind of testing a big factor
8:01:48is obviously the number of users you're
8:01:50expecting your application uh to be used
8:01:53by and also the budget and time um so if
8:01:58you don't think that the interface will
8:02:01be used by a lot of users then I think
8:02:03the budget and time is probably also
8:02:05going to be relatively small and uh but
8:02:08you really want to get it right the more
8:02:10users uh you have and budget and time um
8:02:14shouldn't be the main reason why you
8:02:16don't test I do think that you should
8:02:18always try to make room in the budget
8:02:20and in your time for some kind of
8:02:23getting user feedback for some kind of
8:02:25feedback from
8:02:26users finally the experience of your
8:02:29team may also be a factor contributing
8:02:30to the choice EV relation method so for
8:02:32example if you're leaning towards a
8:02:34different style if you have an
8:02:36established process or workflow in the
8:02:38company in which you're working U so
8:02:40these are all kinds of factors that
8:02:42together determine which of these
8:02:43methods you might be
8:02:46using again I don't think that any of
8:02:48the methods I'm going to show you today
8:02:50will replace user testing in fact we
8:02:53should actually focus our attention on
8:02:54usability testing and uh thinking more
8:02:57about what ways of actually making this
8:02:59work for XR so let's look at usability
8:03:02inspection first there is this I there
8:03:05is the concept of euristic evaluation
8:03:08which is informal holistic and actually
8:03:11checks against established principles
8:03:13and guidelines it's usually done by
8:03:15experts really understanding the type of
8:03:18interface and the technology in our case
8:03:20AR and VR so euristic evalation is
8:03:23something we can actually do in this
8:03:24course since we are learning about the
8:03:26guidelines and the principles of
8:03:28Designing XR interfaces so we can look
8:03:31at each other's interface and provide
8:03:33some kind of expert opinion uh based on
8:03:37the experience that we have both the
8:03:39things you've learned in the course but
8:03:40maybe also some of your prior experience
8:03:43uh in terms of good user experience
8:03:46design the second concept is the idea of
8:03:48a cognitive walkthrough so the cognitive
8:03:51walkthrough is a procedural walkthr of
8:03:53the system it is usually task oriented
8:03:56so you're imagining tasks you're
8:03:57actually picking tasks that you imagine
8:04:00your users will carry out with the
8:04:01system and it checks if simulated user
8:04:04goals and the memory content at the time
8:04:07at which they are actually pursuing
8:04:09those goals can be assumed to lead to
8:04:13the correct action in other words if
8:04:16you're thinking like the users and we're
8:04:19just assuming the knowledge that we have
8:04:21from operating the interface at the time
8:04:23so the flow so far the screens and the
8:04:26scenes we have seen so far can we assume
8:04:30that what they know at the time will
8:04:32lead to the correct action and there is
8:04:35this idea of formal usability inspection
8:04:37and that is now something that I often
8:04:38do in my class it's a combination of the
8:04:40above methods and I'll walk you through
8:04:43it a little bit so this is something I
8:04:44often do in classes so the idea is to
8:04:46assemble stakeholders so this could be
8:04:48different kinds of designers and also
8:04:51users and then also product owners and
8:04:53all the kinds of stakeholders you can
8:04:54imagine for a project and assign roles
8:04:56to them then you're going to distribute
8:04:58the design artifacts so there is a team
8:05:01that is essentially reviewing the
8:05:03interface and they will receive the
8:05:06design artifacts so this could be the
8:05:08personas the storyboards or any kind of
8:05:11scenarios so narrative storyboards and
8:05:13also obviously any kind of interface
8:05:15prototypes because that is what they're
8:05:17going to be looking at so these could be
8:05:19paper prototypes or an existing kind of
8:05:22implementation in the form of a digital
8:05:24prototype or even if working in
8:05:27interface on an XR device and then they
8:05:30will inspect the design so they will
8:05:32actually go through this and look at it
8:05:34from uh the principles and guidelines
8:05:37that they know about good XR design and
8:05:40also things that may not be as good so
8:05:43essentially they're performing a crit a
8:05:45design crit of your
8:05:48interface and then once the team has
8:05:51finished inspecting there is a courtroom
8:05:54style meeting so this is a moderated
8:05:56meeting you by an independent moderator
8:05:59and there is a design team those that
8:06:01have created the interface and a
8:06:03critique team and so the critique team
8:06:05will provide will provide their critique
8:06:07and the design team gets to defend and
8:06:10rebut some of the things and it's
8:06:12usually supposed to be a constructive
8:06:16criticism and a a constructive
8:06:18discussion a lively discussion around
8:06:20some of the issues often design is about
8:06:22tradeoffs and compromises and so this is
8:06:25an interesting type of conversation that
8:06:28I think is very interesting so I often
8:06:30do this kind of formal usability
8:06:32inspection in my classes so we have like
8:06:35two teams one is uh design team first
8:06:37and then the inspection team and then
8:06:39the other way around so they review each
8:06:41other's designs and then they meet in
8:06:43the classroom and then they hold this
8:06:45cotro style meeting and discuss the
8:06:47design choices and their rationale
8:06:49behind everything and then they get
8:06:51feedback that way now the important
8:06:54thing after this is how to prioritize
8:06:56and decide decide on which of the
8:06:59defects to fix this is really important
8:07:02so after any kind of usability
8:07:04evaluation in this case the inspection
8:07:06you will get feedback now if this is
8:07:09done by experts you can assume to be the
8:07:11you can assume the feedback to be
8:07:14correct and uh valuable at least and so
8:07:18but independent of who gives the
8:07:20feedback I would always take into
8:07:22consideration the whole thing and not
8:07:25just like okay if you fix it implement
8:07:28this we change this don't make all these
8:07:29promises listen um take it in reflect on
8:07:34it and then prioritize and then decide
8:07:38on what to fix you do not most of the
8:07:41time you don't have the time to fix
8:07:43everything and I don't expect that but I
8:07:46do think it's very important to show
8:07:48that you have considered the feedback
8:07:49you have reflected on it and and and
8:07:52that's the way to work with feedback so
8:07:55let's look at the next method usability
8:07:57test in so I said usability testing
8:07:59involves users and here are common
8:08:01usability metrics Effectiveness
8:08:04efficiency satisfaction learnability
8:08:07memorability and I say predictability
8:08:10here so Effectiveness means that users
8:08:12can complete their task accurately
8:08:15efficiency users can perform task
8:08:17quickly and easily satisfaction users
8:08:20will think the interface is Pleasant to
8:08:21use learnability and memorability are
8:08:24not the same thing so learnability is
8:08:26about the idea of accomplishing tasks
8:08:29the first time they see the interface
8:08:31and memorability it's easy to
8:08:33reestablish proficiency after not using
8:08:35the interface for a while so most user
8:08:38studies usually don't look too much into
8:08:40memorability there are specific user
8:08:43studies designed to look into
8:08:44memorability so there's a so for example
8:08:47there are experiments around uh gesture
8:08:50guessability and memorization and those
8:08:53experiments then bring back users after
8:08:55a certain time so maybe a day later or
8:08:58even a week later and that way you can
8:09:01assess
8:09:02memorability and then
8:09:04predictability uh whether the interface
8:09:06responds as expected now these are
8:09:08relatively common usability metrics for
8:09:11for these there are some kind of
8:09:14questionnaires that we are often using
8:09:16some kind of instruments efficiency you
8:09:18can measure in terms of time so we call
8:09:21this time on task user satisfaction and
8:09:24Effectiveness would
8:09:26be through subjective feedback so
8:09:29self-reported for example liquor scale
8:09:31items um from strongly agree to strongly
8:09:35disagree with a certain statement I
8:09:37thought this interface was pleasant to
8:09:39use for example so these are relatively
8:09:42common usability metrics and then I
8:09:44thought okay what are the things that we
8:09:46should care about when it comes to XR
8:09:48and so I I add a few here and you could
8:09:51come up with additional ones but these
8:09:53are the ones that I came up with
8:09:55thinking about this what is specific to
8:09:57XR so I think when it comes to XR we
8:10:00really have to think about the realism
8:10:02so whether an interface especially in VR
8:10:05involes feelings of presence now I said
8:10:08presence and immersion there are
8:10:09specific instruments kind of like type
8:10:11of questionnaires uh for that and that
8:10:14doesn't directly relate to usability but
8:10:17in some but there is this interesting
8:10:19trade-off between making an interface
8:10:22realistic and making it usable uh not
8:10:25everything in the real world is simple
8:10:27and easy to use and so it is I just
8:10:29think it is an interesting consideration
8:10:32to add here safety very important you
8:10:35should really have as little as possible
8:10:38risk of physical or mental harm so
8:10:40mental really also emotional or
8:10:43psychological harm so really those
8:10:46things are no goes and and that's
8:10:48something obviously and obviously any
8:10:51kind of risk of that would really deter
8:10:54from the usability of the interface
8:10:56flexib
8:10:57now that's a little bit of a vague
8:10:59metric here but I was thinking you know
8:11:01in XR we want to support different kinds
8:11:04of input modalities so not just gesture
8:11:07but also speech or sometimes even
8:11:09multimodal gesture and speech so I think
8:11:11with XR interfaces it's very important
8:11:14to support alternate modes of input so
8:11:17provide that flexibility and uh it it
8:11:20actually having that additional support
8:11:22so these alternate modes of input can
8:11:24also increase the complexity of an
8:11:27interface and that may again detract
8:11:29from the usability finally Comfort is
8:11:32something that we should somehow think
8:11:34about measuring because especially with
8:11:37headsets so both VR but also AR extended
8:11:41use might actually be something that is
8:11:44hard to achieve and again if something
8:11:47is not comfortable that may also have to
8:11:50do with usability so for example if
8:11:52you're requiring certain gestures to be
8:11:55performed or relatively long or
8:11:56complicated speech commands then you
8:11:59will also see that the Comfort levels
8:12:00are relatively low so these are
8:12:03additional dimensions and they're
8:12:06relatively unique to XR at least the way
8:12:08I interpret these terms here and
8:12:10together we now have a comprehensive set
8:12:12of metrics and things we can look at
8:12:15when it comes to usability testing with
8:12:17users again it's on us to translate some
8:12:19of these into real instruments I
8:12:22mentioned presence and immersion
8:12:23questionnaires and I link those as
8:12:25resources uh then obviously liquid scale
8:12:28uh ratings and self-reported uh scales
8:12:31for several of the qualitative measures
8:12:33and then we do have quantitative ways of
8:12:35measuring some things you have to
8:12:37structure the usability test into tasks
8:12:40and then there's this idea of assuming a
8:12:43a a a perfectly solved task so basically
8:12:47this model task and using this model
8:12:49task you can also then determine the
8:12:52efficiency of users how much do they
8:12:54actually differ from that perfect
8:12:57execution of the task for example
8:12:59finally I wanted to talk about automated
8:13:01usability testing so automated usability
8:13:03testing basically comes in three main
8:13:05forms this this idea of custom logging
8:13:08and log analysis so you basically add
8:13:10additional things into your interfaces
8:13:12when you're programming them so to keep
8:13:14track of what users are doing uh and
8:13:17generate some kind of user related
8:13:19events and then you can collect those in
8:13:21a database and data perform log analysis
8:13:23so log analysis often used to figure out
8:13:25how users for example navigate through
8:13:28an interface what are the common paths
8:13:30and then you can think about how many
8:13:32sequences do they need so how many steps
8:13:33in the path and is there like an optimal
8:13:36path and uh can we actually make it
8:13:39simpler should we remove some of the
8:13:41paths maybe there are too many options
8:13:44uh and so these kinds of log analysis
8:13:46help you to actually well optimize an
8:13:49interface there are platform based
8:13:51analytics Tools in fact Unity comes with
8:13:54an a form of analytics now these
8:13:56analytics are usually more uh about user
8:13:58retention or player retention and
8:14:01whether or not um how often users use an
8:14:04interface and where and like the
8:14:06geography and and basically a little bit
8:14:08of demographics but not too much really
8:14:10about the internal so the internal usage
8:14:12of the interface so this is really what
8:14:14you have to accomplish through custom
8:14:16logging and log analysis and then there
8:14:18is a new set of tools out there
8:14:21dedicated XR analytics tools is what I
8:14:23say here we have also worked on one in
8:14:26our own research
8:14:27uh the on the mix reality Analytics tool
8:14:30kit I'll talk about this a little bit
8:14:32later and but there are tools out there
8:14:35like cognitive 3D and others I provide
8:14:38them as resources to you I don't have
8:14:40too much experience working with them
8:14:43they they are common in the sense that
8:14:45they're actually quite good in terms of
8:14:47collecting data and visualizing data
8:14:48they produce nicely looking heat Maps
8:14:51the really hard part is the making sense
8:14:53of it so that is just something that we
8:14:57still have to figure out uh what are the
8:14:59best ways to actually assess the user
8:15:02experience in XR are there new kinds of
8:15:06metrics uh is there something we can do
8:15:09automatically I think especially on the
8:15:11accessibility side there are things we
8:15:13can look at more or less in an automated
8:15:15way just like if you think about web
8:15:17design we have really good sets of tools
8:15:19from the w3c on web accessibility we
8:15:23have relatively poor support on web
8:15:25usability I mean we do have tools that
8:15:27allow us to perform online user tests
8:15:30and things like that so more like tools
8:15:32that wrap around user studies but not
8:15:35really automatically determining a
8:15:37usability score these automated tools
8:15:40can help so and so much as I said really
8:15:42on the accessibility side so you can for
8:15:44example look at the contrast of elements
8:15:46so especially in AR that's very
8:15:48interesting uh AR and against the
8:15:50sunlight with like a visor display kind
8:15:52of display like with a hollowland very
8:15:54difficult to see anything uh and so so
8:15:57you can you can find those things by
8:15:59doing Simple color tests and and
8:16:02checking for the the contrast and this
8:16:05the saturation and those kinds of things
8:16:07uh so those things can be implemented
8:16:09and there is also research on that but
8:16:12that a a tool tells you automatically
8:16:16whether your interface will be usable or
8:16:18not H well no such thing I must tell you
8:16:22though in theory must be possible
8:16:23because when I look at an interface I
8:16:25have a certain intuition
8:16:27whether something is usable or not not
8:16:31well I'm not always right but it's just
8:16:33like this intuition that I wished we
8:16:36would develop and that intuition you can
8:16:38develop uh embracing the guidelines I
8:16:40talked about uh doing more and more of
8:16:43your own XI projects and developing
8:16:45experience that way which allow which
8:16:47will allow you in the future to make
8:16:49some of these assessments again none of
8:16:52this you could be this perfect expert
8:16:54designer like you could be Superman
8:16:57uh you can never really predict the
8:16:59usability of an interface unless you
8:17:01really test it with users so think of
8:17:03these methods as different kinds of
8:17:05tools that together will help you and
8:17:07again consider factors such as time and
8:17:10budget or the experience in your team or
8:17:12any kind of limitations that you have
8:17:15that will help you choose the right
8:17:17method
8:17:35in this video we're going to talk about
8:17:36creating your XR design portfolio now if
8:17:40you've taken the course and especially
8:17:43worked on some of the honors content
8:17:45you've actually more or less completed a
8:17:47project step by step by following all
8:17:50the assignments now the cool thing about
8:17:53that is that this is really an important
8:17:55byproduct of this course I mean
8:17:57obviously this project helps you learn
8:17:59about the methods and tools because you
8:18:01practice them but also it might actually
8:18:04be something cool that you can add to
8:18:06your CV and the question is what's the
8:18:09best way to build this portfolio an XR
8:18:14design portfolio and so this is what
8:18:16this video is about I want to give you a
8:18:18few tips obviously I've worked with a
8:18:20lot of students and in my courses I
8:18:22usually actually make it an assignment
8:18:23and an exercise to build out that part
8:18:26for you um and I think it's always a
8:18:29good idea after every project to go
8:18:31through some of these steps and really
8:18:33properly documented now the one thing I
8:18:36want to say before we get started really
8:18:39it is very important that even
8:18:40throughout the project you do a really
8:18:42really good job documenting everything
8:18:44and taking pictures and kind of like
8:18:46writing already some of these materials
8:18:48that I tell you about and this should
8:18:50not happen at the end because at the end
8:18:52it's often too late often you don't have
8:18:53access to all the prototypes anymore and
8:18:56you'll be very sad when something is
8:18:58missing and um so really with this view
8:19:02towards building a portfolio keep this
8:19:04always in mind even as you're working on
8:19:06your projects but now let's get
8:19:09started so an xile design portfolio
8:19:12starts from the problem and we talked a
8:19:14lot about finding good problems ideation
8:19:18and problem Framing and scoping which
8:19:20are really really important things to do
8:19:22otherwise we're taking on something too
8:19:24big too challenging or not clear enough
8:19:27as a problem so you want to be very
8:19:29clear about the problem so then we
8:19:32actually need to describe our method or
8:19:34methods that we Ed throughout most of
8:19:36you will probably have followed the
8:19:38methods that I taught in this course
8:19:40here and so this is a more traditional
8:19:42interaction design method but applied or
8:19:46adapted for
8:19:48XR and then also you should discuss your
8:19:50results yeah you show off what you've
8:19:52done and you document the design
8:19:55Evolution so not just a final thing but
8:19:58also the intermediary steps Your Design
8:20:00concept initially your personas
8:20:04storyboards all those are design
8:20:06artifacts so not just the prototypes not
8:20:08just the final prototype really really
8:20:09important to keep that in mind and now
8:20:11it's very important to connect these
8:20:13three things with a story that story has
8:20:17to be clear from the beginning to the
8:20:18end there has to be a red thread through
8:20:21your design portfolio now that story is
8:20:24easier to do if it's just for one prodct
8:20:26project and it's harder to tell if
8:20:29you're trying to describe multiple
8:20:30projects you've worked on as part of
8:20:32your portfolio now this is just
8:20:35something that I can tell you based on
8:20:37the experience of working with students
8:20:39now I did not have to apply for a lot of
8:20:42jobs in the Exile industry and I'm still
8:20:45a professor here at the University of
8:20:46Michigan but I do work with a lot of
8:20:49people that want to recruit uh in fact
8:20:51several of my students have now landed a
8:20:54job and that makes me really proud of
8:20:55them obviously that's really cool and uh
8:20:58I think it's important um that you take
8:21:01these application processes really
8:21:04really seriously and that you try to do
8:21:07the best job you can when putting
8:21:09together your portfolio so what are the
8:21:11key components these are the five key
8:21:14components a problem
8:21:16statement H this has to be concise it
8:21:18has to be clear it has to feel important
8:21:20I'll tell you more about it physical
8:21:23prototypes you should really include
8:21:24those physical prototypes because that's
8:21:26what you have done essentially is part
8:21:28of your project physical prototypes are
8:21:30really cool because they think without
8:21:32technological constraints this is where
8:21:34you can demonstrate that you can think
8:21:36outside the box which is really really
8:21:38important be creative and then digital
8:21:40prototypes are much more about showing
8:21:43that you master these technologies that
8:21:45you can use the digital prototyping
8:21:47tools or or tools like webxr and then
8:21:51maybe A-frame and unity and unreal but
8:21:54this is really what you want to
8:21:55demonstrate here and then every project
8:21:57and portfolio really has to translate to
8:22:00some lessons learned and some experience
8:22:03and skills gained and I would spell
8:22:05those out really it's important to say
8:22:07explicitly what you have learned both
8:22:10about the problem and then the
8:22:12prototypes that you have created and the
8:22:14projects and like you obviously as a
8:22:17designer how you have grown that is
8:22:19something you should really really tell
8:22:21in this last part so I'll go over each
8:22:24of these parts now and you know provide
8:22:27more detail for each of
8:22:29them so in the problem statement I think
8:22:33it's very important that you say it's a
8:22:37significant design problem so when it
8:22:39comes to the problem statement it's very
8:22:41important that you have a design problem
8:22:44that you're talking about not an
8:22:45information problem so the problem
8:22:47shouldn't be like what's the best way to
8:22:49find and compile that information the
8:22:52problem needs to be a significant design
8:22:55problem what's the best way to design
8:22:57this interaction okay so I'll just say
8:23:00it like that and if you run into this
8:23:03issue of not knowing what kind of
8:23:05information needs to go into your
8:23:08interface then you're not working on a
8:23:10design problem you're working on
8:23:11information problem and that kind of
8:23:14information dashboard problem is not
8:23:16always a good problem to talk about in
8:23:19these kinds of design
8:23:20portfolios so you really need to be
8:23:22clear about the strengths of XR and X
8:23:25here is really the placeholder because
8:23:26because you need to be more specific so
8:23:28why did you use AR or why did you use VR
8:23:30what did you learn about AR and VR for
8:23:33this problem why do you think it is a
8:23:35good solution what makes you choose AR
8:23:38or VR for this solution and for problems
8:23:41where both an AR or VR solution seems
8:23:44feasible and you probably just did one
8:23:46of them you should really explain why
8:23:48you chose the AR or VR version of it
8:23:52maybe a lot of things were already
8:23:54explored in VR and you just wanted to
8:23:55see how much as possible on AR or maybe
8:23:58you have a different reason but it's
8:24:00important that you actually state it
8:24:02next when it comes to the physical
8:24:03prototypes it's very important that you
8:24:05highlight the advanced use of paper
8:24:08prototyping a lot of people want to see
8:24:10that you think outside the box they want
8:24:12to look at these physical prototypes
8:24:14they want to get a sense for a larger
8:24:16design concept that raises all kinds of
8:24:18interesting issues in digital
8:24:19prototyping we usually make a lot of
8:24:21compromises because of the Technologies
8:24:24and other kinds of concerns like budget
8:24:26time and money but when it comes to
8:24:29physical prototyping you can do whatever
8:24:32this is really the place where you can
8:24:34live out your creativity and you should
8:24:36show
8:24:37it and I want to emphasize here that
8:24:39from the physical prototypes it should
8:24:41be clear that you are designing for an
8:24:43AR of VR environment I actually want to
8:24:45see that now often this is accomplished
8:24:47to some kind of color coding where I can
8:24:49see the physical world mostly black and
8:24:51then the augmented World some kind of
8:24:53blue green or red or something so this
8:24:56is often done to tell the story already
8:24:58in the storyboards and then obviously
8:25:00also in the physical prototypes again as
8:25:02a reminder when I say storyboards I mean
8:25:04the wireframes that actually form the
8:25:07foundation of what becomes a paper
8:25:09prototype or a physical prototype later
8:25:12when I say physical prototype I often
8:25:14think of this idea of bringing in
8:25:16additional media like transparency for
8:25:18AR and then clay for 3D modeling and
8:25:21those kinds of
8:25:23things and um don't be afraid of just
8:25:27doing these methods that I teach here
8:25:29and just doing them well that is already
8:25:32a really good accomplishment not
8:25:34everybody uses the methods here and so I
8:25:36think this already should give you a
8:25:38little bit of an edge uh when it comes
8:25:40to
8:25:42portfolios be clear about arvr
8:25:45prototyping differences so that's
8:25:47something that I think we learn more the
8:25:50more prototypes we do but some things
8:25:53are just easier to prototype for VR and
8:25:55some things are easier to prototype for
8:25:57AR so for example because VR usually
8:26:00requires an entire environment unless
8:26:03you are in an environment that is close
8:26:05to it it's often a little harder and the
8:26:08same is actually true with ar but
8:26:10sometimes we can simulate it better and
8:26:12AR maybe the actual 360 or the Sky Box
8:26:16around the user doesn't matter as much
8:26:18it's more like how the interactions
8:26:20relate to the physical world and then AR
8:26:22prototyping AR physically actually can
8:26:24become easier and then when you move to
8:26:27digital actually prototyping AR is
8:26:30harder I find those differences very
8:26:32interesting now this is not always true
8:26:34for every project that I've worked on
8:26:36and it may also be different for
8:26:37different kinds of tools that are using
8:26:40but this is more or less a takeaway from
8:26:43my past few years of prototyping AR and
8:26:45VR
8:26:48experiences next digital prototypes so
8:26:51let's talk about digital prototypes when
8:26:53talking about digital prototypes it's
8:26:55often hard to know
8:26:56how to get started so often we tell this
8:26:59story from the physical prototypes and
8:27:01how our designs evolved into the digital
8:27:04prototypes and then we kind of like Get
8:27:06Lost sometimes in some of the details or
8:27:09we don't really do a good job explaining
8:27:12the strengths and the weaknesses of our
8:27:14digital prototypes so the best way I
8:27:17think to structure a presentation of a
8:27:19digital prototype is to just break it
8:27:21down in terms of the key interactions
8:27:23that you've prototyped so be clear about
8:27:25the interaction you chose to prototype
8:27:28there might be many more things but you
8:27:29chose to prototype those three key
8:27:32interactions for example everybody knows
8:27:34there's a login screen and you did not
8:27:36prototype the login screen first of all
8:27:38because Michael would yell at you
8:27:39because that's what you really don't do
8:27:41because we prototype the things we don't
8:27:43know so much about we prototype the
8:27:46unknowns so that we learn more about it
8:27:48that's what I told you and that's this
8:27:50meage message that you should carry on
8:27:53and tell the other person as well
8:27:56explain why you chose each tool and what
8:27:59for that's really important so in
8:28:01digital prototyping there's a lot of
8:28:03tools involved usually not just Unity or
8:28:05not just like one type of tool so often
8:28:07some kind of Patchwork process between
8:28:09these tools some kind of workflow that
8:28:11is a little messy but that's just how it
8:28:13is and so uh and so it's important to
8:28:16demonstrate that you know uh the tools
8:28:18and that you understand their role and
8:28:20their relevance in the larger process
8:28:23and that's what you should talk about
8:28:25obviously there's often an opportunity
8:28:27to connect with somebody who has
8:28:29experiened sharing the same pain points
8:28:31with an existing tool so this could be a
8:28:33good opportunity as well to share uh
8:28:36some of these
8:28:38experiences it's also very very
8:28:40important to talk about limitations of
8:28:42the tools and how you dealt with them so
8:28:44every tool has a ceiling we talked about
8:28:47that and um so it's important if you
8:28:51felt that seting then it's important H
8:28:53that often very Creative Solutions
8:28:55around it and I would stress and
8:28:56emphasize those when it comes to
8:28:59emphasizing your skills with the tools I
8:29:01think that's a really good a good
8:29:04approach so now we have talked about the
8:29:06problem we have explored the problem
8:29:08more through physical prototyping and
8:29:09then we are nearing our solution with a
8:29:12digital prototype after the digital
8:29:14prototype could be an actual development
8:29:17with some AR devices platforms and tools
8:29:21out there but um I'll just leave it here
8:29:25because
8:29:26at this point in the course you would
8:29:29have concluded the course and so you
8:29:31should talk about your Lessons Learned
8:29:33and I think just from all that knowledge
8:29:35that you've uh gathered by building on
8:29:37your prototypes and doing the ideation
8:29:41and the problem framing you've learned a
8:29:43lot and there's a lot to talk about so
8:29:45you can now already talk about the
8:29:46Lessons Learned for example what was
8:29:49hard easy or surprising to you I would
8:29:52stress those things especially the
8:29:55surprising part
8:29:56but make sure that the surprising part
8:29:59to you is not straightforward to anybody
8:30:02else so for example that prototyping VR
8:30:06or ar is hard with paper and that this
8:30:09is surprising to you okay that's fine
8:30:11it's the first time you do it maybe
8:30:12that's fine but there should be
8:30:14something more specific to your project
8:30:16that you found surprising or hard to do
8:30:19if something was easy I wouldn't
8:30:21emphasize it as much it's just much more
8:30:24interesting to talk about the ch es and
8:30:27again something surprising if it's truly
8:30:30surprising can actually be a really
8:30:32really cool part of the story to tell in
8:30:34your
8:30:36portfolio you should really also talk
8:30:38about what worked well and didn't work
8:30:40well in your team assuming that you have
8:30:43worked in a team I think um talking
8:30:46about those experiences working
8:30:48collaborative on projects um are very
8:30:50interesting a lot of students uh come to
8:30:53my office hours to talk about about
8:30:56issues and challenges they face with a
8:30:58team and I usually encourage them and
8:31:00tell them you know what this is good
8:31:01that this happened because now you have
8:31:03something to talk about an interview
8:31:05because you need to get out of this you
8:31:07need to actually work with your team and
8:31:09so then you have a story to tell and
8:31:12that's what you should focus on so all
8:31:14these challenges that you face
8:31:15throughout the project are the really
8:31:17really important things to talk about so
8:31:19keep that in
8:31:20mind finally this is something that an
8:31:23interviewer or a professor usually ask
8:31:26at the end of a master thesis or
8:31:27Bachelor thesis or some kind of research
8:31:29project so if you could do it all over
8:31:31again what would you do differently and
8:31:34um that's a that's an interesting
8:31:37question because often you know better
8:31:40later and so in retrospect or in
8:31:43hindsight maybe you could have done
8:31:45things differently and so it's very
8:31:46interesting to consider that and at
8:31:48least have this prepped even if you
8:31:51don't make it part of your portfolio it
8:31:52might still be a question that comes up
8:31:54in an interview so think about about it
8:31:57finally we're talking about the
8:31:58experience and skills so the experience
8:32:01here I mean your experience how you have
8:32:03gained more experience from doing this
8:32:06project and you should emphasize again
8:32:09any kind of advanced usage of techniques
8:32:12I encourage you to even talk about just
8:32:15the physical prototyping techniques that
8:32:16we've learned in the course here because
8:32:18I do think they are actually still
8:32:20Innovative they are not standard they
8:32:22are not straightforward and they are
8:32:25still Improv improving and evolving so
8:32:28yes describe what you've learned from
8:32:31practicing these methods and applying
8:32:33these techniques also what I often find
8:32:37very interesting is when a student that
8:32:40has previously done projects in
8:32:42interaction design or graphic design and
8:32:44they have very sophisticated skills in
8:32:47really popular tools out there like
8:32:48Photoshop sketch or in vision and figma
8:32:52and XD and all these prototyping tools
8:32:55out there if they still exist by the
8:32:57time of this recording well these are
8:33:00popular prototyping tools and if and
8:33:02they are not actually designed for AR or
8:33:04VR at least not at this stage so if you
8:33:06can demonstrate how you have used those
8:33:08tools to support your Innovative
8:33:11prototyping I think that is really cool
8:33:14so for example when it comes to 360
8:33:16templates and 360 paper prototyping
8:33:19there's also the there is the grid the
8:33:21VR grid and there is actually there are
8:33:24articles that tell you how to use
8:33:26Photoshop and so I think that's a really
8:33:28good way to demonstrate how you
8:33:30transition and how you transform some of
8:33:32these tools how you adapt your own
8:33:35knowledge and your own methods to XR so
8:33:39this ability to transform your own way
8:33:41of working I think that is super
8:33:43powerful if you can demonstrate that and
8:33:46finally when it comes to arvr specific
8:33:48tools so I mean both like interaction
8:33:51design tools like Pro to iio that are
8:33:53starting to incorporate some support for
8:33:55arvr and then more AR Focus tools like
8:33:58Facebook's spark AR Studio or Snapchat's
8:34:02lens studio and then obviously really
8:34:05like broader platforms like web XR unity
8:34:09and even unreal so if you've used any of
8:34:11these tools as part of your projects you
8:34:13should really really emphasize what
8:34:16kinds of things you've learned with
8:34:17these tools and how you've grown your
8:34:18skills in these tools very important
8:34:21these are the items that go into the
8:34:24portfolio and in the next we do I'm
8:34:25going to tell you more about how I would
8:34:28actually tell the story so these are the
8:34:31main ingredients that go into a
8:34:33portfolio but not necessarily in this
8:34:35order also when you do multiple projects
8:34:38you don't actually have to do it in the
8:34:41order in which you have done the
8:34:42projects no you should choose the order
8:34:45that best fits the story and so we're
8:34:49going to talk a little bit about how to
8:34:51present a portfolio how to present a
8:34:54project in the next video
8:35:09so in this video we're going to talk
8:35:11about five common weaknesses of exi
8:35:13design presentations now as a professor
8:35:15I see a lot of presentations and um here
8:35:18are a few tips and things that I think
8:35:20we should change and improve and then
8:35:22we're going to go through that effort
8:35:24together we're going to put put together
8:35:26an EXO design presentation first we're
8:35:28going to use a normal structure maybe
8:35:30the one that immediately comes to mind
8:35:32and then we're going to make adjustments
8:35:34to emphasize try it and tested
8:35:35techniques and this is really to
8:35:37emphasize my Approach you don't have to
8:35:40do it exactly like me but I have had
8:35:43some good success in terms of
8:35:45presentations uh and coaching students
8:35:47that way I think their presentations
8:35:49were very well received and I was very
8:35:51proud of them so that's at least
8:35:53something in this video we're going to
8:35:55first talk about the trouble with XR
8:35:58design
8:35:59presentations now as I said I see a lot
8:36:02of presentations all the time obviously
8:36:05as a researcher and as an instructor uh
8:36:07we do a lot of XR projects I participate
8:36:10in a lot of research and I see a lot of
8:36:12presentations and new projects and you
8:36:15know please invest into this idea kind
8:36:17of thing um and so yeah I feel like ah
8:36:21there are these five common things that
8:36:23I always find not annoying but like so
8:36:27repetitive I don't know who teaches it
8:36:29to do it this way is it is it our
8:36:31intuition that tells us to do it this
8:36:32way well we'll figure it out and I make
8:36:35a few adjustments here and there and um
8:36:38let's see uh I encourage you to
8:36:40incorporate those into your own practice
8:36:43and try it out you know it could be a
8:36:46transition from the kinds of things that
8:36:47you're used to to a little bit a little
8:36:50bit different and so let's look at it
8:36:53together so here we go all right right
8:36:55so the trouble with xra design
8:36:57presentations on the left I'll start out
8:37:00with the issues that I see often the
8:37:03slides are designed to support the
8:37:04presenter which makes sense right you
8:37:07feel like ah I need to remember all
8:37:08those things so you you create slides to
8:37:11support you as a presenter now this is
8:37:13both you presenting a design project but
8:37:17also you presenting a research project
8:37:19I've seen this a lot and all the time
8:37:22the slides are well designed to support
8:37:24the presenter to help them remember but
8:37:27not the audience and so I'm going to
8:37:30talk about that in a second not really
8:37:33framing the work you're not really
8:37:34telling me what the problem is I really
8:37:36need to know what the problem is I
8:37:37thought we talked about ideation and
8:37:39problem framing so really modate the
8:37:42work and frame it and scope it properly
8:37:44and tell me what it is about a very
8:37:46concise problem statement at the
8:37:47beginning demo last yeah a lot of these
8:37:52presentations they they come I mean if
8:37:55they're well done they have a really
8:37:56nice story and they go on forever and
8:37:59they prepare us to finally in the last
8:38:03slide before we W wrap up we finally get
8:38:06to see the work the demo last
8:38:09approach very unfortunate but so
8:38:12commonly done so maybe we can do it
8:38:15differently uh when we talk about
8:38:17implementation we focus on the Technical
8:38:19Solutions like oh we used webxr and
8:38:22A-frame and I used this and that package
8:38:24in unity and uh that was really what so
8:38:27solved it fine but solved what what is
8:38:31the problem what were the challenges ah
8:38:34and now it's starting to make
8:38:36sense and then uh there is this section
8:38:39usually in presentation which is called
8:38:41future work and when I see those
8:38:44presentations they often read like
8:38:46future things I have to fix so future
8:38:49things I didn't get to implement future
8:38:51things I didn't get to do uh future
8:38:53things I still have to do Implement
8:38:56design whatnot but that's not future
8:38:59work it's not about your future work so
8:39:02here are my suggestions okay first of
8:39:05all we do the slides and we design them
8:39:08to support the presentation especially
8:39:10the audience not just the presenter make
8:39:13it visual not just bullet points I mean
8:39:15this is a really bad slide as an example
8:39:18but you know I need to get across the
8:39:19message here and then not really framing
8:39:22the work well how about stating the
8:39:24problem using an example and choosing an
8:39:27example that a lot of people can relate
8:39:28to something simple not too complex and
8:39:31it'll be Crystal Clear instead of demo
8:39:34last try the Demo First really like
8:39:38after a little bit of problem
8:39:39description here is our demo and then
8:39:42I'll tell you how we did it this is a
8:39:44really refreshing way to actually
8:39:46approach a design presentation so don't
8:39:48leave the demo Until the End don't make
8:39:51me wait for
8:39:52it implementation focused on design
8:39:55challeng challenges compromises I think
8:39:56it's a much better way to do it than
8:39:59here is the few Technologies we chose
8:40:02and we don't tell you why we don't tell
8:40:04you about the challenges we don't tell
8:40:06you how we had to make those compromises
8:40:09so structure your implementation around
8:40:11that not just like the buzzword and all
8:40:12the tech but really why why did you
8:40:16implement those kinds of things the way
8:40:17you did finally instead of future work
8:40:20AKA things I have to do things I have to
8:40:23fix how about thinking about it this way
8:40:25future work as in like future research
8:40:28and design that others could do building
8:40:31on this knowledge so you telling others
8:40:34how to take this work further that is
8:40:36really important it's quite a difference
8:40:38and so you should you should think about
8:40:40it this way so this was really important
8:40:43to me and now we're going to make it a
8:40:45little bit more actionable by actually
8:40:47looking at how to well fix the standard
8:40:51presentation format that I always get to
8:40:53see as the first draft and um so here we
8:40:56go XR design presentation okay we have
8:40:59an intro we talk about our physical
8:41:01prototypes digital prototypes maybe we
8:41:02do a demo almost last as I said right
8:41:05and then we summ we summarize we we
8:41:07provide a conclusion or something like
8:41:08that yeah pretty common pretty standard
8:41:11okay so what goes into the introduction
8:41:15yeah how about you make it in motivation
8:41:18okay so you make it a motivation and
8:41:20you're using an example and that example
8:41:22is designed to motivate the the problem
8:41:26and actually it needs to be to the point
8:41:28like it needs to be very clear you need
8:41:30to prep the problem statement and the
8:41:31problem statement needs to be clear and
8:41:33concise and you you might even write it
8:41:35out our problem is and then text yeah
8:41:39something short not a really long not
8:41:41the but a short concise statement that
8:41:44would be great so instead of the
8:41:46physical prototypes digital prototypes
8:41:48and then the demonstration we're going
8:41:51to do the demonstration next right after
8:41:53the motivation I've just MO the problem
8:41:55and here is what we have done and now
8:41:58what would be super cool is if you gave
8:42:00a live demo right run that thing maybe
8:42:03using mixed reality capture so that
8:42:05other people can see better what's going
8:42:07on in the world in the virtual world
8:42:09around you as you're like stepping into
8:42:11this virtual world if you don't know
8:42:13exactly what mixed reality capture is
8:42:15well very briefly it's like basically a
8:42:17view on you from the outside but we can
8:42:20see how the virtual reality or augmented
8:42:23reality content actually relates to you
8:42:25we basically see it from the perspective
8:42:27of the camera where we see both you and
8:42:29the virtual content around you very cool
8:42:31way to capture this kind of things and
8:42:33we have done this in the what is XR
8:42:35lecture in the very first course in the
8:42:38very first week and I've used these
8:42:40techniques throughout the moo and so
8:42:42maybe you thought this was actually
8:42:44pretty cool turns out it's not too hard
8:42:46to do okay your backup is a record
8:42:49presentation and you should always have
8:42:50that backup it's very important have a
8:42:53backup okay and have it recorded So on
8:42:56the day when usually everything goes
8:42:58wrong on the day of the demonstration
8:43:00and you're trying to do this live demo
8:43:02if something goes wrong if you're like
8:43:04anything like me you probably get a
8:43:06little nervous and maybe even angry at
8:43:08yourself and that's not good so you have
8:43:11to have that knowledge that comfort that
8:43:13you have this backup video this backup
8:43:15recording and then you just use it to
8:43:17switch it then you just use it and you
8:43:19switch to it and you don't apologize too
8:43:21often and this is more like a no to self
8:43:23because I keep apologizing all the time
8:43:26but it's not a good use of your time
8:43:28okay so instead of physical prototypes
8:43:30and digital prototypes you structure
8:43:32this around design Evolution because
8:43:35that's why we did those prototypes okay
8:43:36and then you talk about the prototypes
8:43:38but only a select few just a key
8:43:40iterations the most important ones not
8:43:42like what I see a lot in presentations
8:43:45is like people are trying to tell me how
8:43:46much work they did so they're showing me
8:43:48all the prototypes they have done and
8:43:50it's it's definitely impressive it's
8:43:51cool all the sketches and blah but it's
8:43:54really difficult to follow and actually
8:43:55extract really how did this thing
8:43:58progress how did the design evolve and
8:44:02so it's very important to structure your
8:44:04presentation around a select few
8:44:06prototypes and then really spell out the
8:44:07decisions so the the creating the so you
8:44:11know you do the prototypes to create
8:44:13choices and then you make choices and as
8:44:16you're making those choices you need to
8:44:17provide a good design rational for those
8:44:20choices and those are the three key
8:44:22things that I want to see the prototypes
8:44:24the decisions
8:44:25and the rationale very
8:44:28important and now we have done
8:44:30motivation demonstration design
8:44:32Evolution and then we're going to do the
8:44:33summary and now on to the last part
8:44:36instead of summary focus on the Lessons
8:44:38Learned really how you have grown right
8:44:41we've just discussed all the things that
8:44:43go into the design portfolio and I said
8:44:44Lessons Learned and skills and those
8:44:46things are really important you need to
8:44:47spell it out so this is where this goes
8:44:50Lessons Learned and then the limitations
8:44:53so the compromises you have to make and
8:44:56basically the assumptions you decided to
8:44:58make and the limitations that result
8:45:01from this and then comes the really key
8:45:03part future work and again future work
8:45:05is not like things you have to do in the
8:45:07future no what can others do building on
8:45:12the knowledge of your work that is
8:45:14you're basically telling me what I can
8:45:16do with the knowledge what could others
8:45:18do in the future or what might you do in
8:45:20the future I mean that's fine but the
8:45:22most important thing is don't talk about
8:45:24all the little things you need to fix to
8:45:25make it better that doesn't help anybody
8:45:28right so you're so deep down in your
8:45:30project and I understand how this
8:45:31happens you just finished it you're
8:45:33excited to give this presentation you
8:45:35didn't have enough time to actually zoom
8:45:36out but as part of the presentation that
8:45:39project presentation you really have to
8:45:41zoom out and you have to tell us this
8:45:43bigger story and um and then there might
8:45:48be a questions part in your presentation
8:45:50and the way you navigate the questions
8:45:52and how you you're demonstrating this
8:45:54ability to go down into the detail
8:45:56picking this design rational bringing it
8:45:58up maybe even questioning it yourself
8:46:01considering the other idea that the
8:46:03person that critiqued you had and then
8:46:06not being too defensive but like
8:46:08embracing that idea and providing an
8:46:10opinion about that idea maybe suggesting
8:46:13another issue while that idea sounds
8:46:15really cool and I admit we haven't
8:46:18looked at it there could be these other
8:46:20problems coming from this because in my
8:46:22experience what I've learned from this
8:46:24project
8:46:25we had this problem and sorry I forgot
8:46:28to tell you this no you don't do this
8:46:29sorry we don't apologize we don't do we
8:46:31don't we don't apologize okay now we
8:46:33should have all the things we need to do
8:46:35really cool design and then also tell
8:46:38the story about those designs okay so
8:46:41good luck um if you're using the
8:46:43materials from this course especially if
8:46:45you're doing the honors track stuff so I
8:46:47think there should be a lot of little
8:46:49things that you can add to your
8:46:50portfolio and now combined with this
8:46:52lecture you also can use it as a
8:46:54guideline or a setup guidelines to give
8:46:55a nice little presentation about it let
8:46:58me know how it goes I'm always eager to
8:47:00hear how it's going with my
8:47:15students welcome so I'm going to have a
8:47:18conversation with Katie Lewis Katie
8:47:20Lewis used to work in my lab U we've
8:47:23actually worked together for two years I
8:47:25got to know her in a design Jam in fact
8:47:27and then uh well liked her a lot she's
8:47:29really great student and I and now she's
8:47:31out there in the world gred and so I
8:47:34thought hey Katie can we chat uh you and
8:47:37I had lots of discussions around
8:47:38portfolios and you were so experienced
8:47:40so um we've done this actually before
8:47:42I've invited her to a few of my classes
8:47:44so um I'm looking forward to having a
8:47:46conversation with Katie about the design
8:47:49portfolios and how it's out there uh
8:47:52working a little bit in the XR space so
8:47:55Katie how's it going it's going well so
8:47:58uh right now I work at IBM as a ux
8:48:01designer um but I also was working
8:48:04recently on a team uh working in XR uh
8:48:08for data visualizations and so uh I've
8:48:11gotten some real world experience to
8:48:14apply uh my uh academic education
8:48:17towards uh but everything uh yeah is
8:48:22good okay that's that's great so I mean
8:48:26you know you didn't have the traditional
8:48:28kind of like I just do my bachelor and
8:48:30then I do my masters and then I I don't
8:48:33even know whether that's traditional but
8:48:35I remember your uh CV was a little
8:48:38unique so can you tell us a little bit
8:48:39more about your background even before
8:48:41you joined
8:48:43Michigan yeah absolutely um so uh I came
8:48:47from the Fine Arts so my bachelor's is
8:48:49actually in painting and print making
8:48:51and uh after school I explored the world
8:48:54of uh being a fine artist and uh
8:48:58gradually adapted more of a digital
8:49:01approach to creating uh things
8:49:03interfaces websites and so I kind of
8:49:07stumbled upon uh HCI and fell in love
8:49:11with the idea of using psychology to
8:49:14create um interesting experiences with
8:49:18technology and so I realized that um I
8:49:21had 10 years of actual experience but I
8:49:24really wanted a strong foundation in
8:49:26understanding all of those principles of
8:49:28HCI so that's kind of what brought me
8:49:30back to uh grad school cool all right so
8:49:34then we've worked together on a few
8:49:36projects uh I mean what have you done so
8:49:39a few research papers together I think
8:49:41um you did a master thesis you obviously
8:49:43graduated from the program and you took
8:49:47um at least one of my courses uh or did
8:49:49you take both of them I don't remember
8:49:50no it was a advanced um AR okay so the
8:49:54more technical
8:49:55so that would translate to the
8:49:57development oriented course in our moo
8:49:59um so yeah that's great and you said
8:50:02origal originally you a designer uh and
8:50:04how would you describe yourself today
8:50:06are you a designer or a
8:50:08developer uh I would say I'm a designer
8:50:11but uh having development skills has
8:50:14been very helpful for you know being
8:50:16able to ideate on Concepts and create
8:50:20something that's um functional to show
8:50:23people my ideas is so um I would say I'm
8:50:26a designer that uses all the tools that
8:50:29I can get my hands on that's great okay
8:50:33and uh well so today we're going to talk
8:50:36a little bit about your portfolio and I
8:50:38thought we could uh bring it up and
8:50:41share it and then we now we kept it like
8:50:45at a version that is almost like when
8:50:48you gradated uh but your CV is updated
8:50:52right so this is like what we're going
8:50:53to look at today that um so Katie you
8:50:56have worked now at IBM for like several
8:50:59months is it already a year I don't a
8:51:01year it's been a year oh my God okay
8:51:03time passes quickly all right so I'm
8:51:05going to bring up your portfolio ktie so
8:51:07this is the first thing we'll see when
8:51:09we go to your portfolio did you submit
8:51:11this uh this is the website that you
8:51:13usually use to submit uh with your
8:51:16applications yes all right cool so then
8:51:19you say ux designer and researcher here
8:51:21at the time and have a little bit of a
8:51:24statement about yourself and so I'm kind
8:51:27of like functioning as a reviewer a
8:51:28little bit and um I'm basically seeing a
8:51:31number of really cool projects you seem
8:51:34to have paid attention to you know
8:51:36choosing some representative screenshots
8:51:38but tell us a little bit about how you
8:51:40went about designing this
8:51:42portfolio so I wanted to pick uh a
8:51:46variety of projects that kind of spoke
8:51:49to my different disciplines so I wanted
8:51:52to really focus on XR here because when
8:51:55I was doing the job hunt I was really
8:51:58looking to work in that space and so I
8:52:00wanted people that came here to see that
8:52:03you know I have experience uh ideating
8:52:06on you know AR VR work um and I also
8:52:09wanted to show that I have strong ux
8:52:12design skills um because I was hoping to
8:52:15translate those into the you know XR
8:52:18space and so I've got um development
8:52:21projects I have uh uh XR projects but I
8:52:25also have straight up ux uh research and
8:52:29design projects as well so I really
8:52:31wanted to focus on those kind of three
8:52:33domains yeah so these are all projects
8:52:35that I actually remember mostly from
8:52:38your time here so you made a decision
8:52:39not to include a lot of your your
8:52:42earlier work um why why why did you make
8:52:45that
8:52:46decision so I wanted to um really focus
8:52:50on those three domains and I felt like
8:52:54the language that you create during um
8:52:58the master's program really kind of uh
8:53:03is standard across the jobs I was
8:53:05applying to where my research
8:53:07methodologies and my development
8:53:10capabilities they were just not as
8:53:12strong because I didn't have the
8:53:14foundation that you know the education
8:53:17kind of gave me so um I felt like well I
8:53:20could create a story out of my previous
8:53:22projects they didn't reson as well as
8:53:25what I was able to do during um during
8:53:28the
8:53:29courses okay yeah I I understand okay so
8:53:33that makes sense and so it is I mean
8:53:35still impressive right this is two years
8:53:36of work and we've actually done even
8:53:39additional projects so if you you've
8:53:40picked you've picked some but uh yeah so
8:53:44this hasn't been updated in a year so
8:53:46need need a few more pages now okay
8:53:48you've been busy okay now so let's look
8:53:50at this uh kind of like at the time of
8:53:52your graduation I think this is very
8:53:54relevant to our Learners so um which one
8:53:57should we talk a little bit more about
8:53:58in depth I think this is something that
8:54:00also happens in in interviews they let
8:54:02you pick or I mean you you can tell us
8:54:05what it was like for you when you
8:54:06applied but which one would you want to
8:54:08talk about yeah absolutely I would love
8:54:10to talk about AR kitchen all right so
8:54:12walk us a little bit through it and I I
8:54:14can scroll down a little bit as you're
8:54:15talking about it absolutely so this
8:54:18project um when I was putting this
8:54:21together in this portfolio I really
8:54:23wanted to stick to um the storytelling
8:54:27aspect that you know kind of Lent itself
8:54:29to this project really well uh so I was
8:54:32working Michael's lab and uh a kitchen
8:54:35manufacturer came to us to create a uh
8:54:39showroom experience that leveraged AR
8:54:42and VR technology and so uh with that
8:54:46problem space we uh had the opportunity
8:54:48to ideate around a few um different
8:54:52concepts um which you know involved like
8:54:55a lot of sketching at first um and then
8:54:59we kind of iterated into the digital
8:55:02space where we created several
8:55:04prototypes and so uh yeah so you'll see
8:55:07some kind of initial Concepts where we
8:55:09were looking into projected AR
8:55:11experiences and so this was a really
8:55:14great opportunity for um our team to
8:55:17explore research associated with
8:55:20projected augmented reality as well as
8:55:23some like kind of commercial ideas uh
8:55:25and so yes so there was lots of
8:55:27cardboard and lots of um using the lab
8:55:30to kind of mimic what a kitchen showroom
8:55:33might look like uh and we were even even
8:55:35able to leverage some of our uh paper
8:55:37prototyping skills um that I'm sure
8:55:40Michael has talked about before uh and
8:55:43so preaching still preaching I still I
8:55:46still believe this is the way to go but
8:55:48yes P prototyping cool but it helped us
8:55:51it helped us really iterate through a
8:55:53lot of ideas Qui L um to see what you
8:55:56know like uh what kind of uh interfaces
8:55:59we wanted to develop and so if you
8:56:01scroll down a little bit um this is kind
8:56:05of where we landed on prototypes uh so
8:56:09we uh defined that having a controller
8:56:12for a showroom host um to then uh be
8:56:16able to operate and customize a kitchen
8:56:19cabinet projection for a customer was
8:56:22kind of the the realm we wanted to stay
8:56:24in and so we developed a VR version that
8:56:28um could uh provide a 360 view of the
8:56:32cabinets as well as a an AR version that
8:56:35could show the Cabinetry uh within the
8:56:38the space for the customer but then uh
8:56:41they also control the projected screens
8:56:44um and so with this we developed like
8:56:46four different prototypes to then show
8:56:48the cabinet manufacturer like the
8:56:50possibilities that XR could provide yeah
8:56:53so for you said Okay so let's talk about
8:56:56so tablet you talked us through mobile
8:56:58AR so that's cool and then here we have
8:57:03the
8:57:04projection and here we have a mini
8:57:08kitchen projection and we we we did a
8:57:10good job right we involved the
8:57:11stakeholders and everything throughout
8:57:12the project and there was even a fifth
8:57:15prototype which is like crazy custom
8:57:17display prototype oh yes mhm that nobody
8:57:21nobody liked but okay the hologram the
8:57:24Hologram yeah I was very passionate
8:57:26about that building my custom screen and
8:57:29trying to uh you know uh convey that
8:57:32idea but okay so then you demos and
8:57:34interviews cool so we have essentially
8:57:37okay so from a portfolio perspective it
8:57:39seems like you're like picking really
8:57:42prominent prominent uh screenshots and
8:57:45like photos uh have a little bit of a
8:57:47teaser image um story who is
8:57:51involved uh maybe what kinds of tools
8:57:53you use what kinds of skills and
8:57:55experience you gained um that's
8:57:57something I'll read when I read across
8:57:59your portfolio as well and then it's
8:58:01very visual so I really like I really
8:58:03like uh how how you made it Visual and
8:58:05and uh very appealing actually so uh
8:58:10what is so from your experience um is
8:58:12this a project that you usually talked
8:58:14about uh at the time in some of your
8:58:17interviews yes it usually came up
8:58:19because it it definitely leveraged all
8:58:23of the skills that I really wanted to
8:58:26show to potential employers so it showed
8:58:29um creative thinking it showed using low
8:58:32Fidelity to High Fidelity prototyping
8:58:35which is incredibly valuable in the
8:58:37workspace when you know resources are
8:58:39really expensive um it also showed my
8:58:43development capabilities uh as well as
8:58:46um being able to like craft you know uh
8:58:49pieces for customers and then being able
8:58:52to demo those back to uh executive
8:58:55stakeholders so uh I felt like it was a
8:58:57really like um thorough project that
8:59:01that helped leverage everything that I
8:59:02have cool all right let's go back to
8:59:05your um portfolio overview um is there
8:59:10what what else should we talk about well
8:59:12um what do you feel what what else would
8:59:14you like to share about your
8:59:16portfolio let's see um
8:59:19I I believe that uh I could show a
8:59:24little bit more uh ux research work um
8:59:28and so maybe together Chicago is a good
8:59:32um uh example of that
8:59:35okay uh and this is a traditional like
8:59:39ux research and development work I
8:59:41worked with a team from the University
8:59:44of Michigan engineering um uh department
8:59:48and uh yeah and so we were again
8:59:52provided with a problem um by a
8:59:54stakeholder which was a nonprofit called
8:59:57together Chicago and uh they were
9:00:01looking to develop a mapping interface
9:00:04to help provide resources to the people
9:00:07of Chicago um uh connect resources that
9:00:11other nonprofits were providing to
9:00:14people that needed them and so with this
9:00:17mapping interface again we like were
9:00:18able to start at a very low Fidelity and
9:00:22have many rounds of iteration and
9:00:24testing uh and then we were able to
9:00:26develop uh you know different levels of
9:00:29fidelity and provide that testing back
9:00:31to the stakeholders to users uh and so
9:00:34we were able to gather a lot of data and
9:00:36then provide iterations on that and so
9:00:39um with this project I really wanted to
9:00:42show uh my capabilities of not only
9:00:46development but also being able to
9:00:49iterate and test on you know those
9:00:51developed
9:00:52interfaces well thanks SK for that
9:00:54review and letting us see some of your
9:00:57uh portfolio items and and pieces and
9:00:59there's a lot of love in each of these
9:01:01projects so I can see that and great
9:01:02care um so I wanted to so one thing that
9:01:05came through is like you wanted to um
9:01:08build a solid foundation this is my ex
9:01:10skills and then here's some of my more
9:01:13experimental uh the research that you
9:01:15did and um XR oriented projects um did
9:01:19you think uh you have to aim for that
9:01:22kind of like mix of
9:01:24uh projects like striving some balance
9:01:26between the crazy XR and the
9:01:27foundational uxx um what what when in
9:01:30your thinking when you put that Prof
9:01:31portfolio together I don't think it's
9:01:34entirely necessary um I think it really
9:01:37just depends on you know showcasing your
9:01:40experiences that you've had and being
9:01:42able to uh use all the skills that
9:01:45you've created to then you know get the
9:01:48jobs that you want um that just happened
9:01:51to be an experience that I had that
9:01:53really impacted the way I work with
9:01:56development the way I work with
9:01:58stakeholders and so I felt like that was
9:02:00a really representative project but if
9:02:02that project had been XR then that would
9:02:04have been the story I would be telling
9:02:06but it was more around the experience of
9:02:08working on a multi-disciplinary team
9:02:10that I really wanted to talk about okay
9:02:13so and then uh how did the the
9:02:16interviews go did any any kind of
9:02:19experience you want to share there with
9:02:20us did it go like was the portfolio
9:02:23really helpful were you able to drive
9:02:25the conversations um how do you think
9:02:27about those
9:02:28things so I think the portfolio is just
9:02:31a visual representation of how you tell
9:02:33your story uh so it helped me to Define
9:02:38what those stories are uh so that when I
9:02:40walk into the interview I'm able to talk
9:02:43about in depth things that worked things
9:02:45that didn't work um I'm able to recall
9:02:47those stories really well because I
9:02:50crafted them you know in this in this
9:02:52place that Vier can go and look and and
9:02:56be able to also get that experience um I
9:02:59feel like it's a great exercise to help
9:03:03you you be able to tell those stories
9:03:05easier when you're under under the
9:03:08pressure of an interview yeah okay which
9:03:11I hope you're not at the moment but you
9:03:13know no
9:03:15no all right cool Katie so uh well
9:03:18there's this other thing I wanted to ask
9:03:19you so right now you've been working at
9:03:22IBM as a ex researcher and designer for
9:03:26uh a year you said and so um how did you
9:03:30translate or take some of these exr
9:03:32things you've you youve you've learned
9:03:34and into that job can you tell us a
9:03:36little bit about that yeah so you're
9:03:39right so I'm currently working as a ux
9:03:41designer on a product that is not XR um
9:03:44but I have found a way uh to manage all
9:03:49of my interests in XR while still being
9:03:52a part of IBM uh so I helped uh with a
9:03:56coworker I helped found the IBM reality
9:03:58lab which is an interest group at IBM
9:04:01and kind of like a club where we meet
9:04:04and we uh talk about XR we do
9:04:08development demos we're kind of like a
9:04:10make lab but within the IBM space so uh
9:04:15even though you don't necessarily have a
9:04:17role of XR you can still impact the
9:04:20company you work in um with your
9:04:23interest
9:04:24so I I found you know being able to do
9:04:26that has really helped kind of balance
9:04:29my uh passions at work um and has also
9:04:32opened opportunities for me to be able
9:04:34to do XR on other projects um with other
9:04:38people at IBM which has been really
9:04:39great so are there any last thoughts you
9:04:42know our Learners are like learning all
9:04:45this XR stuff at the moment kind of like
9:04:47the way you did it uh as well obviously
9:04:49you have everybody's unique you have you
9:04:51had a very professional background in in
9:04:53design you could always catch better
9:04:55than me in fact I show some of the
9:04:57things you've done like the Star Trek I
9:04:59show that and I say it's KD so this is
9:05:02actually the KD but yeah so uh you know
9:05:05if a learner ask you uh what shall I do
9:05:08uh do you have some kind of advice for
9:05:10me um should I pursue XR more or should
9:05:14I do something else any any last
9:05:17thoughts you have for our Learners yeah
9:05:20well I mean it seems like you're in the
9:05:21right place so you know
9:05:24the internet has so many resources
9:05:25available and you know right now is like
9:05:28a wonderful time to explore and dive
9:05:30into all these new technologies uh and
9:05:33so I feel like you know just grab a hold
9:05:35of anything that you can find online
9:05:37like this book and you know uh learn as
9:05:40much as you can and and any employer
9:05:43that you go after with you know this
9:05:45educational experience will be uh really
9:05:48excited that they found someone that can
9:05:51teach themselves new skills yeah that's
9:05:54uh actually yeah that's a really good
9:05:56point uh cool all right so well thanks
9:05:59Katie for um spending some time with us
9:06:01today and looking at your portfolio
9:06:04together and uh allowing me to look at
9:06:06the older version rather than the latest
9:06:09so that was that was very nice so so we
9:06:12have a snapshot of Katie in well from a
9:06:15year ago so but we also had the Katie
9:06:17here so thanks Katie again for speaking
9:06:19with us and um yeah hope to see you soon
9:06:22again okay stay stay healthy and
9:06:40safe congratulations you've reached the
9:06:43end of the course the second course is
9:06:46part of the xam the course focused on
9:06:49design and uh well I hope you've enjoyed
9:06:53it
9:06:54I feel like we've covered a lot of
9:06:55ground and I hope you feel like you've
9:06:57learned something um so let's review a
9:07:00little bit of what this course was about
9:07:02and what should be the main
9:07:04takeaways so well we started out with
9:07:08this overview and you have probably
9:07:10finished the four quizzes uh to assess
9:07:12your learning the first quiz focused on
9:07:15some of the what doing XR involves and
9:07:19how to get started and XR process
9:07:23questions
9:07:24so I hope you finished some of the
9:07:25quizzes and learned a lot through doing
9:07:28the quizzes as well but in terms of
9:07:31topics we've covered what doing XR
9:07:33involves how to get started and finding
9:07:36your XR process and then we've uh
9:07:40covered quite a bit on design thinking
9:07:42so really starting out with the original
9:07:44method and then adapting it to XR and
9:07:46then thinking about all the issues
9:07:49around design ethics design guidelines
9:07:52that we did design critiques
9:07:54and we also talked about design jams as
9:07:57a way of collaboratively solving
9:07:59problems uh through participatory or
9:08:02codesign with various stakeholders and
9:08:05so that's like here and that was only
9:08:08the first half of the course and then we
9:08:10focused the second half of the course on
9:08:13physical and digital prototyping and so
9:08:15we started with storyboarding first then
9:08:18created paper then physical prototypes
9:08:21and we went into digital prototyping
9:08:23tools for AR and for VR and also talked
9:08:26about immersive prototyping actually
9:08:27using AR and VR for
9:08:30prototyping so quite a lot of stuff and
9:08:34so with those methods and tools in your
9:08:37toolbox you should feel equipped for
9:08:40xile design uh we haven't talked so much
9:08:42about the actual development so
9:08:44programming with these Technologies but
9:08:46we have really well set our minds uh so
9:08:50that we can actually think about
9:08:52designing and actually doing uh the
9:08:55heavy lifting in terms of prototyping uh
9:08:58already testing with users to some
9:09:00extent and and then prototyping both
9:09:02physically and digitally and so that is
9:09:07really really an important part of the
9:09:09process this is not the entire XR
9:09:11process we obviously still need to
9:09:13implement things and that's where the
9:09:15other courses uh connect to this course
9:09:17but before I get to that let's quickly
9:09:19review uh obviously there was the honors
9:09:22track and maybe you chose it and that
9:09:24means you have completed all these
9:09:27exercises here and these are important
9:09:29assignments
9:09:31so uh from a design critique looking
9:09:34looking at both the guidelines and the
9:09:37ethics uh of XR and then storyboarding
9:09:41your paper prototype your digital
9:09:43prototype and then also if you had the
9:09:46opportunity to do an immersive
9:09:49prototype again as a reminder immersive
9:09:51prototype means you're using actually a
9:09:53and VR Technologies and why you are
9:09:56designing and so it's concurrent design
9:09:59and testing as opposed to what happens a
9:10:01lot in development is like you're
9:10:04programming against these Technologies
9:10:06but on your computer usually a desktop
9:10:09PC and then you deploy and I'll also
9:10:12tell you how to rapidly preview things
9:10:14how to do quick deploys and and how to
9:10:17even avoid this by building on web
9:10:18Technologies in uh some of the other
9:10:20courses as part of the xmook
9:10:23so let us review a little bit of what I
9:10:26think should be the key takeaways from
9:10:28this
9:10:30course so if you remember I started out
9:10:33with making an analogy to film making I
9:10:35talked about these various roles here on
9:10:36the left director Director of
9:10:39Photography uh made the point that
9:10:41actually the person in charge of the
9:10:42camera as the XL user talked about
9:10:45various aspects of design and who other
9:10:47the people involved and then went into
9:10:50some of the more difficult topics where
9:10:51I think uh when it com to creating new
9:10:54kinds of interactions we still need to
9:10:56rely on
9:10:57developers and people who are okay with
9:11:00programming because what you can do in
9:11:02in existing tools is pretty
9:11:04Limited in existing prototyping tools I
9:11:07mean because what you can do in existing
9:11:10prototyping tools is so far pretty
9:11:13limited we talked about these different
9:11:16paths to being an EXA Creator basically
9:11:18depending on your background whether you
9:11:20are into web game or mobile development
9:11:24you should have an easier or more direct
9:11:26path and then if you are on the mobile
9:11:29development side you can also continue
9:11:31developing against and if you have a
9:11:33mobile development background you can
9:11:35continue developing natively against the
9:11:39uh Android and iOS sdks but then also
9:11:42pick up specific uh AR uh versions AR
9:11:46kit and AR core for Apple and Google so
9:11:48for Apple for IOS and
9:11:50Android can develop for arcade and a cor
9:11:53so for IOS and Android and then can also
9:11:56develop natively against some of the
9:11:58virtual reality tool kits and then we
9:12:00also have uh some kind of crossplatform
9:12:04approach through Unity nonre we talked
9:12:06about that which I think speaks
9:12:08relatively nicely to game developers and
9:12:11then also mobile developers who are used
9:12:13to doing crossplatform
9:12:15development uh through phone Gap uh and
9:12:18things like that and then the web and
9:12:21the role of the web uh which I think is
9:12:23going to be even more in the next few
9:12:26years but for now we have a basic stack
9:12:29and set of things we can do in browsers
9:12:31and we are relatively well equipped on
9:12:33the VR side and we're getting better on
9:12:35the AR side as
9:12:38well and then I talked a lot about
9:12:40process need finding brainstorming
9:12:43storyboarding and prototyping and then
9:12:46we talked about development and testing
9:12:47but we didn't cover that much in this
9:12:50course but still part of it's still part
9:12:52of the overall process
9:12:53and uh together with deployment
9:12:55analytics that's something that we'll
9:12:57expand more in the third course as part
9:12:59of the
9:13:00exam so the process was really actually
9:13:04one of the most important things for me
9:13:06to take away from this course obviously
9:13:08we we don't we we were looking a lot
9:13:10obviously we were looking a lot into
9:13:12different kinds of methods for design
9:13:14and then also tools but what is
9:13:16important is that you put the picture
9:13:18together the puzzle pieces fit into a
9:13:20larger process now that process is
9:13:23relatively standard it's just an
9:13:25expansion from interaction design so
9:13:27it's standard if used to user experience
9:13:29interaction design but some of the
9:13:30methods actually need to change and
9:13:32adapt which has to do with the fact that
9:13:34traditional need finding methods for
9:13:35example don't really work because we
9:13:38don't actually have a lot of XR users at
9:13:40this stage and we don't have a lot of
9:13:42problems directly in XR we are more like
9:13:44figuring figuring out which of these
9:13:46problems that we know from the world
9:13:48could actually benefit from a solution
9:13:49in XR and that requires a slightly
9:13:51different approach and so we talked a
9:13:53lot about that we also spend time
9:13:56covering design thinking so going
9:13:58through the six-step method empathize
9:14:00Define id8 prototype test and Implement
9:14:03so we covered all that and again I
9:14:05highlighted some of the things if you
9:14:07remember from this lecture um that we
9:14:10cannot directly expect and translate so
9:14:14because XR is different we need to think
9:14:16about XR and design thinking for XR also
9:14:19a little bit
9:14:20differently we talked about ethics which
9:14:23I think was very very important and I
9:14:26tried to share a few techniques here and
9:14:29an approach a design ethics review of XR
9:14:31applications we did this in a scenario
9:14:33based way uh we looked at issues such as
9:14:36the context and situation the sensory
9:14:38information and data that is being
9:14:40collected and processed actually then
9:14:42the processing pipeline which I think is
9:14:44really important to consider and then
9:14:46also once you have collected the data
9:14:47how do you communicate first of all and
9:14:49get informed consent but then also uh
9:14:51how do you maintain and even future use
9:14:54of this data is a very very important
9:14:55topic so we talked about all these kinds
9:14:57of ethical
9:14:59considerations the one thing I want to
9:15:01say here is that this ethics topic is
9:15:03still evolving it's very important and
9:15:06not enough people actually embrace it in
9:15:08their approach and so for me it was very
9:15:10important that we get this right into
9:15:12the design course that we don't have it
9:15:14like a future issue or some I don't know
9:15:17issue uh afterthought no it has to be
9:15:20incorporated somehow now when you teach
9:15:23this stuff it's difficult to to do both
9:15:26the ethical stuff and the guidelines and
9:15:28then learning about the methods and the
9:15:31tools and and you know elevating the
9:15:34ethics and the ethics standards um but I
9:15:37hope that having a dedicated lecture
9:15:39relatively early in this course raised
9:15:41at least your awareness and encouraged
9:15:43you to think about this more and uh well
9:15:46we will see what what happens in the
9:15:48projects I'll I'll I'll keep monitoring
9:15:50I I keep taking a look I'm curious what
9:15:52kind of projects you are designing as
9:15:54part of the Honest track and to which
9:15:57extent we raise interesting ethical
9:15:59issues we are in this together we are
9:16:01figuring this out together so even if we
9:16:03make a mistake well we can still learn
9:16:06from it so that is really really
9:16:07important and then I covered another
9:16:09important topic to me which is ideation
9:16:11because XR is really the space where we
9:16:13seem to be trying out a lot of things
9:16:15and not always in a very Progressive Way
9:16:18so I tried to cover seven techniques to
9:16:19help you innovate I talked about uh I
9:16:23drew parallels from research um I talked
9:16:26about for example that everything has to
9:16:29start from an idea and that even the
9:16:32prototyping out of that idea that helps
9:16:33you better understand the problem
9:16:35normally probably think about
9:16:37prototyping as learning more about the
9:16:39solution but that's that's that's only
9:16:41partially true it's true but really
9:16:43prototyping to better understand the
9:16:45problem because you can try out these
9:16:46prototypes with users and obviously this
9:16:48is already you're already starting in
9:16:50the design process so so we looked at
9:16:53all these seven techniques ideate
9:16:54Implement inspect iterate a design cycle
9:16:57you know the killer app problem and you
9:16:59know novelty versus creativity they're
9:17:01not the same and I talked about this
9:17:03quite a bit the problem promise premise
9:17:05approach emphasizing obviously the
9:17:08promise of XR Technologies and then also
9:17:09the assumptions that you're making one
9:17:12that is more inspired from research but
9:17:13I think it helps this idea of writing
9:17:15the intro first like before you write
9:17:18the rest of the paper before you do the
9:17:19rest of your project figure out what
9:17:21this project is about what do you
9:17:22contribution is and that way um you you
9:17:26get to motivate the problem better and
9:17:27understand the problem better as well we
9:17:29did talk about the idea hexagon that's
9:17:31really something from rahish resca that
9:17:33came out of one of his talks and I
9:17:36really like it and I wanted to cover it
9:17:38here because I do think it's a it's a
9:17:40cool idea to think about how to take
9:17:42existing ideas further as I said
9:17:44everything starts from an idea and then
9:17:46we talked a little bit more about some
9:17:48of the more interaction design oriented
9:17:50object control design Matrix this idea
9:17:52of rapidly break brain storming
9:17:53different kinds of ideas around objects
9:17:56and how to control them we did the snow
9:17:58boots example or the ice scraper uh that
9:18:02can be shaken example for example so
9:18:05here we had really from the Obscure to
9:18:08the really futuristic kinds of ideas
9:18:11that can come out that way and I think
9:18:13it's an interesting way to innovate and
9:18:16finally we also talked about threshold
9:18:17and seating and Tool design because a
9:18:19lot of us are thinking about creating
9:18:21tools in the XR space and so I think it
9:18:24helps actually to have some metrics uh
9:18:27you know talking about the threshold the
9:18:29barrier to entry and then the ceiling
9:18:31what you can accomplish with that tool
9:18:33um which I think like you really want to
9:18:35set these uh nicely and not everything
9:18:38should be like Unity or unreal which
9:18:41actually both have relatively High
9:18:43barriers to entry which we haven't
9:18:45explored so much in this course but
9:18:47that's what the third course focused on
9:18:48development is really dedicated to so
9:18:51you'll feel the pain that's what I
9:18:54usually
9:18:55say all right and throughout this course
9:18:58I heavily emphasized storyboarding and
9:19:00prototyping so a lot of storyboarding
9:19:02and wireframing physical prototypes and
9:19:05digital
9:19:05prototypes and then I drew out these
9:19:09different quadrants and how do we
9:19:11actually increase the Fidelity so
9:19:14increasing Fidelity and which of these
9:19:16methods are easy and which of these
9:19:17methods are hard and I did this for
9:19:19storyboarding but I talked about
9:19:21traditional paper based storyboard 360
9:19:23story boards a 360 photo story and then
9:19:26also 3D story boarding still which is
9:19:29which is immersive prototyping
9:19:31essentially but you're doing it for this
9:19:33purpose of storyboarding and I think
9:19:35this is one of my favorite new
9:19:37techniques that I definitely want to
9:19:40increase uh the usage and that's one of
9:19:43my new favorite techniques that I
9:19:45definitely want to get better at
9:19:47myself talked about physical prototyping
9:19:49how we can still start from paper but
9:19:51then how we incorp other media like clay
9:19:55like playdoh for 3D modeling and how we
9:19:58become more physical and build a diarama
9:20:00to actually you know create these scenes
9:20:03that you have in mind not just Envision
9:20:05them but actually prototype them out
9:20:07physically and we also talked about a
9:20:09specific set of techniques that
9:20:10developed in my research but also um a
9:20:13lot of practitioners are looking at this
9:20:15which is 360 paper
9:20:19prototyping and then we went into
9:20:21digital prototyping and here I talked
9:20:23about the different techniques
9:20:25distinguished between digital and
9:20:26immersive authoring digital authoring
9:20:28usually happening on a desktop computer
9:20:30it's like desktop publishing and then
9:20:32immersive authoring usually happens in
9:20:34AR or VR so basically on a tablet or on
9:20:37a phone for AR and then usually in a
9:20:40headset for VR and then also specific
9:20:42versions of mixed reality also on a
9:20:44headset we then also talked about some
9:20:47of the higher Fidelity harder to do but
9:20:50really good approaches web-based
9:20:52development crossplatform development
9:20:54and Native development so these
9:20:57approaches do however require
9:21:00programming skills and we will focus and
9:21:03as I said the third course will be
9:21:05dedicated to taking the digital
9:21:07prototyping ideas further and really
9:21:09learning more about development using
9:21:11these three
9:21:15approaches so I think throughout the
9:21:17course I wanted to make it clear how
9:21:19important prototyping really is it does
9:21:22does help us uh ask a lot of questions
9:21:25and find some answers in the process of
9:21:28finding some answers we even create
9:21:30things like we establish requirements
9:21:32for sensing tracking for displays and we
9:21:36produce assets even though they might
9:21:38not be the final assets they might
9:21:40already be early placeholders uh early
9:21:42props that we might use later in in our
9:21:45prototypes and refine throughout and we
9:21:47learn about user needs and skills which
9:21:49is really really important because we
9:21:51can test these prototypes already and
9:21:53that's really also the last topic that I
9:21:55covered which was designing a usability
9:21:58test or user study so we talked a lot
9:22:00about evaluation we talked about what
9:22:02goes into designing these kinds of
9:22:04things and preparing things in terms of
9:22:06goals so what are the goals of your
9:22:07evaluation kinds of tasks who do you
9:22:10recruit how do you recruit what kinds of
9:22:12devices do you want to use any kind of
9:22:14metrics and how do you actually do the
9:22:17analysis so definitely quite a lot of
9:22:20stuff and I feel like you should be well
9:22:22prepared to with this knowledge to
9:22:25approach your next Exile design project
9:22:28again uh I hope you did participate in
9:22:31the honors track and actually chose to
9:22:33do some of the exercises I think you'll
9:22:35learn a lot there as well and what I do
9:22:37now in the last part is just telling you
9:22:40about some of the other courses and how
9:22:42this course relates to the other courses
9:22:44here we go we are in the exam Muk
9:22:47specialization you have just finished
9:22:50the second course on EXA
9:22:53design um and then in the first course
9:22:56which you probably took already but just
9:22:59a reminder um it's really an intro toxr
9:23:02so if you had some issues with some of
9:23:05the terminology in this course then the
9:23:07first course really sets all these
9:23:09Basics and fundamentals we talk about
9:23:11applications Concepts Technologies
9:23:14really how arvr Works behind the scenes
9:23:17what does it actually make a virtual
9:23:18reality and how does this really work
9:23:21and then also the issues
9:23:23and uh there are also Honest track uh
9:23:27exercises and a lot of content so for
9:23:30example you do classify applications
9:23:32along the reality virtuality Continuum
9:23:34so basically figuring out whether an app
9:23:36is AR VR Mr or XR and you should choose
9:23:40the most specific uh class and then we
9:23:43did uh or you would do an exercise on
9:23:46technology selection which I think is
9:23:48really cool going through some kind of
9:23:50XR decision tree uh that I have have a
9:23:53mental basically this XR decision tree
9:23:55is a mental model of all the kinds of XR
9:23:58options you have when it comes to
9:24:00Technologies devices and platforms and
9:24:02then we also talk in depth about the
9:24:04issues the ethical social privacy
9:24:06security accessibility and Equity issues
9:24:10so lots of issues to talk about and
9:24:12there are also exercises focused on that
9:24:15because it's important that you
9:24:16experience these things for yourself now
9:24:18if you have just finished the second
9:24:20course again congratulations and then
9:24:22the third course would focus on
9:24:23development so there you would learn
9:24:26more about development methods and tools
9:24:28you would actually create your first 3D
9:24:29scene so taking something existing from
9:24:32a book or a movie recreating it in 3D
9:24:34and that way learning about uh 3D and
9:24:38making that jump from 2D to 3D really
9:24:39really important and then we are
9:24:41expanding our knowledge into the VR
9:24:43space so we're learning how to create
9:24:45virtual reality scenes basically the
9:24:47environmental design but also things
9:24:49like physics and then obviously
9:24:51interactions
9:24:53and then we talk about AR both Mar based
9:24:55and Mar L AR and how to design for both
9:24:59and in each of these two last courses
9:25:02there will be a project throughout going
9:25:04through these stages and then they will
9:25:07end with a peer review in each of these
9:25:09courses and I think this peer review is
9:25:12really really cool really important you
9:25:14get to hear from other Learners what
9:25:16they think obviously I also chime in and
9:25:18give you some feedback and we do have
9:25:20some ideas in the future for actually
9:25:23coming together in some kind of
9:25:25exhibition there might still be
9:25:26something that we will add to this
9:25:28course and also to some of the other
9:25:30courses now this is a topic that I am
9:25:32very active in terms of research and
9:25:35also teaching so you can expect updates
9:25:38to this course there's always something
9:25:40going on and I will keep refining and
9:25:42revising some of the contents over the
9:25:44next few years and I hope we learn and
9:25:46continue to learn together more about XR
9:25:50so thanks for taking this course again I
9:25:53invite you to look at the other courses
9:25:55it would be great to see you again but
9:25:57otherwise well good luck with wherever
9:25:59you are and I wish you the best of
9:26:14luck welcome back to the Honest track so
9:26:17you are now actually towards the end
9:26:20congratulations from making it that that
9:26:22far and I hope you've been working on
9:26:24these exercises and not just like
9:26:26looking at ahead and uh you really want
9:26:29to see what's happening in digital
9:26:31prototyping well I understand digital
9:26:33prototyping is also very very exciting
9:26:36it can be challenging this is really the
9:26:38part where it becomes tricky it depends
9:26:40on a lot of factors like what kind of
9:26:43equipment do you have available uh is
9:26:45your for example if you have a
9:26:47smartphone is it compatible can you do
9:26:49AR with it um do you have any kind of VR
9:26:53equipment and so I uh I will treat this
9:26:57uh exercise as let's let's just do it
9:27:00broadly as long as we someh move
9:27:03digitally ideally this would work uh
9:27:06your prototype would work on an AR or VR
9:27:08device something that you would call XR
9:27:11device uh that would be perfect okay so
9:27:14that's where we want to get to and um
9:27:17why are we doing this well we are
9:27:20finishing up we are like on track to
9:27:22this minimum viable product the MVP uh
9:27:26our final prototype um it's like with
9:27:29everything in life there's a deadline
9:27:31and then um you know you have to be done
9:27:34and obviously you could iterate and go
9:27:36crazy and expand the main point here is
9:27:39to really just learn about the methods
9:27:41playing around with some of the tools
9:27:43along the way what comes out at the end
9:27:45well if you follow the methods it can't
9:27:48be that bad and um one thing I feel I
9:27:52should emphasize at this stage is we are
9:27:54entering the final stages of the honors
9:27:57track and the project that we are
9:27:58developing as part of the honors track
9:28:00and there will be a peer review so you
9:28:02will you will get to review other
9:28:04people's projects you've already been
9:28:06looking at some of them as part of the
9:28:08activities along the way if you're
9:28:10looking at various examples in the
9:28:12Galleries and but now we're actually
9:28:14formally going to review each other's
9:28:16prototypes not now that's after now
9:28:18we're going to do the digital prototype
9:28:20um but keep that in mind
9:28:22as we kind of like Get Ready um for this
9:28:26final stage of the Honest track so in
9:28:30this uh exercise the task is to create a
9:28:32digital prototype of an Expeditions like
9:28:34XR app okay so you know the rules I'm
9:28:37not going to repeat it we're still
9:28:38keeping with that app and uh you know
9:28:41usually in the residential program there
9:28:43are questions like can I change my
9:28:45project can I work on a different kind
9:28:48of um well solution or problem even and
9:28:51uh I normally say depending on where we
9:28:54are in the project um it's not a good
9:28:57idea so in when it comes to the digital
9:28:58prototype it is not a good idea to now
9:29:02kind of change it given the progress
9:29:04that you've made uh regarding
9:29:05storyboarding wi framing physical
9:29:07prototyping um so I feel like well why
9:29:10not just you know make it through maybe
9:29:13you come up with something cool here and
9:29:14you you like your idea again again as
9:29:18part of this uh of course the whole idea
9:29:20is to just have one new idea we got to
9:29:22mostly focus on what's existing it helps
9:29:25us also do aition prototype like kind of
9:29:27like recreating more or less and that
9:29:30Expeditions like XR app that we're using
9:29:32that we're modeling our work after so
9:29:35your goal is to prototype one of the key
9:29:36interactions of that existing app in a
9:29:40and I'm now going to be very flexible
9:29:42here 360 3D AR VR digital tool so
9:29:47there's a couple of key things here it
9:29:48should be digital okay we we moving away
9:29:50from paper yeah finally getting there
9:29:53and uh we are going to use uh tools
9:29:57there are tools focused on 360
9:30:00prototyping or you could even just do
9:30:02photoshop with a 360 template that would
9:30:04be acceptable um there are 3D uh
9:30:07modeling and authoring tools um and you
9:30:10would be digital but I think so don't do
9:30:13the just 3D modeling um but if you if
9:30:17that's what you want to learn and like
9:30:18you want a tool you want to use a tool
9:30:20like model viewer or sketch Fab that
9:30:23actually bring things into arvr that is
9:30:26also kind of still playing by the rules
9:30:29it's okay I'll show you what I had in
9:30:31mind uh I was working with A-frame
9:30:34actually uh so that required a little
9:30:36bit of development but I was also
9:30:37playing around with immersive authoring
9:30:39tools for VR and for AR so I'll show you
9:30:42what I've
9:30:43done um in the Second Step you should
9:30:45really sketch sculpt or model whatever
9:30:48the language is of the tool that you're
9:30:49using is a digital tool now to to three
9:30:52main characters so this will bring us
9:30:54closer to the story to a digital version
9:30:56of the story in step three I want you to
9:31:00think about that environment that we
9:31:02actually designing for and to which
9:31:03extent should that environment that
9:31:05physical environment should that be
9:31:07represented uh in our digital prototype
9:31:10so I'll say uh make a virtual copy of
9:31:13the target physical environment really
9:31:15capture the context don't just like
9:31:17sketch into like thin air and just focus
9:31:20on the virtual content no and make it
9:31:23clear even from your digital prototype
9:31:25what kind of experience is this supposed
9:31:27to be so in my AR prototypes that I'm
9:31:30doing in VR I'll actually sketch out the
9:31:32table to define the scale of my AR
9:31:34experience so then the next step is to
9:31:36actually record a video and narrate the
9:31:38interactions so I want you to as much as
9:31:42possible uh
9:31:44record your prototype and different
9:31:46tools have different means to do that a
9:31:49screenshot is usually not sufficient
9:31:51since you're focused on interactions a
9:31:53video would be good maybe you uh can
9:31:56learn how to do screen capture on
9:31:59whatever com type of operating system
9:32:01and computer you're using or you're
9:32:02getting one for your mobile phone um so
9:32:06I've used a number of different tools
9:32:08throughout this Muk um and um yeah I'm
9:32:12not here to make really the
9:32:14recommendations uh that's something we
9:32:15can figure out in the Forum but anyway
9:32:17the goal is to record a video and
9:32:18narrate the interactions uh within that
9:32:21digital
9:32:22uh or immersive uh tool in fact
9:32:25optionally I would like you to try out a
9:32:27different digital or immersive authoring
9:32:29tool I make this optional just for
9:32:31workload uh reasons but I do think
9:32:34actually you should you should really
9:32:36try out a few different digital tools so
9:32:38rather than go deep in one and like go
9:32:40crazy I'd rather have you try out two to
9:32:44three quickly and then choose the one
9:32:46that you like and then in this fifth
9:32:48step maybe come back to one of the other
9:32:49ones that you had previously well um not
9:32:52discarded but like not focused on so
9:32:55what do we have at the end of this we
9:32:58should have a lot so we actually then
9:33:00have a final prototype that is an MVP a
9:33:03minimum viable product if you do it
9:33:04right so it's as close as possible to
9:33:06the real thing it might even be able we
9:33:09might even be able to test it with users
9:33:11um in a VR or ar headset so that could
9:33:14be interesting you went broad so you
9:33:18really have this understanding of the
9:33:19environment the 3D characters the
9:33:21interactions you prototype them multiple
9:33:23times in different fatalities now this
9:33:24is the last time you prototype them uh
9:33:27Now using digital tools and you went
9:33:30deep so uh you at this stage you will
9:33:33actually find out about arvr specific
9:33:35requirements and development needs so
9:33:37that is actually quite important so
9:33:39these digital prototypes can really be a
9:33:42function if they're done well function
9:33:44as a specification that you can give to
9:33:46a developer an arvr developer or if you
9:33:49have obviously the skills or if if you
9:33:51want to develop the skills I'll invite
9:33:53you to take the the development oriented
9:33:55xim moo course but um yeah so this is
9:33:59there could be a handoff and this uh if
9:34:01you if you don't do it yourself there
9:34:03could be this handoff and the digital
9:34:04prototype might actually very well
9:34:06provide this template uh for the
9:34:11developers so I'm still doing this solar
9:34:14system I never got bored I've actually
9:34:17been playing around quite a bit it's
9:34:19been fun um so this is is a little bit
9:34:22of you know going back in time when I
9:34:24started out doing the design critique
9:34:26you probably did a different Expedition
9:34:29or Expedition like app anyway let's
9:34:31stick with mine so I had the Prototype
9:34:34uh various prototypes from before so
9:34:36various physical prototypes I had a
9:34:38paper prototype that I was then playing
9:34:40around with I created the 3D models
9:34:43along the way using Play-Doh or some
9:34:46kind of clay and I played around with
9:34:49them so that was really important so
9:34:50playing with these 3D models in the
9:34:52physical world but then also capturing
9:34:54it a little bit digitally and getting
9:34:55some kind of sense for how this might
9:34:58feel in an AR or VR um experience so
9:35:01that was cool and then I uh did this
9:35:04diorama and I was actually playing
9:35:06around with uh more of these objects and
9:35:09composing them together and even using
9:35:12AR captures with my little AR camera and
9:35:16then bringing in uh even still some of
9:35:18the physical objects so really flexible
9:35:20use of both both physical captured
9:35:23virtually materials and then bringing
9:35:25back in physical so um yes I'm just
9:35:28advertising again for the physical
9:35:29prototyping Parts but now we are doing
9:35:31digital prototyping I don't know always
9:35:34get excited when I see physical
9:35:35prototypes but I'm also going to get
9:35:37excited about the digital prototypes if
9:35:39they are done well now the problem is
9:35:41you could actually spend a lot of time
9:35:43and it doesn't look like anything in the
9:35:44end so that is uh something that we
9:35:47should all be aware of so especially
9:35:49when you look at others um each other's
9:35:52Solutions don't judge just because it
9:35:55doesn't feel like wow you didn't do a
9:35:56lot or something um no maybe we
9:35:59struggled maybe we had technical issues
9:36:01maybe we just couldn't commit the time
9:36:03maybe we just wanted to learn
9:36:04superficially about um this process
9:36:08without going too deep like we wanted to
9:36:09have this overview and that's totally
9:36:11fine as well so we're going to do
9:36:12digital authoring we may do immersive
9:36:15authoring so digital as I explained in
9:36:18the lecture is really more like with 3D
9:36:20tools on a desktop and then uh you do
9:36:23previews deploy to device or you somehow
9:36:25stream to a device immersive authoring
9:36:28means you're actually doing it directly
9:36:29on the AI or VR device so I want you if
9:36:32possible if at all possible to try this
9:36:34out and then we can actually think about
9:36:38uh some of these uh platforms and
9:36:41approaches uh that are more development
9:36:43oriented uh we can also think of some of
9:36:46their tools for digital prototyping so
9:36:48A-frame has an inspector and I'm pretty
9:36:51pretty fast with HTML CSS and JavaScript
9:36:54so I think of A-frame as a prototyping
9:36:56tool not everybody may agree and and
9:36:59that's fine but Unity that I put here
9:37:02under crossplatform or unreal these are
9:37:05some of the examples I'll spend more
9:37:07time on later these are acceptable
9:37:11actually relatively High Fidelity even
9:37:14complex um AR VR platforms if you will I
9:37:19mean they have support for them they are
9:37:21traditionally game engines actually and
9:37:24in any case um what I'm aiming for is
9:37:27like we're picking something that maybe
9:37:30not as heavy as uh unity and unreal
9:37:33because in the third course focused on
9:37:34development you can spend more time in
9:37:37these tools so try something so here is
9:37:39me playing around with imersive ar on
9:37:42the iPad and I was using Adobe Arrow
9:37:47which I'm not sure uh whether it's free
9:37:49or not I have an education license
9:37:52and um I can just play with it um it was
9:37:56a little Limited in terms of how I could
9:37:58represent my Planet so I just had to
9:38:00choose from these
9:38:01predefined 3D objects and that's a
9:38:04limitation of the tool so just that just
9:38:06admit not admit it but just mention it
9:38:09if somebody is like hey you didn't do a
9:38:11proper sun earth or Moon well this is
9:38:15what I had to work with also working
9:38:17with these more abstract representations
9:38:19is fine as long as it helps you you
9:38:21learn something about for example the
9:38:24composition uh the scale of the
9:38:26experience or just like this mere fact
9:38:28of being in AR and doing an immersive
9:38:31authoring uh session is is just cool and
9:38:34the main thing is I want you to try it
9:38:36out and um well so I'm doing it with
9:38:39arrow here and then later in the video
9:38:41that I'm going to share with you I'll
9:38:43also play with reality composer and so
9:38:47reality composer obviously apple on the
9:38:50iPad um at least I could put in a few
9:38:54primitive shapes in there and it's just
9:38:57good enough for planets I really like
9:38:58that I chose the solar system and
9:39:00nothing more complicated this was hard
9:39:02enough okay so um don't make it too too
9:39:06difficult on yourselves um you just want
9:39:08to learn about the methods and the tools
9:39:10and um you can work on the complicated
9:39:13project that you've always wanted to
9:39:14solve and do you can do that later it
9:39:17doesn't have to be in this course H go
9:39:19through this course take everything uh
9:39:22with you along the way you know build
9:39:24your toolbox and and then uh apply it to
9:39:27your uh own project and if you do please
9:39:31let me know it would be cool if any of
9:39:34this was helpful in whatever you wanted
9:39:35to do um that would be that would make
9:39:38me proud in any case I was then also
9:39:41playing around with the immersive VR uh
9:39:43authoring tool here I'm using
9:39:46blocks and you know funny enough because
9:39:50I'm doing a solar system and most tools
9:39:52actually have these relatively basic
9:39:53shapes in there H you you just can play
9:39:57around here I have color so that was
9:39:59cool and um later in the video I'm going
9:40:01to go a little bit forward I'm grabbing
9:40:04the objects and playing around with them
9:40:07and trying out different compositions
9:40:10um and I hope I'm just taking the moon
9:40:13maybe we missed that part I like my
9:40:15little hand there it's really quite cute
9:40:18and zooming in and zooming out okay I'll
9:40:20show all that obviously but here
9:40:22maybe yeah this is the part that I
9:40:24wanted you to see and you check out the
9:40:26video later in more depth if you want
9:40:28but this is why it's cool because now in
9:40:30immersive authoring I can actually play
9:40:32around using my tiny little blocks hand
9:40:35there uh can't get over it and um you
9:40:38know do it uh experiment and play and
9:40:43that's uh well this you can't say that
9:40:46this is a final experience okay so I'm
9:40:48just doing immersive authoring uh it'll
9:40:51get a little better I'll I'll I'll I'll
9:40:53increase the Fidelity by one more bit um
9:40:56whatever the unit is and here is uh our
9:40:59one more bit so I'm showing you U and
9:41:02I'm going to share this whole video with
9:41:03you as well um proudly present to you
9:41:06our A-frame based um so web XR
9:41:11based solar system now this is something
9:41:15that uh I started and then I had a few
9:41:19students also contribute to it it it's
9:41:21also the kind of exercise and example
9:41:24code I will share with you and it's a
9:41:27also something that I share with my
9:41:28residential Learners and it's um you
9:41:31know it's not too complicated a few
9:41:34spheres in there they all textured
9:41:35nicely so we got the right NASA pictures
9:41:38public domain of the planets and then we
9:41:41added a little bit of interactions here
9:41:44and some text uh text doesn't work so
9:41:47well in AR or VR but whatever you know
9:41:48we're trying that out that's this is not
9:41:51supposed to be the final Design This is
9:41:52where you feel like okay if I write a
9:41:55lot of text above the planets that may
9:41:57not be the best experience um and and
9:42:00that's that's what this is for so that's
9:42:02okay um I'm going to jump ahead because
9:42:05I also later run it in AR oops that was
9:42:08like right in there and it is directly
9:42:11here in the studio and I was just like
9:42:14you know also talking a little about the
9:42:17tracking and how the tracking doesn't
9:42:18work so well sometimes but um this was
9:42:21pretty cool and um yeah so if you feel
9:42:25like it at the end I'll also show
9:42:27cardboard uh it feels like a step back
9:42:30but this is my target device if you want
9:42:32so I didn't do too much for the Oculus
9:42:35Rift I mean teleporting in space seems
9:42:37like an interesting concept um tele we
9:42:41are teleporting I know that in Star Trek
9:42:43they have so I I I know about Star Trek
9:42:47don't worry I know that they have
9:42:48beaming there but no teleporting in
9:42:50space just doesn't feel right uh but um
9:42:54I wanted to think about different
9:42:55Locomotion techniques again this is just
9:42:59our digital prototype so again when it
9:43:01comes to digital prototyping keep in
9:43:03mind we can also create multiple
9:43:05versions of our digital prototypes you
9:43:06don't have to you know cover everything
9:43:09that you had in mind I want you to be
9:43:10selective I said one key interaction and
9:43:13um yeah it should be important to your
9:43:16project not just the login or whatever
9:43:18uh no something key and you can do it in
9:43:21two versions and maybe add more to it um
9:43:24you don't have to go all the way I mean
9:43:26ideally we would get there at the end we
9:43:28would actually have our MVP but you know
9:43:30that doesn't have to be the scope of how
9:43:32you go through this exercise uh you can
9:43:34work with one tool and if it's too
9:43:36limited or you feel like you want to add
9:43:37a second tool you don't have to
9:43:39prototype the same thing with both tools
9:43:41I'll talk about prototyping strategies
9:43:42just as a reminder in a second but you
9:43:44could also just fit in another tool
9:43:46because it allows you to just create
9:43:47that 3D model that you need and then you
9:43:48can import it into that other authoring
9:43:50tool and so that would be cool too so
9:43:53here are the digital prototyping
9:43:54strategies I talked about in the lecture
9:43:57um so you can do uh parallel prototyping
9:44:00that's especially when you work in teams
9:44:02is very effective but even just you
9:44:04prototyping the same feature with
9:44:05different tools so if you like don't
9:44:09know which tools to use yet um you could
9:44:12uh choose a very simple feature or a
9:44:14particularly hard one and uh prototype
9:44:17it just a little with a few different
9:44:19tools uh you could prototype different
9:44:21different features with the same tool um
9:44:23so that's iterative prototyping because
9:44:24ultimately you create this entire set of
9:44:27features in one tool and that may
9:44:29actually then be your prototype um you
9:44:32could do the divide and conquer and
9:44:33actually prototype different features
9:44:35and different tools now now for this
9:44:38activity I would recommend uh strategy
9:44:41one or two probably go with um just
9:44:45iterative and just use one tool but then
9:44:48in a an optional fifth step I want you
9:44:51to try out a different um uh tool you
9:44:55don't have to you know prototype the
9:44:57same thing again so it could be that you
9:44:59parallel prototype or it could be that
9:45:01you divide and conquer at that stage if
9:45:02you make it to that fifth uh step in the
9:45:05in the in the current
9:45:07task so what that would mean is uh if
9:45:11you have followed through in the honors
9:45:14track you have done your design critique
9:45:16your storyboarding physical prototypes
9:45:18and now actually a digital prototype or
9:45:21with an immersive tool um with a digital
9:45:24or an immersive tool and um so that
9:45:27means you have some subset of all of
9:45:29these
9:45:30deliverables and I say deliverables
9:45:32because uh at the end of this um we will
9:45:35actually do a peer review um and that is
9:45:38something where both you will actually
9:45:41get feedback but also provide feedback
9:45:43to others and um so this peer review
9:45:48activity is uh requires a little bit
9:45:50more more explanation and um so I'll
9:45:53have a more information actually in
9:45:55depth in writing for you don't worry so
9:45:58much about the grading we're still here
9:46:00to learn and but we do need to establish
9:46:04some basic rules so that we treat each
9:46:06other fairly and with respect which is
9:46:08really important to me so I'll talk
9:46:10about that just getting ready for that
9:46:11part as you can tell no um at this stage
9:46:15um don't think too much about the peer
9:46:17review I just wanted to mention it I I
9:46:19didn't want to bother you before but you
9:46:21saw it in the syllabus it was mentioned
9:46:23um but it's not the main thing uh we
9:46:25don't do this to be graded we we do this
9:46:28to learn and so in this last step in the
9:46:32Honest track um the digital prototype I
9:46:35think you learn a lot um don't um don't
9:46:38just keep these other steps very short
9:46:40and then spend like forever on the
9:46:42digital prototypes or um spend too much
9:46:46time and running out of energy on these
9:46:47lower fality ones and then don't do
9:46:50anything almost
9:46:51no all of these are important um none of
9:46:54these replace each other they're all
9:46:56here to learn um I wouldn't skip too
9:46:58many of these steps a I wouldn't do
9:47:02every single type of storyboard or type
9:47:05of physical prototype or you know using
9:47:08all kinds of tools when it comes to the
9:47:10digital prototype I wouldn't do that in
9:47:13a project I usually have at this stage I
9:47:14have developed an intuition which which
9:47:16tool to use or I'm just stuck and do
9:47:18everything with A-frame or Unity just
9:47:21because I feel like I'm fast um which is
9:47:24not always true okay so don't get me
9:47:26wrong there but so the point of this was
9:47:29that we are developing this picture book
9:47:31style um you know approach to XR
9:47:36prototyping uh doing it by the book is
9:47:39fine first and then later you can cut
9:47:42corners but don't cut corners and not
9:47:44understand the whole thing that would be
9:47:47a bit um you know wouldn't wouldn't
9:47:49provide the right foundation so that's
9:47:51why we're doing it this way and I hope
9:47:53you will have some fun in this uh final
9:47:55week of our digital prototyping
9:47:58activities and uh I'm looking forward to
9:48:00seeing uh a lot of the prototypes coming
9:48:03together from this course and I'll see
9:48:05you again for the peer review
9:48:20again welcome to the honors track this
9:48:22is the Exile design critique this is the
9:48:25exercise that is going to get us started
9:48:27as part of the honors track so uh for
9:48:30every exercise I'll provide a little
9:48:32introduction video and The Specific
9:48:35Instructions regarding the exercise will
9:48:37be shared through uh the moo uh the
9:48:40readings that we have but this video
9:48:43provides an overview and talks about
9:48:45some of the main steps and why this
9:48:48exercise actually exists so design
9:48:51critiques are very important I'll talk
9:48:53about that in its dedicated lecture uh
9:48:55it's a ritual in fact in my course when
9:48:58I teaches residentially every student
9:49:00has to critique an existing arvr app and
9:49:03it's really cool because when you bring
9:49:04all bring together all these students
9:49:06and they critique these existing apps
9:49:09everybody learns about existing apps and
9:49:11also we practice our skills of doing a
9:49:14thoughtful productive uh but also with
9:49:18constructive criticism uh a rich
9:49:20critique uh something that we can learn
9:49:23from and so that's what we're trying to
9:49:25exercise here so I'm going to walk you
9:49:27through the main steps of this
9:49:29exercise so the task should you choose
9:49:32to accept it is to critique an existing
9:49:35Expedition like XR app in 360 VR or ar
9:49:40so I say um this so that we have maximum
9:49:43flexibility you don't necessarily have
9:49:45to have VR or ar you can just do it on a
9:49:49smartphone with 3 60 so and I say
9:49:52Expedition like XR app uh I'm thinking
9:49:54Google Expedition and I'll tell you a
9:49:56little bit more about Google expedition
9:49:57in a second but also it could be your
9:50:01favorite museum if it has an app or
9:50:03something for the city or something that
9:50:06feels like you can go on a tour um I
9:50:09want it to be not too complex an app but
9:50:11the activity is really to critique it
9:50:14and if it's too complex might probably
9:50:16be one of the main criticisms so what
9:50:19I'm thinking in the task should be to
9:50:21record yourself describing the XR
9:50:23experience using thinkaloud it's very
9:50:25important so the thinkaloud part is
9:50:28really important yeah you should you
9:50:29should play around with it and then you
9:50:31play around with it one more time and
9:50:32then you record uh what you're thinking
9:50:35even your most often the first time
9:50:37impressions are really the ones that are
9:50:39really really cool to capture in this
9:50:40think aloud but maybe we're not used to
9:50:43this idea of thinking aloud as we using
9:50:46something and that's something you have
9:50:48to practice and um so and I'll show you
9:50:51an example of me doing a thinkaloud
9:50:54while playing around with an existing
9:50:56Google Expedition uh I'll chose the
9:50:59solar uh system
9:51:01one I want you to describe the context
9:51:04of your XI experience so I want you to
9:51:06well basically share uh when you share
9:51:08the results of your critique uh whether
9:51:10you have operated this app in 360 in VR
9:51:12in AR have you been in a private or
9:51:15public setting um so if you were uh you
9:51:18don't have to be too specific but I
9:51:20think it may have impact on how you
9:51:22critique the app maybe it was not
9:51:24designed for the context in which you
9:51:25used it and so that's then uh something
9:51:28that we could also discuss as well I
9:51:31want you to critique the Expedition
9:51:33using the template the I like I wish
9:51:35what if template okay so I introduced it
9:51:38in the lecture and I'm going to briefly
9:51:41mention it again in the next few slides
9:51:43I want you to analyze the expedition in
9:51:45terms of guidelines so really design
9:51:47guidelines first and I provided some
9:51:49examples of guidelines I mean as you
9:51:51know from my lecture I'm never really
9:51:53happy with all the guidelines we have we
9:51:54don't have enough guidelines and those
9:51:56we have are not good enough so um anyway
9:52:00this a starting point I want you to also
9:52:03articulate ethical social or privacy
9:52:05concerns one might have I mean you you
9:52:07might have them you might not have them
9:52:10uh but users could have them and so I
9:52:12want you to think about that so the
9:52:14expected results of this activity and
9:52:17this exercise are that you have now
9:52:19provided a balance critique of positive
9:52:22and negative design aspects so really
9:52:24start with a positive keep
9:52:27remember you have developed initial
9:52:29ideas for how to change and improve the
9:52:30design that will be very helpful for you
9:52:33as you continuing to work on this and go
9:52:35into the next stage which is
9:52:36storyboarding and as part of
9:52:38storyboarding we do brainstorming and
9:52:41you also have a better sense of what to
9:52:42look for in terms of design guidelines
9:52:46so that would make me really happy if we
9:52:48achieved all these if these were really
9:52:50truly really truly the the results so
9:52:53that would be cool um but uh that's what
9:52:55I'm hoping we get out of this and if you
9:52:58practice critiques and uh you really
9:53:01think broadly about some of the concerns
9:53:03then I think this will be very valuable
9:53:06and uh might be a lot of fun to also
9:53:08discuss with your peer Learners here so
9:53:10please we'll have forums available and
9:53:12we can
9:53:14chat so I said the expedition that I had
9:53:17in mind um the actual app is Google
9:53:19Expedition so now Google Expeditions
9:53:21comes in VR and in AR and it's actually
9:53:24popular in classroom environments so
9:53:26Google is targeting education scenarios
9:53:29and I thought this is a good app for us
9:53:31but I say Expedition like um it could be
9:53:34really anything but I don't want it to
9:53:36be too complicated ideally it's publicly
9:53:39available so that others can actually
9:53:41also try it out so you should be clear
9:53:44um how you obtained uh the app if it
9:53:47wasn't Google
9:53:49expeditions so here's an example of me
9:53:51running the world cities Expedition and
9:53:54I've covered that in the lecture and
9:53:57here is a video of me trying it out um
9:54:01actually here in the studio the solar
9:54:04system uh app all right I'm going to do
9:54:07a design crit an exra design crit of the
9:54:12Google Expeditions uh the exploring the
9:54:15solar
9:54:16system and uh we're going to start it in
9:54:19AR so initially
9:54:21um I think this is this works quite well
9:54:24detects the surface surface okay and
9:54:27then the the problem is that the planet
9:54:31is like really really big that we're
9:54:33getting and any attempt to resize
9:54:38doesn't work I can just raise it but I
9:54:42cannot I have not figured out a way to
9:54:45resize or
9:54:47rotate and that is actually something
9:54:49that I would normally expect from an
9:54:53from an app then you can learn more
9:54:56about the Sun and what we're going to do
9:55:01instead is we're going to look at Earth
9:55:03so Earth is obviously smaller than uh
9:55:05the Sun but it's still definitely
9:55:08definitely definitely quite big and um
9:55:12same issue but I do like the inner
9:55:14planet so that's like kind of cool when
9:55:17they then appear above the keyboard here
9:55:20so there's
9:55:22Earth and we can look at some of the
9:55:27detail maybe even
9:55:31find parts
9:55:34um well maybe you can find where you
9:55:38live and um that could be
9:55:42interesting and tracking uh works it's
9:55:44tracking is really robust obviously so
9:55:46that's kind of nice and then the moon
9:55:49again a monster
9:55:52boom and it's just
9:55:56massive and we have the outer
9:56:00planet so that's also pretty
9:56:04cool and so it would be nice to have
9:56:07labels on
9:56:09these I really like the look of the
9:56:11planets but like having labels would be
9:56:13very
9:56:14useful and so that you're actually
9:56:17learning something I mean right now it's
9:56:18kind of cool the asteroid belt it kind
9:56:22of looks cool and nice the dwarf planets
9:56:25but then you would really want to learn
9:56:26a little bit more
9:56:29and um the interactions here are
9:56:33definitely pretty
9:56:34limited um it does look cool though um I
9:56:38must admit that so I would say I like
9:56:42the look of the planets uh that's pretty
9:56:46cool and I like how
9:56:51um I especially like the inner planets
9:56:55and I like how they have breaken down
9:56:58the solar
9:56:59system um although it's harder for me to
9:57:02build the relationships between the
9:57:04different kinds of views that I have on
9:57:05the different planets and the asterid
9:57:08belt and the moon normally obviously
9:57:10going around Earth so would have been
9:57:12nice to see that um I wished I were able
9:57:17to rescale resize
9:57:21um so this I just don't understand why I
9:57:24would want to raise it like crazy it
9:57:28does make sense when I put it
9:57:30on when I relocate it but I also can't
9:57:33relocate it so I don't know how to do
9:57:37that can send feedback
9:57:41um yeah so what
9:57:45if you
9:57:47were allowing me to scale it once
9:57:50ones and then also like I think the
9:57:53other thing is the distance when I then
9:57:56switch I'm sure it's going to just be
9:57:59again oh it does it does keep in mind
9:58:02what I wanted in terms of distance oh it
9:58:05does it does not rotate and
9:58:11scale um so that's really my biggest
9:58:14complaint
9:58:16here and so here is what you just saw
9:58:19and at least a subset of it and uh these
9:58:23are all the screens so this is the
9:58:25download screen obviously then you enter
9:58:27it you place it I didn't have a
9:58:29screenshot for the placing part and then
9:58:31once you have placed it you have like
9:58:32this card interface down there and uh
9:58:35this Edge is really really key it
9:58:37suggests that there is more so you swipe
9:58:39to the left and then you get to see
9:58:41Earth you can swipe up and you get to
9:58:43see some of the information about Earth
9:58:46and then you swipe to the left again you
9:58:48bring up this screen and so forth
9:58:51and uh this is how the Expedition works
9:58:54and I want you to critique uh the
9:58:56Expedition app the Expedition like app
9:58:58that you chose I wanted to use the I
9:59:00like I wish what if template so what you
9:59:03like about it important to start with
9:59:05something positive and there should be
9:59:07two positives okay uh what could be
9:59:09improved and here really try to be
9:59:11specific try to be objective uh not
9:59:14subjective and uh what could be approved
9:59:17as improved as one thing but how would
9:59:19you improve it how could it be improved
9:59:21this is really the hard part this is
9:59:23what we're after here this will help us
9:59:25develop some new ideas as we are going
9:59:28to improve upon the
9:59:30app so here are some examples the look
9:59:33of the planets that's pretty cool I
9:59:35think that's something I said in the
9:59:36video how they have broken down the
9:59:38solar system is something that I liked I
9:59:40could resize and rotate the planets I
9:59:42really wished I could and I could better
9:59:44understand the relationships between
9:59:45planets it was really just not enough uh
9:59:47like the scale was really unclear here I
9:59:50could adjust the scale and rotation
9:59:52using pinch to zoom yeah it was very
9:59:53confusing to me I remember um the going
9:59:56up and down part um that was weird like
10:00:00why does it only change around the Y and
10:00:03um so how would I improve it um well
10:00:05using pinch to zoom for example which is
10:00:07a very common gesture and how could I
10:00:10adjust the distance to the planets and
10:00:12then the other idea is to adjust the
10:00:14distance to the plane using drag and
10:00:16drop so that was referring to the fact
10:00:19that once the expedition was placed I
10:00:21basically had to restart the app I
10:00:23couldn't figure out how to move it to a
10:00:25different plane maybe it is somewhere in
10:00:27the app and I just didn't notice but
10:00:29then we need to improve upon that aspect
10:00:32that is called bad usability if people
10:00:35cannot discover what you can do with the
10:00:36app then yeah you have a problem so this
10:00:40was a review based on the design
10:00:42guidelines using the I like I wish what
10:00:44if template the next thing I want you to
10:00:46do is also to perform an Ethics review
10:00:48and I was talking about design ethics
10:00:50review as part of the XR design ethics
10:00:53lecture and I was describing these four
10:00:56main steps or considerations as part of
10:00:59the review so the context and situation
10:01:02the sensory information and data the
10:01:03processing pipeline as much as you
10:01:05understand it I mean clearly you will
10:01:07not have developed the app but um maybe
10:01:09you are developing intuition for what
10:01:10goes on behind the scenes or the app
10:01:12informs you and this informing part in
10:01:15terms of uh terms and what you consent
10:01:17to um this is also really key that we
10:01:20want to look at for data ownership and
10:01:22governance so in my example that would
10:01:25be that I was using the AR version uh in
10:01:28the studio here I was actually the only
10:01:31user in the device vicinity I mean my
10:01:33bunny was probably in the background but
10:01:36uh we don't know uh the sensory
10:01:39information well it sees mostly me it
10:01:40sees the furniture it does see expensive
10:01:43equipment here uh keep in mind that this
10:01:45equipment is on loan from the University
10:01:47of Michigan so this is not my own
10:01:49equipment
10:01:50um the surface area of the table and the
10:01:53floor um so alog together um you can
10:01:58obviously uh you can actually make some
10:02:00sense of this information both for
10:02:03tracking and registration but then also
10:02:06um maybe in a negative way and so that
10:02:08goes more under the ethics part
10:02:09obviously um the processing pipelines I
10:02:12think it's a pretty naive application it
10:02:14does standard surface detection doesn't
10:02:16do a lot with spatial mesh in fact it
10:02:19doesn't have a spatial mesh there's no
10:02:20scene understanding uh it may build
10:02:22Anonymous usage statistics I don't know
10:02:24about that but uh it keeps a library it
10:02:26keeps track of the the different kinds
10:02:29of expeditions I was downloading there
10:02:31is a teacher mode which I think is a bit
10:02:34more advanced and multi-user and that
10:02:37has potentially a more complex
10:02:39processing Pipeline and actually will
10:02:40raise additional
10:02:42concerns uh Expeditions app manages the
10:02:44downloads as I was just talking about
10:02:46this the SL tab so that's the kind of
10:02:48information that I know
10:02:50um I don't know what they collect and I
10:02:51also don't know because these
10:02:53Expeditions are actually developed
10:02:54independently so there are some Google
10:02:56ones but there are also ones developed
10:02:58by others H it would be interesting to
10:03:00understand that Expeditions framework a
10:03:02little bit better and understand what
10:03:04kind of data is actually being collected
10:03:05and shared with those uh third-party
10:03:09vendors that's it right so we're going
10:03:13to learn a lot about ethics um by
10:03:16actually thinking it through but we're
10:03:18also going to do a really a design
10:03:20critique uh more broadly thinking
10:03:22obviously also about the social and
10:03:24ethical concerns as I was just talking
10:03:26about but also thinking a lot about the
10:03:28design and design guidelines and so pay
10:03:30attention to whether you're doing an AR
10:03:32or VR Expedition or something in between
10:03:34like 360 share that with us and fellow
10:03:37Learners so that we actually understand
10:03:39your critique better uh let us know how
10:03:42and where you used it and what you found
10:03:44confusing and uh well we should get a
10:03:47first indication of the kinds of things
10:03:49you think you would be improving as you
10:03:51keep working in this Honest track on the
10:03:54next steps which would then be
10:03:55storyboarding and wireframing we need
10:03:58brainstorming new ideas and flashing
10:04:00them out

About this transcript

This page contains the full transcript of USER EXPERIENCE INTERACTION DESIGN FOR AR / VR / MR / XR | MODULE -2 by LEARN FROM EXPERT, generated from the public captions YouTube serves with the video. The transcript has 103,689 words across 14,653 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to translate the transcript into 100+ languages.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, translation to 100+ languages, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.