A Glimpse of the Future of Augmented Reality
What do we really know about augmented reality? What is the difference between augmented and mixed realities? How can augmented reality improve our daily lives? Find the answers in the article below and learn more about the most innovative AR-applications developed by FCS students in the Interaction Design course.
Theory and examples
Imagine that you arranged to meet a friend at the city square, but you have forgotten to agree on a particular place. Yes, you can write to the friend on arrival and specify the spot, but searching for each other in a crowded area requires some effort from both of you. This is where Neon comes in handy.
Neon AR, Inc. / neon.fyi
This particular app allows you to find a person by using annotations overlaid on top of the image obtained from one's phone camera, providing that the person has granted you access. It is called augmented reality, or AR for short. Amazing things are possible with AR, for instance, less distracting in-car navigation, smart choice of goods in a supermarket... Even tasks such as cooking and choosing dishes in a restaurant can be made easier using AR.
Many people believe that the term "augmented reality" implies that a user sees annotations superimposed into his or her field of view, and that the real environment in which he or she experiences AR is not taken into account (watch this video), while mixed reality implies that the environment is mapped and directly taken into account, and virtual objects the user sees are attached to real things, i.e. they live within the environment. This is a common misconception. Paul Milgram and Fumio Kishino, who back in 1994 coined the term "mixed reality," defined AR as "any case in which an otherwise real environment is 'augmented' by means of virtual (computer graphic) objects." It’s worth mentioning that they considered AR apps to be apps where the amount of augmentation is not that great, i.e., where the real world is not blocked. Mixed reality, in their opinion, is a more general term that includes AR and implies merging of real and virtual worlds to varying degrees. Thus, you are free to use both terms as synonyms.
Right now, everyone who has a modern smartphone or tablet can experience AR. Take Yandex.Maps as an example, whose iOS version has an AR mode for pedestrian navigation:
Yandex / goo.gl/LvWoA1
However, you do not need to have an iPhone or iPad to experience AR. Modern technologies allow you to do it through a web browser installed on your smartphone. If you are sitting in front of your PC or Mac, just open this image in a new tab. Then open this web page (goo.gl/gJz5q2) on your smartphone, allow the page access to the smartphone camera, and point the camera at a monitor of your computer. Voila! You should see this:
Jerome Etienne / goo.gl/c4wBbW
It works on most modern smartphones based on Android, iOS (starting from 11th version), and Windows. If it doesn't work, try updating your mobile browser.
However, this way of experiencing AR has obvious limitations. Firstly, holding a phone right in front of you for a prolonged period of time is not that convenient. Secondly, seeing the world through a small screen is not that immersive. Thirdly, all interactions with the content presented by an AR app are still happening on the screen. There is a way to experience AR without using your hands and still be surrounded by augmentations placed in the world: AR glasses. Let's look at digital eyewear developed by DAQRI, which is designed to be operated by workers on factory floors:
Not all AR headsets are huge. For instance, ODG R8 glasses provide similar capabilities in a more compact form. Many modern headsets, such as Meta 2, the upcoming Magic Leap One and a DIY-headset North Star from Leap Motion allow a user to interact with virtual objects and menus with their bare hands:
Some people predict that AR glasses will substitute mobile phones. Tim Sweeney and Mark Zuckerberg are among them. Many tech giants are investing in this kind of technology. Google does it indirectly by having a share in Magic Leap. Microsoft is working on their HoloLens successor. Rumour has it that Apple is also working on AR glasses. Of course, AR glasses will not take over instantly. It will be a long journey from the current state of the art tech to the future, such as this one proposed in 2016 by Keiichi Matsuda in his video called Hyper-Reality:
It might be a little scary, but looking at the world depicted in the video, it is possible to imagine the future that awaits us. To be prepared for what's coming, here at the Faculty of Computer Science Higher School of Economics, we have already started teaching how to design for AR. The course, called "Interaction Design", taught by Pavel Manakhov, is intended to introduce students to the process of designing unconventional user interfaces. Designing for AR is in the spotlight. However, the course also includes topics such as product design, user research, usability evaluation, visual design, and design of tangible, conversational and multi-modal interfaces. Some of these topics are covered by the course companion on Coursera, in which you can enroll right now.
FCS students projects
This year, students in the course worked in teams on app concepts for AR glasses. They were defined as something in between Magic Leap One and Leap Motion's North Star, but much more compact. Towards the end of the course, they successfully managed to produce a design, and some of them even implemented them in the form of an interactive prototype.
One of the teams was designing an app that helps with furniture assembly. The app provides the user with guidance, tracking all parts of the piece of furniture.
The AR glasses do not in any way limit a person’s movements. The person does not need to remember the assembly sequence. Since it is possible to scan the QR code on the box, the app does not even require that the person choose a piece of furniture from the catalog before assembly.
Another team of students was working on a design of an app that allows you to make a shopping list using voice commands at home, and then it efficiently navigates a person along the shelves in a supermarket, picking up products according to the personal preferences.
An important feature of the app is that it helps to avoid buying products which a person is allergic to.
The next team proposed a design of an app for bikers. The app was designed specifically to help them to navigate through cities. It has fascinating features that allow bikers to move the map with one hand using the "grab" gesture. The "okay" gesture is used to center the map on the place where the biker is.
When a biker is choosing the route to a destination point, it’s enough just to just nod their head to confirm the default setting.
Additional functions are located on the biker's hand and can be triggered by bending a corresponding finger:
The app can also be operated by voice.
The team consisting of Mikhail Tabak, Nikolay Priakhin, Vladimir Miller and Nikita Konstantinovsky designed an app for visitors of a restaurant that provides access to the virtual menu and also enables the diner to build a custom burger. The members of the team implemented a prototype for Android phones using existing software such as Unity and Vuforia.
As you can see, AR tech opens up new possibilities. It promises to make people's lives simpler. With time, AR headsets will become cheaper and more accessible. If you work in the tech industry or are planning to work in it, you should be ready for the widespread use of AR.
Where to Learn More
If you are as fascinated by augmented reality as we are, check out these material recommended by Pavel Manakhov:
Keiichi Matsuda's design-fiction concept film named "Augmented City 3D"
https://vimeo.com/14294054
Albert Hwang's introduction to Spatial computing
http://www.albert-hwang.com/spatial-computing
Milgram and Kishino's "A Taxonomy of Mixed Reality Visual Displays"
https://cs.gmu.edu/~zduric/cs499/Readings/r76JBo-Milgram_IEICE_1994.pdf
Doug Bowman's "3D User Interfaces"
http://www.interaction-design.org/encyclopedia/3d_user_interfaces.html