Mastering AR: App for Microsoft Hololens

Some time ago, we started investigating AR/MR technology and the devices that bring this technology alive. The first device was Sony SmartEyeglass, which is a prominent representative of augmented reality crop. Continuing with our investigation story, we would like to talk about a mixed reality headset, Microsoft Hololens.

Microsoft HoloLens Overview

Microsoft HoloLens is a long-awaited device, since it introduces a new degree of AR/MR devices maturity. Most of other virtual, augmented, and mixed reality devices are based on stereoscopic image projection. Therefore, they have considerable limitations in the field of vision (Google Glass) or need ample space between projection displays and eyes (Oculus Rift, Gear VR, Google Cardboard, etc). The first presents a great restriction in the device application, while the second is altogether unacceptable for AR solutions because of structural bulkiness.

The maturity and difference of Microsoft HoloLens lies in its ability to integrate stereoscopic 3D graphics with real objects in the surroundings and make it feel natural due to accurate real-time digitizing of the physical world and dynamic building of the depth map. To achieve this, HoloLens applies a path breaking technical solution: advanced hardware-accelerated spatial sensing technology allows it to “understand” how the world around looks like, while spatial sound imitates the presence of the digital reality.

There is a specialty in rendering. Unlike VR headsets, which just display a scene created on an external PC, HoloLens performs rendering and gesture processing itself. It means that you cannot rely on the processing power of your work station, HoloLens renders graphics using its own capacity only. The benefit is that the device is self-sufficient, but, on the other side, HoloLens graphic system cannot be compared to those of desktop PCs and cannot be physically updated or replaced. So, I think, we can forget about complex scenes and super-graphics.

The App We Developed

During the Media Hack Weekend, we’ve decided to demonstrate possible applications of this device. Our team came up with an idea of a solution for designers, architects, and event organizers. The solution would measure the existing premises and allow users to place 3D models of interior items in real size. The ability to move, recolor, and otherwise modify 3D models would provide for making immediate adjustments in interior design or furniture arrangements if required.

We have created a prototype using Unity and then have integrated the project into Visual Studio. To create holographic representation of interior items, we applied Unity 3D. We were lucky to have the physical device and could test our prototype under normal usage conditions and get first-hand user experience. If you are not that lucky, you can get emulator online, and give it a try without ordering the device. Sad to say, emulator for HoloLens leaves much to be desired, in the first place, in terms of computing speed. So, it is hardly possible to make a functional app using just an emulator.

Plus, in the opinion of my colleague, Nikolay Mozgovoy, the development of apps for HoloLens using Unity is not very handy. And he has a reason to say so, because Windows 10 SDK wouldn’t work at his laptop running Windows 8.1.

Anyway, if you still want to have a go and happen to have all compatible software and hardware, the whole process is as follows:

• You create and edit a project in Unity 3D;
• Export in for VR;
• Open this project in Visual Studio 2015 Update 3 with Windows 10 SDK (version 1511 or later) and add a code specific for the device;
• Compile the project and install to HoloLens (or try to debug it in the emulator (link https://developer.microsoft.com/en-us/windows/holographic/install_the_tools)).

The problem with this method is that the feedback loop is delayed just like the process itself. A possible solution to this is making a complete working project for VR in Unity only and then adapt it for HoloLens. However, adaptation can bring forth some control problems, since control with gestures differs drastically from standard input methods like a keyboard and a mouse.

Having the device, we didn’t have to go through all these troubles and managed to create a prototype in just 48 hours. The developed prototype has two control modes: voice control and gesture control. Voice control is realized with Cortana SDK, while gestures are recognized and interpreted with the help of SDK for Hololens. It is notable that Cortana operates without Internet connection in HoloLens.

See how it works:

Results

The prototype was developed in 48 hours in terms of Media Hack Weekend. My colleagues and me managed to create a functional prototype in this short time, which is a proof of technology and device maturity.

We loved working with Microsoft HoloLens and its elaborate API.

• #App Development
• #AR/VR