Are there parrots on Java?

The challenge

The hardware hacking team from TNG Technology Consulting GmbH started to work on a device which is capable to do full field of view augmented reality in March 2015 when the company’s winter retreat was taking place.

The mobile “TNG Augmented Rift” is a full field of view Augmented Reality device capable to do e.g. face tracking, face identification, emotion detection, heart rate detection, distance measurements, etc. Speech recognition for command and control is implemented as well.

The goal

The goal was to built a device made with off-the-shelf hardware in one working day. Within this short period of time eleven TNG software consultants were able to implement a prototype which is based on an Oculus Rift DK2 and an Intel RealSense F200 3D-camera. With some additional days and a core team of about six TNG software developers we were able to add some more features for the Augmented Rift.

The TNG Augmented Rift consists of off-the-shelf hardware like the Oculus Rift and the Intel RealSense F200 3D-camera.

The features

The TNG Augmented Rift has the following features:

• See through” Augmented Reality head mounted display with Oculus Rift DK2 (in contradiction to the HoloLense the whole field of view is augmented)
• You can see a lot of real time elements inside the Augmented Rift in 3D

The following visible elements are supported:

• Real environment mixed with augmented information and 3D objects
• Face detection elements
• Face tracking information
• Face landmarks
• Emotions
• Heart rate as text
• (in progress) Heart rate as a graph (“Electrocardiography mode”)
• Speech recognition for command and control (“Okay Rift, this is Martin.” – “Is this really Martin?” – “Yes!” – “From now on I will remember that user 100 is Martin.”)
• “Terminator mode” (world augmented with red-shaded textures)

The setup

We need two cameras to record the real world environment. The video streams of the two cameras are merged to a stereo (3D) capture. The webcam processor itself consists of different parts. We use the OpenCV library to do long distance measuring and face recognition of people which are not in the nearer distance.

The Augmented Rift consists of two cameras for recording the real world. The information gets streamed into the Oculus Rift DK2. The Intel RealSense F200 is used to add features like face tracking, face identification, pulse detection and much more.

The Intel RealSense detector uses an F200 3D camera that can be used for realizing the features mentioned above (near field face tracking, face identification, face landmark detection and tracking, emotion detection a.s.o.). All the Augmented Reality elements we gather are added to our stereo webcam capture to enrich the real world environment capture with useful information.

Wait but why?

Of course this device based on an Oculus Rift DK2 is a little bit awkward. But imagine what would happen if the computers and cameras used are getting smaller and smaller within the next few years? It doesn’t need a Oculus Rift DK2 to realize such projects. What we need is glasses with an integrated full field of view transparent display to enrich the world with Augmented Reality elements in a progressive way.

Nowadays devices are only augmenting a small fraction of the field of view of the user (see Epson BT200 or Microsoft Hololens). You need to focus on a special direction to see Augmented Reality elements on the display. With the Augmented Rift this is not problem but the device is huge. If there will be glasses with transparent displays available we could transfer our solution to these new devices.

In the front view of the Augmented Rift you can see two web cameras for streaming a stereo capture image into the Augmented Rift. This would not be necessary when using normal glasses with transparent displays inside the glasses and an R200 camera.

At least you could argue that the 3D camera used is too huge for such a field of application. But we could also use the very new Intel RealSense R200. It’s such a tiny device and so powerful that you can easily attach it to glasses. The weight is negligibly small.

Intel RealSense R200

We want to make a video of the TNG Augmented Rift as soon as possible. See the world like a real T-800! The future is here!

The talks and demonstrations

The Augmented Rift was presented at the Augmented World Expo in Santa Clara, California, USA at the Intel booth from the 8th to the 10th of June 2015. On the 12th of June we held a talk at the Big Techday in Munich with more than 100 developers. After the talk hundreds of people wanted to try the Augmented Rift with its Terminator mode on their own!

Disclaimer

The TNG Augmented Rift is developed by TNG Technology Consulting GmbH software consultants. The software parts used by the Intel RealSense are developed by ParrotsOnJava.com

Hasta la vista!

This week we’ve received an interesting package by Intel USA with the very new Intel RealSense F200 camera. We are really surpised how small the camera is. The camera is a longer range peripheral 3D camera, perfect for sensing the environment (for Windows and Android tablets, 2-in-1s, and more). The “R” in the cameras model name stands for “Rear” since it is best suitable using it while the camera is facing away from you (instead of the F200 camera which is facing frontally).

The camera has a range of 3-4 metres (inside a room) and a larger range out of doors. The key features of the camera are

• 3D recording (faces, people, environment)
• Depth camera
• Face tracking and face recognition
• Measuring in general

We are now waiting for the SDK to start our first projects with this tiny piece of high-tech.

Intel RealSense R200