Final steps towards the open beta phase for the Deutsches Museum project

August 3, 2015

We are moving towards the final installation of our exhibition project for the Deutsches Museum München. Yesterday we were building in the cable ducts. Before starting the open beta phase we will now do a long time test of the exhibition.


TNG Augmented Rift presented at Augmented World Expo and Big Techday 8

June 15, 2015

The challenge

The hardware hacking team from TNG Technology Consulting GmbH started to work on a device which is capable to do full field of view augmented reality in March 2015 when the company’s winter retreat was taking place.

The mobile "TNG Augmented Rift" is a full field of view Augmented Reality device capable to do e.g. face tracking, face identification, emotion detection, heart rate detection, distance measurements, etc. Speech recognition for command and control is implemented as well.

The mobile “TNG Augmented Rift” is a full field of view Augmented Reality device capable to do e.g. face tracking, face identification, emotion detection, heart rate detection, distance measurements, etc. Speech recognition for command and control is implemented as well.

The goal

The goal was to built a device made with off-the-shelf hardware in one working day. Within this short period of time eleven TNG software consultants were able to implement a prototype which is based on an Oculus Rift DK2 and an Intel RealSense F200 3D-camera. With some additional days and a core team of about six TNG software developers we were able to add some more features for the Augmented Rift.

The TNG Augmented Rift consists of off-the-shelf hardware like the Oculus Rift and the Intel RealSense F200 3D-camera.

The TNG Augmented Rift consists of off-the-shelf hardware like the Oculus Rift and the Intel RealSense F200 3D-camera.

The features

The TNG Augmented Rift has the following features:

  • See through” Augmented Reality head mounted display with Oculus Rift DK2 (in contradiction to the HoloLense the whole field of view is augmented)
  • You can see a lot of real time elements inside the Augmented Rift in 3D

The following visible elements are supported:

  • Real environment mixed with augmented information and 3D objects
  • Face detection elements
  • Face tracking information
  • Face landmarks
  • Emotions
  • Heart rate as text
  • (in progress) Heart rate as a graph (“Electrocardiography mode”)
  • Speech recognition for command and control (“Okay Rift, this is Martin.” – “Is this really Martin?” – “Yes!” – “From now on I will remember that user 100 is Martin.”)
  • “Terminator mode” (world augmented with red-shaded textures)

The setup

We need two cameras to record the real world environment. The video streams of the two cameras are merged to a stereo (3D) capture. The webcam processor itself consists of different parts. We use the OpenCV library to do long distance measuring and face recognition of people which are not in the nearer distance.

The Augmented Rift consists of two cameras for recording the real world. The information gets streamed into the Oculus Rift DK2. The Intel RealSense F200 is used to add features like face tracking, face identification, pulse detection and much more.

The Augmented Rift consists of two cameras for recording the real world. The information gets streamed into the Oculus Rift DK2. The Intel RealSense F200 is used to add features like face tracking, face identification, pulse detection and much more.

The Intel RealSense detector uses an F200 3D camera that can be used for realizing the features mentioned above (near field face tracking, face identification, face landmark detection and tracking, emotion detection a.s.o.). All the Augmented Reality elements we gather are added to our stereo webcam capture to enrich the real world environment capture with useful information.

Wait but why?

Of course this device based on an Oculus Rift DK2 is a little bit awkward. But imagine what would happen if the computers and cameras used are getting smaller and smaller within the next few years? It doesn’t need a Oculus Rift DK2 to realize such projects. What we need is glasses with an integrated full field of view transparent display to enrich the world with Augmented Reality elements in a progressive way.

Nowadays devices are only augmenting a small fraction of the field of view of the user (see Epson BT200 or Microsoft Hololens). You need to focus on a special direction to see Augmented Reality elements on the display. With the Augmented Rift this is not problem but the device is huge. If there will be glasses with transparent displays available we could transfer our solution to these new devices.

In the front view of the Augmented Rift you can see two web cameras for streaming a stereo capture image into the Augmented Rift. This would not be necessary when using normal glasses with transparent displays inside the glasses and an R200 camera.

In the front view of the Augmented Rift you can see two web cameras for streaming a stereo capture image into the Augmented Rift. This would not be necessary when using normal glasses with transparent displays inside the glasses and an R200 camera.

At least you could argue that the 3D camera used is too huge for such a field of application. But we could also use the very new Intel RealSense R200. It’s such a tiny device and so powerful that you can easily attach it to glasses. The weight is negligibly small.

Intel RealSense R200

Intel RealSense R200

We want to make a video of the TNG Augmented Rift as soon as possible. See the world like a real T-800! The future is here!

The talks and demonstrations

The Augmented Rift was presented at the Augmented World Expo in Santa Clara, California, USA at the Intel booth from the 8th to the 10th of June 2015. On the 12th of June we held a talk at the Big Techday in Munich with more than 100 developers. After the talk hundreds of people wanted to try the Augmented Rift with its Terminator mode on their own!

Disclaimer

The TNG Augmented Rift is developed by TNG Technology Consulting GmbH software consultants. The software parts used by the Intel RealSense are developed by ParrotsOnJava.com

Hasta la vista!


First pictures of the Intel RealSense R200 development kit

May 29, 2015

This week we’ve received an interesting package by Intel USA with the very new Intel RealSense F200 camera. We are really surpised how small the camera is. The camera is a longer range peripheral 3D camera, perfect for sensing the environment (for Windows and Android tablets, 2-in-1s, and more). The “R” in the cameras model name stands for “Rear” since it is best suitable using it while the camera is facing away from you (instead of the F200 camera which is facing frontally).

The camera has a range of 3-4 metres (inside a room) and a larger range out of doors. The key features of the camera are

  • 3D recording (faces, people, environment)
  • Depth camera
  • Face tracking and face recognition
  • Measuring in general

We are now waiting for the SDK to start our first projects with this tiny piece of high-tech.

Intel RealSense R200

Intel RealSense R200


New test at the German Museum

May 25, 2015

We are currently in the German Museum for the 5th test of our gesture control demo exhibition. Most of the technical problems seem to be solved right now. We are now looking forward to the open beta phase so we can test the autonomous operation of the installation.

 


Parrots On IoT Developer Day

April 19, 2015

On the 16th of April we held our talk about gesture control using different 3D camera systems like the Intel Real Sense, Leap Motion, and Kinect v2 at the IoT Developer Day in Utrecht, Netherlands. About 100 people were visiting our presentation. The attendees were really impressed with what you can do with gesture cameras.

We demoed many showcases like the LeapMotion synthesizer, our drone showcase and our newest game “Parrots On Target”. The talk was recorded so we hope that we can post a YouTube link to the video soon.


ParrotsOn JavaLand Conference

April 1, 2015

Last week we’ve attented the JavaLand Conference in the  theme park Phantasialand in Brühl near Cologne. We’ve demonstrated our showcases at the “Java Innovation Lab” starting from flying drones with bare hands up to our very new web based Intel RealSense HTML5 game “Parrots On Target”.

TNG Technology Consulting GmbH – the company we are working for as Software Consultants – was supporting us in doing this. We also showed another really cool showcase which was built during the so called “TNG Winterretreat”. This TNG showcase is about an Oculus Rift DK2 enhanced with two cameras to build an “Augmented Rift”. The idea is to view the real world through the Oculus Rift DK2 enhanced by Augmented Reality elements. The Terminator vision is real!

ParrotsOn JavaLand 2015


Release of an article in the technical magazine “Java Aktuell”

March 10, 2015

We wrote an article for the German technical magazine “Java Aktuell“. [1] The article is about gesture control, native interface devices and new developments in the field of 3D cameras. We are having a look back in history showing the evolution of gesture control starting from the year 1990 while analyzing advantages and disadvantages of the different 3D camera technologies. Furthermore we wrote about the state of the art with some C# and Java code examples and different fields of application as well. If you don’t have the current issue at hand you can read the publication here in PDF format. The whole article was published on the Intel Developer Zone too.

[1] Förtsch, M.; Endres, T. (2015). “Gestensteuerung und die nächste Welle der 3D-Kameras”. Java Aktuell 02/2015 (2015): 30-34. Print.

Java Aktuell 02-2015

[1] Förtsch, M.; Endres, T. (2015). “Gestensteuerung und die nächste Welle der 3D-Kameras”. Java Aktuell 02-2015 (2015): 30-34. Print.


Follow

Get every new post delivered to your Inbox.

Join 79 other followers