Getting started with gestures and Project Prague

Microsoft recently opened up a new group of cognitive services that are still in the experimental stage. A great opportunity to get a look in the kitchen and see what Microsoft is up to and gain some experience with these new technologies.

Today we will take a look at Project Prague. I first saw a demo on Build 2017 and have been waiting for ever to get into the beta program. Luckily a few weeks ago, to project released a public beta and now it finally time to play!

Step 2 - Get the SDK running

If you have your hardware settled then you have to install the project Prague SDK. You can find it here: http://aka.ms/gestures/download. It takes a while to download and install. After installing the SDK, some gestures are active and you can control some windows features like opening the start menu and controlling the sound volume. If you take a peek on your desktop you see some sample applications and a cool game you can play using gestures.

Step 3 - Write some code!

When you are done installing the SDK and you hardware is settled you are ready to create your own gestures and integrate them your apps.

In this example, we are going to make a simple app that greets when we show a peace sign.

Create a WPF application and reference 3 dll’s, located in the directory indicated by the MicrosoftGesturesInstallDir environment variable:

  • Gestures.dll
  • Gestures.Endpoint.dll
  • Gestures.Protocol.dll

In your solution make references to:

using Microsoft.Gestures;   
using Microsoft.Gestures.Endpoint;   
using Microsoft.Gestures.Skeleton;  
using Microsoft.Gestures.Stock.Gestures;  
using Microsoft.Gestures.Stock.HandPoses;

Next is to define a hand poses we need 2. The first pose is a fist and the second pose is the index and middle finger raised. Together these poses make a gesture.

var fist = new HandPose("Fist", new FingerPose(new AllFingersContext(), FingerFlexion.Folded));
var peace = new HandPose("Peace",
    new FingerPose(new[] { Finger.Index, Finger.Middle }, FingerFlexion.Open),
    new FingerPose(new[] { Finger.Thumb, Finger.Ring, Finger.Pinky}, FingerFlexion.Folded));

var makePeace = new Gesture("makePeace", fist, peace);

When the gestures are defined it is time to create the gesture service and register the newly defined poses.

var gesturesService = GesturesServiceEndpointFactory.Create();
await gesturesService.ConnectAsync();
await gesturesService.RegisterGesture(makePeace);

Finally you can write the code what happened when the gestured is triggered.

makePeace.Triggered += (sender, args) =>
{
      Dispatcher.Invoke(() => GreetingText.Text = "Hello!!!");
}

To summarize getting started and adding gestures through Project Prague is very easy. It just takes a few lines of code, the installation of an SDK and the right camera. I used a SR300 camera and the detecting of the gestures could be a bit more accurate, the space where in you the gestures are detected is really small. In this example I tested with some basic gestures, but more advanced gestures will take some research as documentation is very little at the moment.

Project Prague is a very impressive project and it is amazing how accurate and fast the gestures are detected. Keep up the good work!

Checkout my demo code on GitHub

Read more: