The Ocean

No preview in Magic Bullet Looks Builder?

I have had a very annoying problem with LooksBuilder for a long time. Most times, when I opened LooksBuilder to edit a look from Premiere Pro, it showed a black screen, or a frame from another clip in my project. This maked using Magic Bullet Looks very annoying and almost impossible, because I had to practically edit my looks blindly and with trial and error. I looked for it on the Internet, but did not find anything helpful. Someone said this is a “Licensing Problem.”

Today when I was editing a relatively complex sequence, I finally found a trick to make this problem go away. First, Let me tell you my system specs. I’m using Red Giant Magic Bullet Looks 2.0.4 with Adobe Premiere Pro CS5.5 on a dual display system (With different aspect ratios) running a 64-bit version of Windows 7 powered by an ATI Radeon HD 5770 GPU. I have installed the latest update of MBL and my LooksBuilder looks like this:

 

So the problem is sometimes when I click Edit from the Effect Controls panel, the Looks Builder shows a blank or an irrelevant preview frame. I think it tries to load the last rendered frame from some place on the memory and often times the right frame isn’t on that part of the memory.

 

 

Whether this is the reason for showing an invalid frame or not, the problem can be solved by simply clicking on the Looks name (The name of the effect) on the Effect Controls panel, wait for the frame to render, and then click the Edit… button. It’s simple as that!

 

Speech and Skeletal Tracking in River Raid X

I’ve been experimenting with Speech SDK lately and was trying to find a way to control games with sounds. In this video, you can see me control River Raid X with Kinect skeletal tracking and voice.

Helper classes for both Speech SDK and NUI (Kinect Natural User Interface) are implemented in the latest changeset of Neat Game Engine’s source code. Implementing voice commands in games for simple tasks such as starting the game, pausing and resuming it, etc. is not a difficult task, because Speech SDK easily handles these stuff with Grammars and other classes. What’s really interesting in controlling games with sounds is finding a natural place for them inside games. Things like finding the position of the sound source in 3D and finding use for “meaningless” voices inside the game is what makes voice integration in games really fun.

I’ll keep on experimenting more with voice and speech in games and keep here updated!

Testing Speech SDK

I am planning on using Microsoft Speech SDK and Kinect’s Microphone Array in one of my future projects codenamed “Kintouch.”

I wrote a few lines of code to test the speech recognition engine, and I am not yet satisfied with the results:

In this video, I am standing around 2 meters away from the Kinect, in an almost quiet room.

Kinect SDK and XNA

It seems many people are looking for the way to draw skeletons and store the RGB camera data into Texture2Ds. The KinectEngine class which is included in Neat game engine does all these stuff, but here’s the direct way to do these in XNA:

1. Initialization

First we have to initialize the Nui Runtime. Assuming we are going to use the first available Kinect device connected to our system, we can write:

public Runtime Nui = Runtime.Kinects[0];
Nui.Initialize(RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The Runtime class is inside the Microsoft.Research.Kinect.Nui namespace. The second line tells the SDK that we are going to use both skeletal tracking and RGB (color) camera.

Now that we initialized the SDK, we have to tell it to do what we want when a video frame or a skeleton frame is ready:

Nui.VideoFrameReady += 
   new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
Nui.SkeletonFrameReady += 
   new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

2. RGB Stream

One of the most common things that XNA developers want to do with Kinect SDK, is to show the video captured by the RGB camera in their game. To do this, first you have to store the stream into a Texture2D object. Therefore, before we begin, we have to create our target texture:

public Texture2D KinectRGB =  new Texture2D(GraphicsDevice, xMax, yMax);

Now, we can perform the conversion when video frames become ready:

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    PlanarImage image = e.ImageFrame.Image;

    int offset = 0;
    Color[] bitmap = new Color[xMax * yMax];
    for (int y = 0; y < yMax; y++)
        for (int x = 0; x < xMax; x++)
        {
            bitmap[y * xMax + x] = new Color(image.Bits[offset + 2], 
                image.Bits[offset + 1], image.Bits[offset], 255);
            offset += 4;
        }
    KinectRGB.SetData(bitmap);
}

The raw image data is stored in e.ImageFrame.Image. The usual method of filling XNA textures is by creating a Color array, fill it with pixel data and feed it to a texture using its SetData method. Color data is stored in BGR32 format, meaning that the value for blue channel is the first value stored, and the red value is the last. For more information about reading the camera data, watch this video.

To begin reading from the RGB camera, we have to open the video stream:

Nui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.Color);

In this example, the value of xMax is 640 and yMax is 480.

3. Skeletal Tracking

Getting skeleton data from Kinect SDK is easy. This example stores the data into an array for further use in the game:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    var trackedSkeletons = from s in e.SkeletonFrame.Skeletons
           where s.TrackingState == SkeletonTrackingState.Tracked
           select s;
    trackedSkeletonsCount = trackedSkeletons.Count();
    for (int i = 0; i < trackedSkeletonsCount; i++)
        Skeletons[i] = trackedSkeletons.ElementAt(i);
}

To draw a skeleton, you just have to get each joint’s position and draw a line between adjacent joints.

This helper function draws a skeleton using Neat engine’s LineBrush class.

public void DrawSkeleton(SpriteBatch spriteBatch, LineBrush lb, Vector2 position, 
Vector2 size, Color color, int skeletonId = 0)
{
    if (Skeletons.Length <= skeletonId || Skeletons[skeletonId] == null)
    {
        //Skeleton not found. Draw an X
        lb.Draw(spriteBatch, position, position + size, color);
        lb.Draw(spriteBatch, new LineSegment(position.X+size.X, position.Y, 
            position.X, position.Y + size.Y), color);
        return;
    }

    //Right Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandRight, size, skeletonId), 
        ToVector2(JointID.WristRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristRight, size, skeletonId), 
        ToVector2(JointID.ElbowRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowRight, size, skeletonId), 
        ToVector2(JointID.ShoulderRight, size, skeletonId), color, position);

    //Head & Shoulders
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderRight, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Head, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderCenter, size, skeletonId), 
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);
            
    //Left Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandLeft, size, skeletonId), 
        ToVector2(JointID.WristLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristLeft, size, skeletonId), 
        ToVector2(JointID.ElbowLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowLeft, size, skeletonId), 
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);

    //Hips & Spine
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);

    //Left foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), 
        ToVector2(JointID.KneeLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeLeft, size, skeletonId), 
        ToVector2(JointID.AnkleLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleLeft, size, skeletonId), 
        ToVector2(JointID.FootLeft, size, skeletonId), color, position);

    //Right foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), 
        ToVector2(JointID.KneeRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeRight, size, skeletonId), 
        ToVector2(JointID.AnkleRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleRight, size, skeletonId), 
        ToVector2(JointID.FootRight, size, skeletonId), color, position);
}

Be sure to read the source code of Neat’s KinectEngine class for more information.