This photo is from a time lapse of a beautiful sunset I shot yesterday in the city. It was so gorgeous that got me so tempted that I was going to just upload this one single clip here so everyone could enjoy it, but I decided to wait instead and stick a couple more time lapses to it and make an awesome clip off of them.

The problem of shooting video in this time of year is that it is currently winter and everything looks a lot boring, also the pollution often makes landscapes look awful in winter. Finding cool time lapse subjects in this time of year is very difficult, but I’m going to look for them anyway.

Let’s keep looking!

What is the problem of a person who is relatively good at drawing images physically, with doing the same thing only on the computer? I had never put any effort in drawing in my life, and I was horrible at it when I had to do it when I used to go to school; Therefore I had no idea how difficult or easy drawing a bunch of simple game sprites would be.

So a couple of days ago, I began doing an experiment by opening Adobe Illustrator and trying to just draw a human figure. I spent some time with Illustrator, and finally came up with the character below:

For a person like me, who has practically done no serious drawings in my entire life, coming up with these images, leads me to a number of conclusions:

• It is definitely not at all difficult to draw computer based illustrations.
• Most people just don’t want to cooperate. If you see someone hesitating even a little bit, DO NOT WASTE YOUR TIME ON THEM.
• Self-confidence is the most important thing in trying something new. I’m sure even if I offered these people a huge amount of money, and gave them high quality drawing tablets, they still couldn’t pull off the job. Because most people lack self-confidence. They don’t show it, but they really do.

So from now on, I’ve decided to draw my sprites by myself. I’m going to use these frames and a couple more in an experimental platformer game to see how they look.

Today when I was editing a relatively complex sequence, I finally found a trick to make this problem go away. First, Let me tell you my system specs. I’m using Red Giant Magic Bullet Looks 2.0.4 with Adobe Premiere Pro CS5.5 on a dual display system (With different aspect ratios) running a 64-bit version of Windows 7 powered by an ATI Radeon HD 5770 GPU. I have installed the latest update of MBL and my LooksBuilder looks like this:

So the problem is sometimes when I click Edit from the Effect Controls panel, the Looks Builder shows a blank or an irrelevant preview frame. I think it tries to load the last rendered frame from some place on the memory and often times the right frame isn’t on that part of the memory.

Whether this is the reason for showing an invalid frame or not, the problem can be solved by simply clicking on the Looks name (The name of the effect) on the Effect Controls panel, wait for the frame to render, and then click the Edit… button. It’s simple as that!

Helper classes for both Speech SDK and NUI (Kinect Natural User Interface) are implemented in the latest changeset of Neat Game Engine’s source code. Implementing voice commands in games for simple tasks such as starting the game, pausing and resuming it, etc. is not a difficult task, because Speech SDK easily handles these stuff with Grammars and other classes. What’s really interesting in controlling games with sounds is finding a natural place for them inside games. Things like finding the position of the sound source in 3D and finding use for “meaningless” voices inside the game is what makes voice integration in games really fun.

I’ll keep on experimenting more with voice and speech in games and keep here updated!

I wrote a few lines of code to test the speech recognition engine, and I am not yet satisfied with the results:

In this video, I am standing around 2 meters away from the Kinect, in an almost quiet room.

1. Initialization

First we have to initialize the Nui Runtime. Assuming we are going to use the first available Kinect device connected to our system, we can write:

public Runtime Nui = Runtime.Kinects[0];
Nui.Initialize(RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The Runtime class is inside the Microsoft.Research.Kinect.Nui namespace. The second line tells the SDK that we are going to use both skeletal tracking and RGB (color) camera.

Now that we initialized the SDK, we have to tell it to do what we want when a video frame or a skeleton frame is ready:

Nui.VideoFrameReady +=
   new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
Nui.SkeletonFrameReady +=
   new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

2. RGB Stream

One of the most common things that XNA developers want to do with Kinect SDK, is to show the video captured by the RGB camera in their game. To do this, first you have to store the stream into a Texture2D object. Therefore, before we begin, we have to create our target texture:

public Texture2D KinectRGB =  new Texture2D(GraphicsDevice, xMax, yMax);

Now, we can perform the conversion when video frames become ready:

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    PlanarImage image = e.ImageFrame.Image;

    int offset = 0;
    Color[] bitmap = new Color[xMax * yMax];
    for (int y = 0; y < yMax; y++)
        for (int x = 0; x < xMax; x++)
        {
            bitmap[y * xMax + x] = new Color(image.Bits[offset + 2],
                image.Bits[offset + 1], image.Bits[offset], 255);
            offset += 4;
        }
    KinectRGB.SetData(bitmap);
}

The raw image data is stored in e.ImageFrame.Image. The usual method of filling XNA textures is by creating a Color array, fill it with pixel data and feed it to a texture using its SetData method. Color data is stored in BGR32 format, meaning that the value for blue channel is the first value stored, and the red value is the last. For more information about reading the camera data, watch this video.

To begin reading from the RGB camera, we have to open the video stream:

Nui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.Color);

In this example, the value of xMax is 640 and yMax is 480.

3. Skeletal Tracking

Getting skeleton data from Kinect SDK is easy. This example stores the data into an array for further use in the game:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    var trackedSkeletons = from s in e.SkeletonFrame.Skeletons
           where s.TrackingState == SkeletonTrackingState.Tracked
           select s;
    trackedSkeletonsCount = trackedSkeletons.Count();
    for (int i = 0; i < trackedSkeletonsCount; i++)
        Skeletons[i] = trackedSkeletons.ElementAt(i);
}

To draw a skeleton, you just have to get each joint’s position and draw a line between adjacent joints.

This helper function draws a skeleton using Neat engine’s LineBrush class.

public void DrawSkeleton(SpriteBatch spriteBatch, LineBrush lb, Vector2 position, Vector2 size, Color color, int skeletonId = 0)
{
    if (Skeletons.Length <= skeletonId || Skeletons[skeletonId] == null)
    {
        //Skeleton not found. Draw an X
        lb.Draw(spriteBatch, position, position + size, color);
        lb.Draw(spriteBatch, new LineSegment(position.X+size.X, position.Y,             position.X, position.Y + size.Y), color);
        return;
    }

    //Right Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandRight, size, skeletonId),
        ToVector2(JointID.WristRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristRight, size, skeletonId),
        ToVector2(JointID.ElbowRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowRight, size, skeletonId),
        ToVector2(JointID.ShoulderRight, size, skeletonId), color, position);

    //Head & Shoulders
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderRight, size, skeletonId),
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Head, size, skeletonId),
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderCenter, size, skeletonId),
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);

    //Left Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandLeft, size, skeletonId),
        ToVector2(JointID.WristLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristLeft, size, skeletonId),
        ToVector2(JointID.ElbowLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowLeft, size, skeletonId),
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);

    //Hips & Spine
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId),
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId),
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId),
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId),
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);

    //Left foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId),
        ToVector2(JointID.KneeLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeLeft, size, skeletonId),
        ToVector2(JointID.AnkleLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleLeft, size, skeletonId),
        ToVector2(JointID.FootLeft, size, skeletonId), color, position);

    //Right foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId),
        ToVector2(JointID.KneeRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeRight, size, skeletonId),
        ToVector2(JointID.AnkleRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleRight, size, skeletonId),
        ToVector2(JointID.FootRight, size, skeletonId), color, position);
}

Be sure to read the source code of Neat’s KinectEngine class for more information.

To be able to build this sample, first you have to install the official Kinect SDK Beta 2 and obtain the latest version of the Neat source code and build it with the KINECT directive.

Neat currently supports Kinect through a component class named KinectEngine. To begin using Kinect in your Neat based XNA games, you have to follow these simple steps:

Initialization

You have to add an instance of KinectEngine to you game’s Components. Write these in your LoadContent method to initialize KinectEngine:

Kinect = new KinectEngine(this);
Components.Add(Kinect);
Kinect.Initialize();
CalibrateScreen.Kinect = Kinect;
Kinect.OpenVideoStream();

Just remember to call Kinect.Uninitialize() when you are shutting down the game.

Calibration

You can monitor the stream from the RGB camera and see the elevation angle in a screen named CalibrationScreen. To use this screen, you simply have to add it to your game’s screens:

Screens["kinect"] = new CalibrateScreen(this);

k_tilt [int] command changes the elevation angle of the Kinect sensor from the console.

Convert positions to XNA Vectors

Use KinectEngine.ToVector2 and KinectEngine.ToVector3 methods to scale and convert the position of each joint to XNA vectors. Now you are ready to interact with your game using Kinect.

Draw Skeletons and the RGB Stream

If you want to draw a skeleton, all you have to do is to call KinectEngine.DrawSkeleton(…) and pass a SpriteBatch and LineBrush to it. The image from the RGB camera can be accessed from the “kinectrgb” texture:

SpriteBatch.Draw(
	game.GetTexture("kinectrgb"),
	new Rectangle(0, 0, 640, 480),
	Color.White);

It has been a slow year and I didn’t have much time to work on games, so there aren’t that many new features in the new release of Neat Engine, however the code is much more polished and stable. Some of the new additions to Neat Engine are:

• A simple 2D physics simulator that can be easily used in regular platformers and other simple 2D games. It uses Neat’s own polygon class and collision detection mechanism.
• Sprite sheets support: Now you don’t have to create a new texture file for each of your sprites. Sprite sheets can increase your game’s performance significantly.
• New geometry and math helper functions such as Line Segment/Line Segment intersection detection, Point on line detection, etc.
• A more functional auto complete mechanism in the Console.
• Draw functions for LineBrush class.
• Windows and Containers in the GUI system, plus new events for controls such as Move, Resize, Focused, etc.
• The ability to load regular image files directly via scripts without using the Content Manager.

I am working on adding Kinect support to Neat using the official Kinect SDK. If you are interested in playing with a Kinect-compatible Neat Engine, you can download the latest version of the source code from the Source Code page of the project on CodePlex.

The first time I heard about these cameras I thought they would be a little expensive; at least as expensive as some of the normal DSLRs. but when LYTRO launched they cameras and I saw the super affordable $399 price, I got very tempted to buy one. But after some thinking, I decided not to, because of a few reasons…

The first and most obvious reason is that these are the first generation products of a new company, therefore, there is a high probability that they suck. After all, this is a truly new invention, and besides that, even the first generation Apple products like the MacBook Air had their own weird problems. Also, the firmware, and the engine that drives this technology might have too many annoying bugs and I wouldn’t wanna waste my money on products that are very likely to fail on me on a short period of time.

Another reason is that if you pay attention to the way they are promoting these cameras, they rarely talk about the output resolution. they try to compare these cameras with the ordinary digital cameras like it’s the new digital vs. analog. But the fact is that these are still digital cameras, so they can’t have infinite resolution. This leads me to believe that the overall quality of the pictures taken with LYTRO cameras is crappy.

And the final reason for not buying a LYTRO is its design. It looks like a weird flashlight! Whoever is in charge of designing the looks of a LYTRO camera, is doing a crappy job. A long body with a tiny display in the back is not the way to design a camera. If this technology must have a long body, why don’t they make it look like a normal camcorder?

So, I think we should still wait a little and give them more time to make a perfect consumer product with this cool technology. We should wait for a LYTRO device that shoots video, has a plugin for Photoshop, After Effects and Premiere, and has the perfect looks and picture quality. Who knows, maybe some other giant company like Canon or Sony buys LYTRO soon and integrates this technology with its cameras. I don’t know about you, but I’d rather buy a Sony Light Field camera than a LYTRO one!