Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs. Based around a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox 360 without the need to touch a game controller, through a natural user interface using gestures and spoken commands.
PrimeSense, whose depth sensing reference design Kinect is based on, released their own open source drivers along with motion tracking middleware called NITE. OpenNI is an open-source software framework that is able to read sensor data from the Kinect, among other natural user interface sensors. Several middleware bundles needed to be installed on the iMac to be able to communicate with the Kinect. They include the OpenNI bundle, NITE, and Avin. In addition, an application called OSCeleton is needed to convert the tracking information from the Kinect to OSC.
http://www.openni.org/
The OpenNI framework is an open source SDK used for the development of 3D sensing middleware libraries and applications.
http://www.openni.org/files/nite/
https://github.com/avin2/SensorKinect/tree/master
https://github.com/Sensebloom/OSCeleton
An alternative to OSCeleton is an application called Synapse. Quartz Composer, part of the developer tools for Xcode, allows users to develop custom applications using the Synapse plug-in.
http://synapsekinect.tumblr.com/post/6305020721/download
https://developer.apple.com/xcode/
Above, a screen shot of the Synapse application. Note the articulation points it tracks. Below, a screen shot of the OSCeleton application, which is much more sparse and CPU-friendly. It's launched from the command line (Terminal window).
To launch OSCeleton we had to create a shell script that provides a path to the application and provides a port number (7110) for communication with the application. It uses local host (127.0.0.1) as the IP address of the machine.
#!/bin/bash
address="`ifconfig|grep "inet "|grep -v 127.0.0.1|cut -d " " -f2`"
~/Desktop/Kinect/osceleton -a $address -p 7110 -f -w -mz .5
Naturally, the path to the OSCeleton application will have to change if the name of the directory it is in changes. This shell script is actually called by a Lingo function, _player.open, in our demo on the file server in the startMovie script, which executes when the application is launched. This specifies the shell script to be run, osc.sh, the path to the script (relative to the Director demo), and the application to run the script (terminal).
_player.open (the moviePath & "osc.sh", "mmlab:applications:utilities:terminal")
After the osc.sh script is called, in the Terminal window note that OSCeleton has appended the port (7110) to the network IP address for the local machine. That port number will need to be used by Director to receive the OSC data. Then initialization of the Kinect device is confirmed. When a user is detected the OSCeleton window that was shown above appears, and OSCdata begins to flow.
Above is a screen shot of the demo for reading Kinect data in Director, called Calibrate_Kinect.dir. It's a modification of the OSCdecode.dir file that is packaged with the asUDP Xtra. The modifications required that we pull discrete values out of the list called bundle, so that we could assign those values to separate variables and display fields. This was done on a looping frame script called "Calibrate". Below is the full script:
global fps
global bundle
global left_hand
global right_hand
on enterFrame me
i = 2
repeat while i <= bundle.count
case bundle[i][2] of
"l_hand":
left_hand[1] = bundle[i][4]
member("LX").text = left_hand[1] & RETURN
left_hand[2] = bundle[i][5]
member("LY").text = left_hand[2] & RETURN
left_hand[3] = bundle[i][6]
member("LZ").text = left_hand[3] & RETURN
"r_hand":
right_hand[1] = bundle[i][4]
member("RX").text = right_hand[1] & RETURN
right_hand[2] = bundle[i][5]
member("RY").text = right_hand[2] & RETURN
right_hand[3] = bundle[i][6]
member("RZ").text = right_hand[3] & RETURN
end case
i = i + 1
end repeat
end if
end
on exitFrame me
go the frame
end
An further modification was the addition of a Flash movie (cam.swf) to the Director file to display a web cam image of the user to assist in calibration of the Kinect device. When the new Intel-compatible version of Director was released, third party libraries that used to allow Director access to the signal from the web cam no longer worked, and none have been updated. Fortunately Dierector can embed Flash movies with full functionality. The Flash file is very simple. It contains a video component object, and a frame action to initiate the video feed.
stop();
var my_cam:Camera = Camera.get();
var my_video:Video;
my_video.attachVideo(my_cam);
Security settings in the Flash Player have to be set to be able recognize the iMac's web cam. Control-clicking the cam.swf file in the Flash Player will display a pop-up menu. Choose settings. The settings dialog will appear. Choose "Allow" and "Remember" to give permission for the local machine to access the web cam. Select the camera icon and set it to "USB Video Class Video".
The demo file Calibrate_Kinect.dir and supporting files (osc.sh, osceleton, and cam.swf) are in Groups > 323 > Novel Inputs > Kinect on the file server.
Thanks to Ryan Walsh for assistance with this demo.