KinectV2 OSX

DusX

I am trying to align the Depth image and the Color image in OF (from Kinect 2).

I can't find information on how this alignment should work. I know the color is 1920*1080 and the depth 512*424

but scaling the depth upto to the height of the color doesn't align the images. Any knowledge / pointers ? I can get it workably close within Isadora.. but not perfect.

Fred

@DusX Ok, lets go back to some fundamentals, no amount of scaling and quad warping will actually line up 2 cameras, or a camera and a projector. The lenses, sensors and imaging systems will produce differently warped images (no image is not warped) and have different extrinsics, instrinsics and FOV's. The offset is not linear and needs a complicated algorithm to transform between one and the other.

Isadora misses fundamental tools to do this. It can be done through some calibration (like you can see with camera calibration in openCV).Microsoft have of course prepared this transformation in their SDK through the coordinate mapper that is accessible in OF in the windows only addon ofxKinectForWindows2

If you are on PC in OF you can see a bit how this works with these functions (this is not the place to go into depth into code so here are the method names)

virtual HRESULT STDMETHODCALLTYPE MapCameraPointToDepthSpace(

virtual HRESULT STDMETHODCALLTYPE MapCameraPointToColorSpace(

You could prepare a mesh, or reconstruct the coordinate mapping and use a shader in the GLSL tools now available in Isadora to achieve this, but first you need to reproduce the coordinate map.

I have wanted this kind of intelligent image manipulation in Isadora for a long time, camera calibration, and camera to world/ projector calibration would be a great tool and is something that underlies many questions that come up on the forum- projection on tracked objects...

DusX

@Fred

Funny, as always.. shortly after writing the previous post I found the coordinate mapper function in the ofxKinectForWindows2 addon (that is what I am working with)

I see that nearly what I want to do is already done in the BodyIndexColor example, so I think I can simply port that code with some minor changes (previously I was building from the Base example)

Thanks for the info. I had hoped that the images were corrected for alignment upfront (in the kinect hardware before exposing the images). Simply wasn't sure of how the Kinects structure/logic is setup.

axelacs

Hi all,

@Fred thanks so much for "KinectV2_Syphon"!
We are testing the app on El Capitan Macbook Pro. Works good. We can't make it work on Yosemite Mac mini though. We tried debugging by unplugging the power chord just like mark recommends and still nothing. Only a black image.
We are wondering if the app does not work on Yosemite?

Thanks!

Fred

Hi, I did not compile the app for Yosemite, I don't have any machines with Yosemite still running. I will see if I get a chance to do it in the coming days and send it.

axelacs

@Fred

We updated to Sierra since we thought it might be a Yosemite issue. Still showing only a black image on Sierra. Do you recommend plugging the kinect to a PC and run SDK to do the handshake or what are we missing?

thanks

Fred

Your Mac mini does have usb3 right?

axelacs

It does not have usb3. Only usb2 ports :I

Fred

The kinect V2 is a USB3.0 device, this is why it does not work. You may be able to use something like this

http://www.sonnettech.com/product/usb3gigethunderboltadapter.html

if you dont need the thunderbolt port. Warning I have not tested this with the kinect.

jhoepffner

Hello,

Just a little bit of personal information/experience

– I am working at the moment on an installation with a kinect V2, using only depth information. I use Processing with Open Kinect library on a mac, it work perfectly, precise and reliable. It possible to make the crunching number process in Processing and send the image to Isadora via Syphon. I tried to use a kinect V1 in supplement to have skeleton information, it works well but finally I doesn't need it.

– In the last beta version of Millumin, you can plug kinect V1 and V2 and obtain depth informations and skeleton information on Mac (Millumin is Mac only). I know the people making Millumin but we are not "friends" and a little bit in concurrency on the software field., I am a little bit the "Isadora guy" ! But it's a proof its possible. Unfortunately, its very hard to output OSC from Millumin.

Jacques

axelacs

@Fred
Thanks for the support!

Fred

@jhoepffner thanks for the info. I knew skeletons with v2 were coming on OSX but this is the first release implementation I saw. I checked out there latest nite and openni and see that it should all be working. I will have some time over the break to see if I can get this going.

Armando

Hello,

I am in need of Kinect version 2 skeleton data in izzy. Ni-mate people sent me a beta that supposedly does that. NOT WORKING unfortunately,

So, sadly I had to turn again to Touchdesigner under pc that allows plugging a kinetic v2 directly and has "soft" lines with gravity, elastoviscosity algorithms. etc,

But my heart belongs to Izzy!

jhoepffner

Hello,

I just received my second kinect adaptors and i tried to plug two kinect V2 to my mac.

It works with one usb3 directly on the mac and the other one through a caldigit thunderbolt box.

No skeleton but depth information (my interest at the moment) perfectly usable.

Unfortunately I must do all the computation and the image production in Processing because there is no way to pass it directly to Isadora.

Jacques

Fred

@Armando there is also this: https://github.com/microcosm It is windows but will get you OSC of the limb parts, although at law check it was xyz only no rotations.

Armando

Thanks, @Fred, I'll try soon and report

Armando

@ Fred @ Mark, well, it is still under pc. And if I use a pc there is the free version of Touchdesigner plus and play skeleton and image with a Kinect v 2 that can stream oscars to anything. But If I do that I might as well gon on using Touchdeigne itself.... that is coming to Mac OS. Next weeks are going to be more difficult for izzy on Mac with Touchdesigner.

jhoepffner

@Armando @Fred @Mark

I use Touchdesigner on Mac since a month. No kinect for the moment but it's coming soon (I hope).

And yes I think it will be a big chalenge for Izzy.

DusX

I have added https://github.com/rwebber/kinect2share Set up to share osc body parts and Spout over the video feeds. Windows only.

Armando

@ jhoepffner

yes, thanks for telling me about the mac version I did my 2 last projects on Touch windows version. Although Touchdesigner is just a gui for python. Without python you don't do much in Touch. But for sure Touchdesigner for Mac is going to bring change and stimuli for Isadora. Although the beta is not free on the mac. But Kinect 2 works like a charm on Touch under windows. Actually better than the Kinect 1 (funny).