[ANSWERED] Optical Flow simulation

RIL

Hello everyone ! I wonder if it is possible to achieve motion tracking the way Touch Designers uses OPTICAL FLOW? Of course leaving aside the Kinect or other depth cameras.

It seems to be something very straight foward this optical flow operator in Touch Designers but I always prefer Isadora!
Eyes Actor +? (possible scenario: an installation where the audience manipulates the projection of a system of particles)
Thanks friends !
RIL

Juriaan

@ril

Not as efficiently as they are doing with TD. Since they have a partnership with Nvidia who makes the SDK "Optical Flow"

EDIT:

Please see Fred's answer for a more detailed answer about this; you can do this thru other means on the CPU with Isadora.

RIL

@juriaan Thanks for your answer !

I think the difficult thing is getting a good sensing device that works well and doesn't crash. Kinect would be our best option for now, right?

Best,

Maxi-RIL

Fred

@juriaan optical flow is a very old technique not owned by Nvidia at all. It is not a complex algorithm and can be found in many open source libraries and there is a great implementation in openCV.

The touch designer optical flow is not the result of a partnership, just that touch designer uses the Nvidia SDK (https://developer.nvidia.com/optical-flow-sdk) which any company can do, including Troika Ranch. As it is Nvidia only this would make it have a pretty limited audience considering the larger number of mac users here.

The Nvidia version does this efficiently on the GPU.

There are several Python implementations of optical flow algorithms that can be integrated into isadora using the Pythoner actor, including openCV (https://learnopencv.com/optical-flow-in-opencv/). This does not have the efficiency of GPU acceleration, but on a modern cpu will still work quite well.

RIL

@fred great info !

Maybe a futured request for Optical flow implementation?

Students and artistic producers have one little word at the point of their tongue: interactivity ( for instalations)

Best,

Maxi-Ril

Juriaan

@fred

Hi Fred, I didn't mention that Nvidia owns the technique 'Optical Flow'. I said that we cant do it as efficient as they are doing with TD.

Other then that love the input that you gave and the links to the OpenCV counterparts that run on the CPU.

Skulpture

@juriaan Is there an GLSL code for this maybe?

Fred

@skulpture Here are some shadertoy examples of optical flow:

https://www.shadertoy.com/view...

The issue with this is that the computation of what is moving where and when is in a shader, this means if you want to do anything with that data, you need to do it in a shader. Getting data out of shaders is not always easy, and is best done with compute shaders, and this means windows only. The Nvidia SDK will do the computation on the GPU and pass data back to the CPU so it can be used for other calculations.

RIL

@fred

Thanks Fred for sharing... I'm going to try them!
Best,
Maxi-RIL

Skulpture

@fred said:

@skulpture Here are some shadertoy examples of optical flow:
https://www.shadertoy.com/view...https://www.shadertoy.com/view... />
https://www.shadertoy.com/view...
The" class="redactor-linkify-object">https://www.shadertoy.com/view... issue with this is that the computation of what is moving where and when is in a shader, this means if you want to do anything with that data, you need to do it in a shader. Getting data out of shaders is not always easy, and is best done with compute shaders, and this means windows only. The Nvidia SDK will do the computation on the GPU and pass data back to the CPU so it can be used for other calculations.

That makes a lot of sense. Thanks.