Optical Flow in VFX

[emlcpfx]

I have dot (and other OF repos) exporting flow data as 32-bit EXR STMaps for use in visual effects.

https://www.linkedin.com/posts/ericlevy_my-little-optical-flow-project-is-coming-activity-7231701952487723008-W9ae?utm_source=share&utm_medium=member_ios

I’d love help — even just conceptually — on how to improve the results.

I have it so I can export an STMap from different reference frames, accumulating the flow forward and then backward from there.

I’m trying to get it as close to SmartVectors in Nuke as possible. I talked with someone who trained their own OF model, who said this about SmartVectors, “ …can output vectors in the SmartVector format. That is still just multiple motion vector sets. The actual processing and conversion to STMaps happens in the NukeX node VectorDistort.”

I’m not smart enough to figure out how to export multiple motion vector sets (presumably from different reference frames,) and then combine them to improve the accuracy and stability of the resulting track.

I can share my code if that helps! Thanks.

[16lemoing]

https://www.linkedin.com/posts/ericlevy_my-little-optical-flow-project-is-coming-activity-7231701952487723008-W9ae?utm_source=share&utm_medium=member_ios

Great project! Has this been done using dot? I would expect single reference to be enough for this kind of short clip.

We have been exploring ways to combine multiple reference frames to further improve the results but it is not yet implemented on this repo.

[emlcpfx]

Hey, this was done with SEA-RAFT, and dot had the exact same output. I'm doing a similar workflow with Blackmagic Fusion, and their results are even more stable:

https://www.linkedin.com/posts/ericlevy_this-blackmagic-fusion-smartvector-workflow-activity-7232052769539457024-aSed?utm_source=share&utm_medium=member_desktop

Do you think we'd be able to join forces to beat them? :)

[16lemoing]

Results look great indeed!

I don’t have much bandwidth right now, but if there’s anything specific I can do to help, please let me know!