Support for online tracking tasks

[Soooooda69]

Hi, thanks for this fantastic work!
I'm running the demo but found it takes pretty long time (~1min 30s) to track batch of points on varanus data. Is there any ways to speed it up, or I'm just wondering if Dot could do online tracking that takes images one by one and generate tracked points for each frame. Thanks!

[16lemoing]

Hi @Soooooda69, you can get considerable speed-ups by playing with the number of initial tracks extracted with CoTracker (this is the time bottleneck in most applications). The flag --num_tracks sets the total number of initial tracks and --sim_tracks sets the number of simultaneous tracks. Here are a few examples of the time you can get on the varanus data:

method	num_tracks	sim_tracks	time for initial tracks	time for refinement	total time
default	8192	2048	1'30	0'13	1'43
parallelize	8192	8192	1'13	0'13	1'26
fewer tracks	2048	2048	0'25	0'13	0'38
"	1024	1024	0'18	0'13	0'31
"	512	512	0'13	0'13	0'26

Reducing the number of tracks may affect performance, but it does not seem to be the case for this video:

varanus_512_tracks.mp4

Further speed-ups may be obtained by parallelizing the refinement steps, but it has not been implemented yet.

Online tracking is an interesting feature. I have not implemented it yet either. I don't think it would help with speed much though, but it would allow to do inference on long videos.

[Soooooda69]

Thanks for the reply! So, based on the examples, I wonder if it is the case that with even fewer num of tracks, e.g. 200, the bottle neck for total running time would lies in 'time for refinement'. Seems like it takes 13 seconds no matter the time for initial tracks. Then Cotracker might be faster than Dot for smaller tracks case, or do I understand this correct?

[16lemoing]

The goal is to predict a dense field of N tracks from a sparse set of K tracks (extracted with CoTracker). In the example above, N is fixed (N=480*856) which is why the refinement stage always takes the same time (no matter K). Reducing K allows to speed up the first step. However, it should be noted that directly producing the N tracks with CoTracker would take much longer (see paper for details).

As mentioned in my response above, the 'time for refinement' could be reduced by parallelizing this loop:

dot/dot/models/dense_optical_tracking.py

Line 128 in e32c6f7

for tgt_step in tqdm(range(T), desc="Refine target step", leave=False):

[luminousking]

Nice work! I was wondering if Dot supports point tracking for streaming unknown length video?

[16lemoing]

Hi @luminousking, this is not supported at the moment. In the current implementation we suppose that we have access to the full video.