VPTR: Efficient Transformers for Video Prediction

Video future frames prediction based on Transformers

Training

Stage 1: train_AutoEncoder.py

Train the autoencoder firstly, save the ckpt, load it for stage 2

Stage 2: Train Transformer for the video prediction

train_FAR.py: Fully autoregressive model
train_FAR_mp.py: multiple gpu training (single machine)
train_NAR.py: Non-autoregressive model
train_NAR_mp.py: multiple gpu training (single machine)

Dataset folder structure

/MovingMNIST
     moving-mnist-train.npz
     moving-mnist-test.npz
     moving-mnist-val.npz \

/KTH
     boxing/
         person01_boxing_d1/
            image_0001.png
            image_0002.png
            ...
         person01_boxing_d2/
            image_0001.png
            image_0002.png
            ... \

     handclapping/
         ...
     handwaving/
         ...
     jogging_no_empty/
         ...
     running_no_empty/
         ...
     walking_no_empty/
         ...

/BAIR
     test/
         example_0/
            0000.png
            0001.png
            ...
         example_1/
            0000.png
            0001.png
            ...
         example_...
     train/
         example_0/
            0000.png
            0001.png
            ...
         example_... \