ready to release

Author: MoonBlvd

README.md

@@ -2,11 +2,11 @@
 
 *Yu Yao, Mingze Xu, Yuchen Wang, David Crandall and Ella Atkins*
 
-This repo contains the code for our [paper](https://arxiv.org/pdf/1903.00618.pdf) on unsupervised traffic accident detection.
+This repo contains the code for our [IROS2019 paper](https://arxiv.org/pdf/1903.00618.pdf) on unsupervised traffic accident detection.
 
-:boom: The full code will be released upon the acceptance of our paper.
+:boom: The code and A3D dataset is released here! 
 
-:boom: So far we have released the pytorch implementation of our ICRA paper [*Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems*](https://arxiv.org/pdf/1809.07408.pdf), which is an important building block for the traffic accident detection. The original project repo is https://github.com/MoonBlvd/fvl-ICRA2019
+This code also contains a improved pytorch implementation of our ICRA paper [*Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems*](https://arxiv.org/pdf/1809.07408.pdf), which is an important building block for the traffic accident detection. The original project repo is https://github.com/MoonBlvd/fvl-ICRA2019
 
 <img src="figures/teaser.png" width="400">
 
@@ -17,9 +17,44 @@ To run the code on feature-ready HEV-I dataset or dataset prepared in HEV-I styl
 	pytorch 1.0
 	torchsummaryX
 	tensorboardX
+## Train and test
+Note that we apply a FOL and ego-motion prediction model to do unsupervised anomaly detection. Thus model training is to train the FOL and ego-motion prediction model on normal driving dataset. We haved used HEV-I as the training set.
+### Train
+The training script and a config file template are provided:
+
+	python train.py --load_config config/fol_ego_train.yaml
+
+### Run FOL on test set and then Anomaly Detection
+For evaluation purpose, we firstly run our fol_ego model on test dataset, e.g. A3D to generate all predictions
+
+	python run_fol_for_AD.py --load_config config/test_A3D.yaml
+
+This will save one ```.pkl``` file for each video clip. Then we can use the saved predictions to calculate anomaly detection metrics. The following command will print results similar to the paper.
+
+	python run_AD.py --load_config config/test_A3D.yaml
+
+The online anomaly detection script is not provided, but the users are free to write another script to do FOL and anomaly detection online. 
+
 ## Dataset and features
+## A3D dataset
+The A3D dataset contains videos from YouTube and a ```.pkl``` file including human annotated video start/end time and anomaly start/end time. We provide scripts and url files to download the videos and run pre-process to get the same images we haved used in the paper.
+
+Download the videos from YouTube:
+
+	python datasets/A3D_download.py --download_dir VIDEO_DIR --url_file datasets/A3D_urls.txt
+
+Then convert the videos to images in 10Hz
+
+	python scripts/video2frames.py -v VIDEO_DIR -f 10 -o IMAGE_DIR -e jpg
+
+Note that each downloaded video is a combination of several short clips, to split them into clips we used, run:
+
+	python datasets/A3D_split.py --root_dir DATA_ROOT --label_dir DIR_TO_PKL_LABEL
+
+The annotations can be downloaded from here. 
+
 ### HEV-I dataset
-**Note:**  Honda Research Institute is still working on preparing the videos in HEV-I dataset. The planned release date will be around May 20 2019 during the ICRA.
+[Honda Egocentric View-Intersection (HEV-I)](https://usa.honda-ri.com/ca/hevi) dataset is owned by HRI and the users can follow the link to request the dataset.
 
 However, we provide the newly generated features here in case you are interested in just using the input features to test your models:
 
@@ -40,8 +75,6 @@ To prepare the features used in this work, we used:
 * Dense optical flow: [FlowNet2.0](https://github.com/NVIDIA/flownet2-pytorch)
 * Ego motion: [ORBSLAM2](https://github.com/raulmur/ORB_SLAM2)
 
-### A3D dataset
-The A3D dataset will be released upon the acceptance of our IROS submission.
 
 ## Future Object Localization
 

config/test_A3D.yaml

@@ -10,14 +10,13 @@ best_fol_model: '/home/brianyao/Documents/tad-IROS2019/checkpoints/fvl_ego_check
 best_ego_pred_model: '/home/brianyao/Documents/tad-IROS2019/checkpoints/fvl_ego_checkpoints/ego_pred_epoch_055_loss_0.0016.pt'
 
 test_dataset: "A3D" 
-test_root: "../data/A3D/frames" 
-label_file: '../data/A3D/A3D_labels.pkl'
+label_file: '/home/brianyao/Documents/tad-IROS2019/datasets/A3D_labels.pkl'
 
 track_dir: "/media/DATA/A3D/deep_sort_clear"
 flow_dir: "/media/DATA/A3D/flownet2"
 ego_motion_dir: "/media/DATA/A3D/ego_motion"
 img_dir: "/media/DATA/A3D/frames"
-save_dir: "/media/DATA/A3D/multi_prediction_results"
+save_dir: "/media/DATA/A3D/fvl_ego_results"
 
 # dataset arguments
 seed_max: 5

datasets/A3D_download.py

@@ -0,0 +1,81 @@
+import sys
+import os
+import glob
+import argparse
+import yaml
+import cv2
+import youtube_dl
+
+parser = argparse.ArgumentParser(description='AnAnXingChe video downloader parameters.')
+parser.add_argument('--download_dir', required=True, help='target directory to save downloaded videos')
+parser.add_argument('--url_file', required=True, help='a .txt file saving urls to all videos')
+parser.add_argument('--to_images', type=bool, default=False, help='downsample the video and save image frames, default is false')
+parser.add_argument('--img_dir', help='target directory to save downsampled')
+parser.add_argument('--img_ext', default='jpg', help='image extension')
+parser.add_argument('--downsample_rate', default=3.0, type=float, help='downsample rate')
+
+args = parser.parse_args()
+
+DOWNLOAD_DIR = args.download_dir
+
+
+# Download videos
+if not os.path.isdir(DOWNLOAD_DIR):
+    print("The indicated download directory does not exist!")
+    print("Directory made!")
+    os.makedirs(DOWNLOAD_DIR)
+
+'''Download videos'''
+ydl_opt = {'outtmpl': DOWNLOAD_DIR + '%(id)s.%(ext)s',
+           'format': 'mp4'}
+ydl = youtube_dl.YoutubeDL(ydl_opt)
+'''
+with ydl:
+    result = ydl.extract_info(
+        'https://www.youtube.com/channel/UC-Oa3wml6F3YcptlFwaLgDA',
+        download=True # We just want to extract the info
+    )
+'''
+url_list = open(args.url_file,'r').readlines()
+ydl.download(url_list)
+print("Download finished!")
+
+all_videos = sorted(glob.glob(DOWNLOAD_DIR + '*.mp4'))
+print("Number of videos: ", len(all_videos))
+
+if args.to_images:
+    IMAGE_DIR = args.img_dir
+    # Downsample the saved videos and save images to another directory
+    try:
+        os.stat(IMAGE_DIR)
+    except:
+        print("The indicated image directory does not exist!")
+        print("Directory made!")
+        os.mkdir(IMAGE_DIR)
+
+    downsample_rate = args.downsample_rate
+    for video_idx, file_name in enumerate(all_videos):
+        video_name = file_name.split('/')[-1][:-4]
+        image_dir = OUT_DIR + video_name + '/'
+        try:
+            os.stat(image_dir)
+        except:
+            os.mkdir(image_dir)
+
+        cap = cv2.VideoCapture(file_name)
+        length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+        fps = float(cap.get(cv2.CAP_PROP_FPS))
+        print("Number of frames: ", length)
+        print("FPS: ", fps)
+        i = 0
+        j = 0
+        while True:
+            # Capture frame-by-frame
+            ret, image = cap.read()
+            if not ret:
+                break
+            if i%downsample_rate == 0:
+                img_name = str(format(j,'06')) + '.' + args.img_ext
+                j += 1
+                cv2.imwrite(image_dir + img_name, image)
+            i += 1

datasets/A3D_labels.pkl

@@ -0,0 +1,41 @@
+'''
+Read the pickle file that saves label of A3D dataset.
+Save the images of each short video clip separately and prepare to run MaskRCNN and flownet2
+
+Feb 19 2019
+
+Assume each video's frames are saved in: ROOT_DIR + '/images/xxxxxx'
+
+'''
+import os
+import pickle as pkl
+import shutil
+import argparse
+
+parser = argparse.ArgumentParser(description='A3D video split parameters.')
+parser.add_argument('--root_dir', required=True, help='the root directory of the dataset')
+parser.add_argument('--label_dir', required=True, help='the pkl label file')
+args = parser.parse_args()
+
+data = pkl.load(open(args.label_dir,'rb'))
+for key, value in data.items():
+    video_name = key
+    video_dir = os.path.join(args.root_dir, 'images', value['video_name'])
+
+    out_dir = os.path.join(args.root_dir, 'frames', video_name)
+    if not os.path.isdir(out_dir):
+        os.mkdir(out_dir)
+
+    out_dir = os.path.join(out_dir, 'images')
+    if not os.path.isdir(out_dir):
+        os.mkdir(out_dir)
+
+    start = value['clip_start']
+    end = value['clip_end']
+    for new_i, old_i in enumerate(range(int(start), int(end)+1)):
+        img_name = str(old_i).zfill(6) + '.jpg'
+        old_img_path = os.path.join(video_dir, img_name)
+        new_img_path = os.path.join(out_dir, str(new_i + 1).zfill(6) + '.jpg')
+        shutil.copy(old_img_path, new_img_path)
+
+