Update readme

README.md

@@ -1,208 +1,36 @@
 # LaneNet-Lane-Detection
-Use tensorflow to implement a Deep Neural Network for real time lane detection mainly based on the IEEE IV conference 
-paper "Towards End-to-End Lane Detection: an Instance Segmentation Approach".You can refer to their paper for details 
-https://arxiv.org/abs/1802.05591. This model consists of a encoder-decoder stage, binary semantic segmentation stage 
-and instance semantic segmentation using discriminative loss function for real time lane detection task.
 
-The main network architecture is as follows:
+We modified the code in order to evaluate the performance on the tusimple dataset. In order to do so, we needed to
+predict the x coordinates of the lanes at given y coordinates:
 
-`Network Architecture`
-![NetWork_Architecture](./data/source_image/network_architecture.png)
+H_SAMPLES = [160, 170, 180, ..., 700, 710]
 
-## Installation
-This software has only been tested on ubuntu 16.04(x64), python3.5, cuda-9.0, cudnn-7.0 with a GTX-1070 GPU. 
-To install this software you need tensorflow 1.10.0 and other version of tensorflow has not been tested but I think 
-it will be able to work properly in tensorflow above version 1.10. Other required package you may install them by
+The tuSimple detection challenge requires us to return a list of x coordinates for each lane. For example if there are
+two lanes, the output might be:
 
-```
-pip3 install -r requirements.txt
-```
 
-## Test model
-In this repo I uploaded a model trained on tusimple lane dataset [Tusimple_Lane_Detection](http://benchmark.tusimple.ai/#/).
-The deep neural network inference part can achieve around a 50fps which is similar to the description in the paper. But
-the input pipeline I implemented now need to be improved to achieve a real time lane detection system.
+We modified the code in lanenet_model/lanenet_postprocess.py to compute the predicted x coordinates based on the fitted
+polynomial. This was challenging since the polynomial is fitted after applying a perspective transform, so to get the
+x coordinates in the source image we had to transform the fitted polynomial back using the inverse transform.
 
-The trained lanenet model weights files are stored in 
-[new_lanenet_model_file](https://www.dropbox.com/sh/tnsf0lw6psszvy4/AAA81r53jpUI3wLsRW6TiPCya?dl=0). You can 
-download the model and put them in folder model/tusimple_lanenet/
+To run the algorithm to find the predicted labels, we modified the script tools/evaluate_lanenet_on_tusimple.py. We ran
+the pretrained network in Google colab. The notebook can be accessed here:
 
-You can test a single image on the trained model as follows
+https://colab.research.google.com/drive/1rGEwNYzrQWybjtZGBohw3_45QdUCXlIv
 
-```
-python tools/test_lanenet.py --weights_path ./model/tusimple_lanenet_vgg/tusimple_lanenet_vgg.ckpt 
---image_path ./data/tusimple_test_image/0.jpg
-```
-The results are as follows:
+Here is the command to evaluate the model:
 
-`Test Input Image`
+!python tools/evaluate_lanenet_on_tusimple.py \
+  --image_dir <path to folder containing test images> \
+  --weights_path <path to lanenet weights> \
+  --save_dir <directory to save output>
 
-![Test Input](./data/tusimple_test_image/0.jpg)
+The result of the above command is stored in 'data/predictions.json'.
 
-`Test Lane Mask Image`
+We then wrote a script to evaluate the prediction accuracy. This is in the file tools/evaluate_performance_tusimple.py.
 
-![Test Lane_Mask](./data/source_image/lanenet_mask_result.png)
+It does not have any arguments and can be run to see the performance of the algorithm on a set of test images as
+compared with the ground truth provided by the tuSimple dataset.
 
-`Test Lane Binary Segmentation Image`
-
-![Test Lane_Binary_Seg](./data/source_image/lanenet_binary_seg.png)
-
-`Test Lane Instance Segmentation Image`
-
-![Test Lane_Instance_Seg](./data/source_image/lanenet_instance_seg.png)
-
-If you want to evaluate the model on the whole tusimple test dataset you may call
-```
-python tools/evaluate_lanenet_on_tusimple.py 
---image_dir ROOT_DIR/TUSIMPLE_DATASET/test_set/clips 
---weights_path ./model/tusimple_lanenet_vgg/tusimple_lanenet.ckpt 
---save_dir ROOT_DIR/TUSIMPLE_DATASET/test_set/test_output
-```
-If you set the save_dir argument the result will be saved in that folder 
-or the result will not be saved but be 
-displayed during the inference process holding on 3 seconds per image. 
-I test the model on the whole tusimple lane 
-detection dataset and make it a video. You may catch a glimpse of it bellow.
-
-`Tusimple test dataset gif`
-![tusimple_batch_test_gif](./data/source_image/lanenet_batch_test.gif)
-
-## Train your own model
-#### Data Preparation
-Firstly you need to organize your training data refer to the data/training_data_example folder structure. And you need 
-to generate a train.txt and a val.txt to record the data used for training the model. 
-
-The training samples are consist of three components. A binary segmentation label file and a instance segmentation label
-file and the original image. The binary segmentation use 255 to represent the lane field and 0 for the rest. The 
-instance use different pixel value to represent different lane field and 0 for the rest.
-
-All your training image will be scaled into the same scale according to the config file.
-
-Use the script here to generate the tensorflow records file
-
-```
-python data_provider/lanenet_data_feed_pipline.py 
---dataset_dir ./data/training_data_example
---tfrecords_dir ./data/training_data_example/tfrecords
-```
-
-#### Train model
-In my experiment the training epochs are 80010, batch size is 4, initialized learning rate is 0.001 and use polynomial 
-decay with power 0.9. About training parameters you can check the global_configuration/config.py for details. 
-You can switch --net argument to change the base encoder stage. If you choose --net vgg then the vgg16 will be used as 
-the base encoder stage and a pretrained parameters will be loaded. And you can modified the training 
-script to load your own pretrained parameters or you can implement your own base encoder stage. 
-You may call the following script to train your own model
-
-```
-python tools/train_lanenet.py 
---net vgg 
---dataset_dir ./data/training_data_example
--m 0
-```
-You can also continue the training process from the snapshot by
-```
-python tools/train_lanenet.py 
---net vgg 
---dataset_dir data/training_data_example/ 
---weights_path path/to/your/last/checkpoint
--m 0
-```
-
-You may monitor the training process using tensorboard tools
-
-During my experiment the `Total loss` drops as follows:  
-![Training loss](./data/source_image/total_loss.png)
-
-The `Binary Segmentation loss` drops as follows:  
-![Training binary_seg_loss](./data/source_image/binary_seg_loss.png)
-
-The `Instance Segmentation loss` drops as follows:  
-![Training instance_seg_loss](./data/source_image/instance_seg_loss.png)
-
-## Experiment
-The accuracy during training process rises as follows: 
-![Training accuracy](./data/source_image/accuracy.png)
-
-Please cite my repo [lanenet-lane-detection](https://github.com/MaybeShewill-CV/lanenet-lane-detection) if you use it.
-
-## Recently updates 2018.11.10
-Adjust some basic cnn op according to the new tensorflow api. Use the 
-traditional SGD optimizer to optimize the whole model instead of the
-origin Adam optimizer used in the origin paper. I have found that the
-SGD optimizer will lead to more stable training process and will not 
-easily stuck into nan loss which may often happen when using the origin
-code.
-
-I have uploaded a new lanenet model trained on tusimple dataset using the
-new code here [new_lanenet_model_file](https://www.dropbox.com/sh/tnsf0lw6psszvy4/AAA81r53jpUI3wLsRW6TiPCya?dl=0).
-You may download the new model weights and update the new code. To update
-the new code you just need to
-
-```
-git pull origin master
-```
-The rest are just the same as which mentioned above. And recently I will 
-release a new model trained on culane dataset.
-
-## Recently updates 2018.12.13
-Since a lot of user want a automatic tools to generate the training samples
-from the Tusimple Dataset. I upload the tools I use to generate the training
-samples. You need to firstly download the Tusimple dataset and unzip the 
-file to your local disk. Then run the following command to generate the 
-training samples and the train.txt file.
-
-```angular2html
-python tools/generate_tusimple_dataset.py --src_dir path/to/your/unzipped/file
-```
-
-The script will make the train folder and the test folder. The training 
-samples of origin rgb image, binary label image, instance label image will
-be automatically generated in the training/gt_image, training/gt_binary_image,
-training/gt_instance_image folder.You may check it yourself before start
-the training process.
-
-Pay attention that the script only process the training samples and you 
-need to select several lines from the train.txt to generate your own 
-val.txt file. In order to obtain the test images you can modify the 
-script on your own.
-
-## Recently updates 2019.05.16
-
-New model weights can be found [here](https://www.dropbox.com/sh/tnsf0lw6psszvy4/AAA81r53jpUI3wLsRW6TiPCya?dl=0)
-
-## MNN Project
-
-Add tools to convert lanenet tensorflow ckpt model into mnn model and deploy
-the model on mobile device
-
-#### Freeze your tensorflow ckpt model weights file
-```
-cd LANENET_PROJECT_ROOT_DIR
-python mnn_project/freeze_lanenet_model.py -w lanenet.ckpt -s lanenet.pb
-```
-
-#### Convert pb model into mnn model
-```
-cd MNN_PROJECT_ROOT_DIR/tools/converter/build
-./MNNConver -f TF --modelFile lanenet.pb --MNNModel lanenet.mnn --bizCode MNN
-```
-
-#### Add lanenet source code into MNN project 
-
-Add lanenet source code into MNN project and modified CMakeList.txt to 
-compile the executable binary file.
-
-## TODO
-- [x] Add a embedding visualization tools to visualize the embedding feature map
-- [x] Add detailed explanation of training the components of lanenet separately.
-- [x] Training the model on different dataset
-- ~~[ ] Adjust the lanenet hnet model and merge the hnet model to the main lanenet model~~
-- ~~[ ] Change the normalization function from BN to GN~~
-
-## Acknowledgement
-
-The lanenet project refers to the following projects:
-
-- [MNN](https://github.com/alibaba/MNN)
-- [SimpleDBSCAN](https://github.com/CallmeNezha/SimpleDBSCAN)
+python
+tools/evaluate_performance_tusimple.py

data/tusimple-sample/test_tasks_0627.json

@@ -0,0 +1,12 @@
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626047222176976_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626126171818168_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626127172745520_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626153155598528_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626155156451704_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626158152981904_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626166147797438_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626171146236124_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626191132352208_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626199127566374_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626224112349377_0"}
+{"h_samples":[160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710],"lanes":[],"run_time":1000,"raw_file":"tusimple-sample/clips/0530/1492626236105069364_0"}