e2e: add end to end solution.

modules/e2e/model/README.md

@@ -0,0 +1,11 @@
+# Quick Start Guide
+## Train Model
+	sh run_train.sh 
+The trainsets is downloaded to the current project ./trainsets directory.
+## Model predict
+	sh run_predict.sh path/to/steering_model path/to/acc_model path/to/test_img t/path/to/result 
+
+## About
+* Data Set Download：<https://console.bce.baidu.com/apollo/task/download#/apollo/home> 
+* Data Set Summary：please view data_overview.md 
+* Model Summary：please view model_overview.md

modules/e2e/model/README_cn.md

@@ -0,0 +1,11 @@
+# 快速入门
+## 训练模型
+	sh run_train.sh 
+训练集下载到当前工程./trainsets目录下。
+## 模型预测
+	sh run_predict.sh path/to/steering_model path/to/acc_model path/to/test_img t/path/to/result 
+
+## 关于
+* 数据集下载：请参考 <https://console.bce.baidu.com/apollo/task/download#/apollo/home> 
+* 数据集概览：请参考 data_overview.md 
+* 模型概览：请参考 model_overview.md

modules/e2e/model/data_overview.md

@@ -0,0 +1,106 @@
+This document describes the data format and relevant evaluation criteria of the end-to-end driving platform in Baidu Apollo Project.
+
+# Introduction of datasets
+
+The data are collected through Baidu’s own map collection vehicles. At present, the data cover the entire road network in China with a total length of millions of kilometers. This dataset provides two types of data including images in front of vehicles and the vehicle motion status. The map collection vehicles capture 360-degree view images. However, due to the limits of file sizes, we only provide the jpg format of images in front of vehicles with the 320 * 320 resolution. The vehicle motion status data include the current speed and the track curvature.
+
+# Acquisition situation
+
+The data acquisition equipment is as shown in the figure. The CCD camera aperture should be 2.8 and the focal length should be ∞. The camera is located in the cover of the car roof. Adjust the front and rear positions of the luggage rack buckle to ensure that the luggage rack itself is perpendicular to the centerline of the vehicle and the two luggage racks are on the same level. Move the equipment bracket to the middle of the luggage rack to ensure that the middle of CCD images coincides with the centerline of the vehicle.
+
+ <img src="./png_resource/1.png" width = "300" height = "320" align=center /> 
+ <img src="./png_resource/2.png" width = "300" height = "320" align=center />
+
+# Data usage
+
+The data comes from the original data of the sensor, including images, laser radars, radars, etc., which are mainly input in the form of images. It outputs the vehicle’s control instructions, such as the steering wheel angle, acceleration, and braking. The input and output are connected through the deep neural network, that is, to directly generate the vehicle control instructions through the neural network to carry out the horizontal control and vertical control of the vehicle. There are no logic programs manually participated. The horizontal control mainly controls the horizontal movements of the vehicle through the steering wheel, that is, the steering wheel angle. The vertical control controls the vertical movements of the vehicle through the throttle and the brake, that is, acceleration and braking.the output of horizontal control model does not use the steering wheel angle. It uses the curvature to be driven (ie, the reciprocal of the turning radius). The reasons are as follows:
+
+(1) The curvature is more universally applicable, which is not affected by the vehicle’s own parameters such as steering ratio, wheel base and so on.
+
+(2) The relationship between the curvature and the steering wheel angle is simple, which can be retrieved through the Ackermann model at a low speed and be fitted through a simple network at a high speed.
+So the horizontal control model obtained is: give the curvature of the vehicle for the driving through the forward images.
+
+# Data classification
+
+The data are divided into two parts, including the training set and the test set. Use the training set to debug algorithms and sue the test set to verify results.
+
+## Training sets
+
+The training set contains two parts of data, including image and attr. Image is the input data and attr is the output data. Among them, image files and attr files correspond to each other through the file names. To read image and attr, it needs the support of the hdf5 library.
+
+### Formats of training sets
+* The training data are organized according to the following directory structure:
+
+	```
+trainsets/				   // top folders of the training data
+├── image					 // it includes the image files in the training data
+│   ├── 119.h5
+│   ├── ..
+│   └── 123.h5
+└── attr			// it includes the attribute files corresponding to image files
+    ├── 119.h5
+    ├── ..
+    └── 123.h5   
+	```
+
+
+* The format of the image files
+.h5 file. Key-Value, which retrieves an image with the timestamp UTC as the index, Key:UTC time, Value:the encoded jpg format of 320*320*3 pixel matrix jpg.Decoding example：
+
+	```
+	with h5py.File(img_file, 'r') as f:
+		for t in f:
+			img = cv2.imdecode(f[t][:], 1)
+
+	```
+
+* The format of the attribute files
+.h5 file, the attribute data of one moment will be stored in the hdf5 in the form of 2D array as a whole. The first dimension is ’attrs’ and the second dimension is the attribute data:［t, VEast, VNorth, curv1, curv2, curv3, curv4, curv5, curv6, x, y, heading,tag］. For an image of the UTC time in the image file, there must be a line of attribute data corresponding to the image. Each line has 13 items of data in the form of 64-bit floating point number. Variables and related instructions are described as follows:
+
+	Column: variables | Units | Description
+	------------ | ------------- | ------------
+	01 : t | decimal system (Unsigned) | Current UTC timestamp
+	02 : VEast|	m/s | Current speed of the vehicle towards the east
+	03 : VNorth|	m/s | Current speed of the vehicle towards the north
+	04 : curv1 | decimal system (Signed) | [t,t+1] Curvature 1, left turning as positive
+	05 : curv2 | decimal system (Signed) | [t,t+1] Curvature 2, left turning as positive
+	06 : curv3 | decimal system (Signed) | [t,t+1] Curvature 3, left turning as positive
+	07 : curv4 | decimal system (Signed) | [t,t+1] Curvature 4, left turning as positive
+	08 : curv5 | decimal system (Signed) | [t,t+1] Curvature 5, left turning as positive
+	09 : curv6 | decimal system (Signed） | [t,t+1] Curvature 6, left turning as positive
+	10 : x | decimal system (Unsigned) | relative displacement against the east x-axis
+	11 : y | decimal system (Unsigned) | relative displacement against the north y-axis
+	12 : heading | degree, decimal floating-point number (Signed） | the clockwise angle to the north
+	13 : tag | decimal system (Unsigned) | Reserved annotation bit, not yet used
+
+# Formats of test sets:
+
+* The format of the test set is consistent with that of the training set. It is organized as follows:
+
+
+	```
+testsets/				   // top folders of the test data
+├── image					 // it includes the image files in the test data
+│   ├── testfile_part01.h5
+│   ├── ..
+│   └── testfile_part04.h5
+└── attr			// it includes the attribute files corresponding to image files
+    ├── testfile_part01.h5
+    ├── ..
+    └── testfile_part04.h5 
+
+	```
+* Write the results in the .h5 file and save in the hdf5 in the form of 2D array as a whole. Save the floating-point numbers. There are two columns, which are defined as follows:
+
+   Column: variables | Units | Description
+	------------ | ------------- | ------------
+	01 : t | decimal system (Unsigned) | Current UTC timestamp 
+	02 : curv|	decimal system (Signed)|	[t,t+1] Curvature, left turning as positive
+	03: acc	|decimal system (Signed)）|[t,t+1] Acceleration, acceleration  as positive
+
+# Descriptions of Evaluation Indicators
+
+The horizontal control model and the horizontal control modelis measured by using the mean square error (MSE) indicator. The mean square error is the quadratic mean of the predicted value and the true value difference. The smaller the mean variance error, the better the effectThis indicator is used to measure the prediction precision, which is defined as follows:
+<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default"></script>
+
+$$ MSE = \frac1{n}\sum_{i=1}^n{(truth- predict )^2} $$

modules/e2e/model/data_overview_cn.md

@@ -0,0 +1,75 @@
+本文档描述了百度阿波罗计划中端到端驾驶平台的数据格式及相关评测标准。
+
+# 数据集简介
+
+数据是通过百度自己的地图采集车采集而来。 目前，该数据覆盖了中国整个公路网，总长达百万公里。本数据集提供车前图像和车辆运动状态两种类型数据。地图采集车辆捕获360度视图图像，但是由于文件大小限制，我们只提供320 * 320分辨率的车前图像jpg格式数据。 车辆运动状态数据包括当前速度和轨迹曲率。
+# 采集情况
+数据采集设备如图所示，CCD 相机的光圈应为 2.8，焦距应为∞。摄像头位于车上方顶盖中。通过调整行李架卡扣前后位置,保证行李架本身与车辆中心线垂直，且两条行李架在同一水平面上。将设备支架移动至行李支架中间，保证 CCD 拍摄图像的中间与车辆中心线重合。
+ <img src="./png_resource/1.png" width = "300" height = "320" align=center /> 
+ <img src="./png_resource/2.png" width = "300" height = "320" align=center />
+# 数据用途
+数据主要来源于传感器的原始数据，包括图像、激光雷达、雷达等。end-to-end输入以图像为主。 输出是车辆的控制决策指令，如方向盘角度、加速、刹车。 连接输入输出的是深度神经网络，即通过神经网络直接生成车辆控制指令对车辆进行横向控制和纵向控制，中间没有人工参与的逻辑程序。横向控制，主要是指通过方向盘控制车身横向移动，即方向盘角度。纵向控制，是指通过油门和刹车控制车身纵向的移动，即加速、刹车等。横向模型的输出没有采用方向盘角度，而是使用要行驶的曲率（即拐弯半径的倒数）。原因如下：
+（1）曲率更普适，不受车辆自身参数如steering ratio、wheel base等影响。
+（2）曲率与方向盘角度之间的函数关系简单，低速时通过Ackermann模型就可以反演，高速时通过一个简单的网络也可以拟合。故得到的横向控制模型就是：通过前向的影像，给出车辆要行驶的曲率。纵向模型的输出是加速度。
+
+
+
+
+# 数据分类
+数据分为两部分，训练集和测试集。使用训练集调试算法，使用测试集检验模型。
+## 训练集
+训练集包含图像image和姿态attr两部分数据。image为输入数据，attr为输出数据。其中image文件和attr文件通过文件名一一对应，读取image和attr需要hdf5库支持。
+### 训练集格式
+* 训练数据按照如下目录结构组织：
+	```
+trainsets/				    // 训练数据的顶层文件夹
+├── image					 // 包含了训练数据中的图像文件
+│   ├── 119.h5
+│   ├── ..
+│   └── 123.h5
+└── attr					// 包含了与图像文件对应的姿态文件
+    ├── 119.h5
+    ├── ..
+    └── 123.h5   
+	```
+
+
+* 图像文件：h5文件。key-value格式，以时间戳UTC time为索引检索一幅图像，key:UTC time， value:320 * 320 * 3的图像编码后jpg格式数据。解码示例：
+
+	```
+	with h5py.File(img_file, 'r') as f:
+		for t in f:
+			img = cv2.imdecode(f[t][:], 1)
+
+	```
+
+* 姿态文件：h5文件。一个时刻的姿态数据以二维数组整体存在hdf5中，第一维为’attrs’,第二维为姿态数据：［t, VEast, VNorth, curv1, curv2, curv3, curv4, curv5, curv6, x, y, heading, tag］。对于图像文件中的一个UTC time的图像，一定存在一行与之对应的姿态数据，每一行有13个数据，类型为64位浮点数。字段说明如下：
+
+	列：变量 | 单位 | 说明
+	------------ | ------------- | ------------
+	01 : t | 十进制（无符号） | 当前UTC time 时间戳
+	02 : VEast|	米／秒 | 当前车辆朝东的速度
+	03 : VNorth|	米／秒 | 当前车辆朝北的速度	04 : curv1 | 十进制（有符号) | [t,t+0.125)间曲率1，左拐为正	05 : curv2 | 十进制（有符号） | [t,t+0.125)间曲率2，左拐为正	06 : curv3 | 十进制（有符号） | [t,t+0.125)间曲率3，左拐为正	07 : curv4 | 十进制（有符号） | [t,t+0.125)间曲率4，左拐为正	08 : curv5 | 十进制（有符号） | [t,t+0.125)间曲率5，左拐为正	09 : curv6 | 十进制（有符号） | [t,t+0.125)间曲率6，左拐为正	10 : x | 十进制（无符号） | 与正东向x轴的相对位移	11 : y | 十进制（无符号） | 与正北向y轴的相对位移	12 : heading | 度，十进制浮点数（有符号） | 与北向的顺时针夹角	13 : tag | 十进制（无符号） | 保留的标记位，暂未使用
+
+# 评测数据格式
+* 测试集只包含图像数据，格式同训练集。测试数据按照如下目录结构组织：	```testsets/				    // 测试数据的顶层文件夹
+├── image					 // 包含了测试数据中的图像文件
+│   ├── testfile_part01.h5
+│   ├── ..
+│   └── testfile_part04.h5
+└── attr					// 包含了与图像文件对应的姿态文件
+    ├── testfile_part01.h5
+    ├── ..
+    └── testfile_part04.h5 	```
+* 使用测试集预测曲率和加速度。预测结果写入.h5文件，二维数组整体存在hdf5中，浮点数存储。
+   列：变量 | 单位 | 说明
+	------------ | ------------- | ------------
+	01 : t | 十进制（无符号） | 当前UTC time 时间戳
+	02 : curv|	十进制（有符号）|	[t,t+0.125）间曲率，左拐为正
+	03: acc	|十进制（有符号）|[t,t+0.125）间加速度，加速为正
+
+# 评测指标说明
+横纵向模型预测效果使用均方差(MSE)指标来衡量。均方差是指预测值与真实值差值的平方的均值，均方差值越小，效果越好。该指标用来衡量预测的精确性，定义公式如下：
+<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default"></script>
+
+$$ MSE = \frac1{n}\sum_{i=1}^n{(truth- predict )^2} $$

modules/e2e/model/model_overview.md

@@ -0,0 +1,74 @@
+# Steering Model
+
+The steering wheel control model is computing curvature.
+
+`code file：run_model.py`
+## Generator Data
+
+```
+def generator_data_steering():
+```
+
+Filter the abnormal values：
+
+```
+def filter(c,v):
+	return v > 1 and c < 0.5
+	
+```
+Sequentially generator the data, due to the collected data contains a large number of straight lines data(curvature less than or equal to 0.0001). After filtering the abnormal values, select the straight line data by probability of 10%.
+
+```
+if filter(curv, speed) and (curv > 0.0001 or random.randint(1, 10) == 1)
+```
+## CNN Model
+`model file：src\steering_model.py`
+
+<img src="./png_resource/steering_model1.png" width = "300" height = "320" align=center />
+<img src="./png_resource/steering_model2.png" width = "300" height = "320" align=center />
+<img src="./png_resource/steering_model3.png" width = "300" height = "320" align=center />
+
+
+
+# Longitudinal Model
+The longitudinal model is computing the acceleration.
+
+`code file：run_model.py`
+## Generator Data
+
+```
+def generator_data_acc():
+```
+
+Filter the abnormal values：
+
+```
+def filter(c,v):
+	return v > 1 and c < 0.5
+	
+```
+Sequentially read the data, defines time_step = 5, filters the abnormal datas, selects the five-frame image of the order, and compute the acceleration value at the time of the fifth frame as the input.Each frame interval of 0.125s, the fifth frame of the acceleration:
+
+<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default"></script>
+
+$$ ACC_5 = \frac{(V_6-V_4)}\{0.25} $$
+
+
+If the sixth frame velocity value is filtered out by the filter, then discard the sample of the five frames.
+
+```
+next_curv = attrs[i+1][4]
+next_speed = math.sqrt(attrs[i+1][1] ** 2 + attrs[i+1][2] ** 2)
+if not filter(next_curv, next_speed):
+	step_cnt = 0
+	continue
+else:
+	acc = (next_speed - step_v[3]) / t_sep
+
+```
+
+## Conv_LSTM模型
+`model file:src\acc_model.py`
+
+<img src="./png_resource/acc_model.png" width = "300" height = "320" align=center />
+

modules/e2e/model/model_overview_cn.md

@@ -0,0 +1,75 @@
+#模型说明文档
+## 横向模型
+
+横向模型即计算曲率模型。
+
+`代码文件：run_model.py`
+### 生成数据
+
+```
+def generator_data_steering():
+```
+
+异常值过滤：
+
+```
+def filter(c,v):
+	return v > 1 and c < 0.5
+	
+```
+顺序生成数据，因采集的数据含有大量直行（曲率小于等于0.0001）的数据。过滤异常值后，选取10%的直行数据。
+
+```
+if filter(curv, speed) and (curv > 0.0001 or random.randint(1, 10) == 1)
+```
+### CNN模型
+`模型文件：src\steering_model.py`
+
+<img src="./png_resource/steering_model1.png" width = "300" height = "320" align=center />
+<img src="./png_resource/steering_model2.png" width = "300" height = "320" align=center />
+<img src="./png_resource/steering_model3.png" width = "300" height = "320" align=center />
+
+
+
+## 纵向模型
+纵向模型即计算加速度。
+
+`代码文件：run_model.py`
+### 生成数据
+
+```
+def generator_data_acc():
+```
+
+异常值过滤：
+
+```
+def filter(c,v):
+	return v > 1 and c < 0.5
+	
+```
+顺序读取数据，定义time_step＝5，过滤异常数据，选取顺序的五帧图，求得的第五帧图时刻的加速度值作为输入。
+每帧图间隔0.125s，第五帧的加速度
+
+<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=default"></script>
+
+$$ ACC_5 = \frac{(V_6-V_4)}\{0.25} $$
+
+如果第六帧速度值经filter被过滤掉，则丢弃该样本，即这五帧图。
+
+```
+next_curv = attrs[i+1][4]
+next_speed = math.sqrt(attrs[i+1][1] ** 2 + attrs[i+1][2] ** 2)
+if not filter(next_curv, next_speed):
+	step_cnt = 0
+	continue
+else:
+	acc = (next_speed - step_v[3]) / t_sep
+
+```
+
+### Conv_LSTM模型
+`模型文件:src\acc_model.py`
+
+<img src="./png_resource/acc_model.png" width = "300" height = "320" align=center />
+

modules/e2e/model/run_predict.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+CUDA_VISIBLE_DEVICES=0 python src/run_model.py \
+    --type=predict \
+    --steering_model_path=$1 \
+    --acc_model_path=$2 \
+    --input_dir=$3 \
+    --output_path=$4 \
+    --logtype="predict"

modules/e2e/model/run_train.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+CUDA_VISIBLE_DEVICES=0 python src/run_model.py \
+    --type=train \
+    --input_dir=./trainsets \
+    --logtype=train