Skip to content

Commit

Permalink
Docs: updated "How to Install Low-Latency/Realtime Kernel"
Browse files Browse the repository at this point in the history 
Added workaround to build Nvidia driver for PREEMPT_RT kernel
  • Loading branch information
@storypku
storypku committed Dec 3, 2020
1 parent 1f2aead commit 4dc6856
Showing 1 changed file with 80 additions and 46 deletions.
126 changes: 80 additions & 46 deletions docs/howto/how_to_install_apollo_kernel.md
View file
Original file line number Diff line number Diff line change
@@ -1,66 +1,60 @@
# How to Install Low-Latency Kernel for Apollo
# How to Install Low-Latency / Realtime Kernel

This document describes the steps to install stock Ubuntu low-latency kernel
with Nvidia driver on Ubuntu 18.04.x.
This document describes the steps to install low-latency/realtime kernel with
Nvidia driver on Ubuntu 18.04.

## Before You Begin

There was once an
[Apollo Kernel](https://github.com/ApolloAuto/apollo-kernel.git) project based
on the Linux 4.4.32 source tree, which provides real-time kernel support for
running Apollo software stack in the vehicle. However, that project hasn't been
updated for long, causing a lot of trouble for the community. The Apollo team
thus decided to find alternative ways to provide low-latency/real-time kernel
support, which, should be easy to migrate to newer Linux kernels.
Low-Latency/Realtime kernel is only required for running Apollo software stack
in the vehicle. If your sole purpose was to develop/test your algorithms or to
run simulation software (e.g., LGSVL simulator) on top of Apollo, then maybe you
don't need to install low-latency/real-time kernels described at all.

The low-latency alternative they found was stock Ubuntu low-latency kernel
(available in Ubuntu repositories). Ubuntu low-latency kernel is completely
capable of low- to no- latency for running Apollo in the vehicle. Preempt
optimization is enabled in kernel configuration, and latency as low as 0.1
millisecond can and has been achieved using it.
## Ubuntu Low-Latency Kernel

In the section below, we will describe the steps to install the low latency
kernel on Ubuntu.
Ubuntu low-latency kernel (availabe in Ubuntu repository) is completely capable
of low- to no- latency for running Apollo in the vehicle. Preempt optimization
is enabled in its kernel configuration, and latency as low as 0.1 millisecond
can and has been achieved using it.

**Note**:

> If your sole purpose was to develop/test on Apollo platform, or to run
> simulation software (e.g., LGSVL simulator), then you don't need to install
> low-latency/real-time kernels at all.
## Installing Ubuntu Low-Latency Kernel

The steps required to install low-latency kernel on Ubuntu is as follows:
The steps required to install Ubuntu low-latency kernel are described below:

1. Install the latest low-latency kernel and its headers.

```bash
# Sync package index with upstream sources.list
sudo apt-get update

# Print your current kernel release
uname -r

# Check if newer Linux image is available.
apt-cache search linux-image | grep "Signed kernel image lowlatency"

# Install the latest low-latency kernel & headers.
sudo apt-get install linux-image-$(uname -r)-lowlatency linux-headers-$(uname -r)-lowlatency
```

**Note**:

> Please change `$(uname -r)` to the latest kernel image if availabe. The latest
> kernel version on Ubuntu 18.04 as of this writing (Nov 17, 2020) is `5.4.0-54`
> Please change `$(uname -r)` to the latest kernel should there be newer kernel
> packages available. (You can view newer packages availabe with
> `apt list --upgradable`.) The latest kernel version on Ubuntu 18.04 as of this
> writing (Dec 02, 2020) is `5.4.0-56`.
2. Reboot to start the low-latency kernel.

```bash
sudo reboot
```

## Install Realtime Kernel

There is a community-contributed
[Building Realtime Linux for ROS2](https://index.ros.org/doc/ros2/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2)
document for building/installing PREEMPT_RT kernel on Ubuntu 20.04, which is
applicable to Ubuntu 18.04 also. Please follow the instructions there to install
the latest stable realtime kernel first.

## Install Nvidia Driver

### Install Nvidia Driver for Low-Latency Kernel

For Ubuntu low-latency kernel, the steps to install Nvidia driver is relatively
straightforward.

1. Download and install the latest Nvidia driver from the
[CUDA Toolkit Downloads](https://developer.nvidia.com/cuda-downloads?target_os=Linux)
Page.
Expand All @@ -75,7 +69,7 @@ sudo reboot

**Note**

> You may need to regist and sign CUDA EULA before continue.
> You may need to regist and sign CUDA EULA to download Nvidia packages.
For example, below is the instructions installing Nvidia driver on x86_64 Ubuntu
18.04.5 with the "deb[local]" approach:
Expand All @@ -99,15 +93,55 @@ sudo apt-get install nvidia-driver-455

3. Run `nvidia-smi` to check if everything is OK.

## Build and Install ESD-CAN Driver (Optional)
### Install Nvidia Driver for Realtime Kernel

You should follow the
[instructions](https://github.com/ApolloAuto/apollo-kernel/blob/master/linux/ESDCAN-README.md#build--install-out-of-tree-esd-kernel-driver)
to build ESD-CAN driver if necessary.
Please follow the first step listed above first. However, as the Nvidia driver
does not support real-time kernels, running
`sudo apt-get install nvidia-driver-455` on a PREEMPT_RT kernel should partially
fail with the following error message:

## Final Words
```text
The kernel you are installing for is a PREEMPT_RT kernel!
In this article, we described briefly the steps to install Ubuntu low-latency
kernel and integrate Nvidia GPU driver with it. This kernel should be sufficient
for Apollo runtime in vehicle. However, we haven't fully test it on real
vehicles. Please use it with caution and report any issues to us.
The NVIDIA driver does not support real-time kernels. If you
are using a stock distribution kernel, please install
a variant of this kernel that does not have the PREEMPT_RT
patch set applied; if this is a custom kernel, please
install a standard Linux kernel. Then try installing the
NVIDIA kernel module again.
*** Failed PREEMPT_RT sanity check. Bailing out! ***
```

We can set `IGNORE_PREEMPT_RT_PRESENCE=1` when build Nvidia driver as a
workaround:

1. Run the following commands to build Nvidia driver:

```bash
# Change to Nvidia driver source directory
cd "$(dpkg -L nvidia-kernel-source-455 | grep -m 1 "nvidia-drm" | xargs dirname)"

# Build Nvidia driver with IGNORE_PREEMPT_RT_PRESENCE=1
sudo env NV_VERBOSE=1 \
make -j8 NV_EXCLUDE_BUILD_MODULES='' \
KERNEL_UNAME=$(uname -r) \
IGNORE_XEN_PRESENCE=1 \
IGNORE_CC_MISMATCH=1 \
IGNORE_PREEMPT_RT_PRESENCE=1 \
SYSSRC=/lib/modules/$(uname -r)/build \
LD=/usr/bin/ld.bfd \
modules

sudo mv *.ko /lib/modules/$(uname -r)/updates/dkms/
sudo depmod -a
```

2. Reboot the system
3. Run `nvidia-smi` to check if everything is OK.

## Install ESD-CAN Driver (Optional)

You should follow the
[instructions](https://github.com/ApolloAuto/apollo-kernel/blob/master/linux/ESDCAN-README.md)
to build ESD-CAN driver if necessary.

0 comments on commit 4dc6856

Please sign in to comment.