Skip to content

Commit

Permalink
add codes and other files.
Browse files Browse the repository at this point in the history
  • Loading branch information
beacer committed Oct 10, 2017
1 parent 9f61ad2 commit ee2099a
Show file tree
Hide file tree
Showing 329 changed files with 89,139 additions and 0 deletions.
29 changes: 29 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
.*.sw[a-z]
*.log
*.o
*.d
!init.d/
*.cmd
*.so
*.vim
tags
cscope*
filenametags
build/
bin/
dpdk-*
tools/ipvsadm/ipvsadm
tools/keepalived/Makefile
tools/keepalived/bin/genhash
tools/keepalived/bin/keepalived
tools/keepalived/config.status
tools/keepalived/genhash/Makefile
tools/keepalived/keepalived.spec
tools/keepalived/keepalived/Makefile
tools/keepalived/keepalived/check/Makefile
tools/keepalived/keepalived/core/Makefile
tools/keepalived/keepalived/libipvs-2.6/Makefile
tools/keepalived/keepalived/libipvs-2.6/libipvs.a
tools/keepalived/keepalived/vrrp/Makefile
tools/keepalived/lib/Makefile
tools/keepalived/lib/config.h
414 changes: 414 additions & 0 deletions LICENSE.md

Large diffs are not rendered by default.

41 changes: 41 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#
# DPVS is a software load balancer (Virtual Server) based on DPDK.
#
# Copyright (C) 2017 iQIYI (www.iqiyi.com).
# All Rights Reserved.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#

#
# Makefile for DPDK-VS (dpvs)
#
MAKE = make
CC = gcc
LD = ld

SUBDIRS = src tools

INSDIR = $(PWD)/bin
export INSDIR

export KERNEL = $(shell /bin/uname -r)

all:
for i in $(SUBDIRS); do $(MAKE) -C $$i || exit 1; done

clean:
for i in $(SUBDIRS); do $(MAKE) -C $$i clean || exit 1; done

install:all
-mkdir -p $(INSDIR)
for i in $(SUBDIRS); do $(MAKE) -C $$i install || exit 1; done
install -m 744 src/build/dpvs $(INSDIR)/dpvs
208 changes: 208 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
DPVS
----

![dpvs.png](./pic/dpvs.png)

# Introduction

`DPVS` is a high performance **Layer-4 load balancer** based on [DPDK](http://dpdk.org). It's derived from Linux [LVS](http://www.linuxvirtualserver.org/) and it's modification [alibaba/LVS](https://github.com/alibaba/LVS).

> the name `DPVS` comes from "DPDK-LVS".
Different techniques are applied for high performance:

* *Kernel by-pass* (user space implementation)
* *Share-nothing*, per-CPU for key data (Lockless)
* *RX Steering* and *CPU affinity* (avoid context switch)
* *Batching* TX/RX
* *Zero Copy* (avoid packet copy and syscalls).
* *Polling* instead of interrupt.
* *lockless message* for high performance ICP.
* other techs enhanced by *DPDK*

Major features of `DPVS` including:

* *L4 Load Balancer*, including FNAT, DR mode, etc.
* Different *schedule algorithm* like RR, WLC, WRR, etc.
* User-space *Lite IP stack* (IPv4, Routing, ARP, ICMP ...).
* *SNAT* mode for Internet access from internal network.
* Support *KNI*, *VLAN*, *Bonding* for different IDC environment.
* Security aspect, support *TCP syn-proxy*, *Conn-Limit*, *black-list*
* QoS: *Traffic Control* (Ongoing)

`DPVS` feature modules are illustrated as following picture.

![modules](./pic/modules.png)

# Quich Start

## Clone DPVS

```bash
% git clone ...
% cd dpvs
```

Well, let's start from DPDK then.

## DPDK setup.

Currently, `DPDK-16.07` is used for `DPVS` and latest `DPDK` stable version will be integrated soon.

```bash
% wget http://fast.dpdk.org/rel/dpdk-16.07.2.tar.xz # download form dpdk.org if link failed.
% tar vxf dpdk-16.07.2.tar.xz
```

There's a patch for DPDK `kni` driver for hardware multicast, apply it if needed (for example, launch `ospfd` on `kni` device).

> assuming we are in DPVS root dir and dpdk-stable-16.07.2 is under it, pls note it's not mandatory, just for convenience.
```
% cd <path-of-dpvs>
% cp patch/dpdk-16.07/0001-kni-use-netlink-event-for-multicast-driver-part.patch dpdk-stable-16.07.2/
% cd dpdk-stable-16.07.2/
% patch -n 1 < 0001-kni-use-netlink-event-for-multicast-driver-part.patch
```

Now build DPDK and export `RTE_SDK` env variable for DPDK app (DPVS).

```bash
% cd dpdk-stable-16.07.2/
% make config T=x86_64-native-linuxapp-gcc
Configuration done
% make # or make -j40 to save time, where 40 is the cpu core number.
% export RTE_SDK=$PWD
```

In our tutorial, `RTE_TARGET` is not set, the value is "build" by default, thus DPDK libs and header files can be found in `dpdk-stable-16.07.2/build`. It could be other values.

Now to set up DPDK hugepage,

```bash
% echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
% echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
% mkdir /mnt/huge
% mount -t hugetlbfs nodev /mnt/huge
```

Install Kernel modules and bind NIC with `igb_uio` driver. Quick start uses only one NIC, normally we use 2 for Full-NAT cluster, even 4 for bonding mode. Assuming `eth0` will be used for DPVS/DPDK, and another stand alone Linux NIC for debug, for example, `eth1`.

```bash
% modprob uio
% cd dpdk-stable-16.07.2/

% insmod build/kmod/igb_uio.ko
% insmod build/kmod/rte_kni.ko

% ./tools/dpdk-devbind.py --st
% ifconfig eth0 down # assuming eth0 is 0000:06:00.0
% ./tools/dpdk-devbind.py -b igb_uio 0000:06:00.0
```

`dpdk-devbind.py -u` can be used to unbind driver and switch it back to Linux driver like `ixgbe`. You can also use `lspci` or `ethtool -i eth0` to check the NIC PCI bus-id. Pls see [DPDK site](www.dpdk.org) for details.

## Build DPVS

It's simple, just set `RTE_SDK` and build it.

```bash
% cd dpdk-stable-16.07.2/
% export RTE_SDK=$PWD
% cd <path-of-dpvs>

% make # or "make -j40" to speed up.
% make install
```

> may need install dependencies, like `openssl` and `popt`.
Output files are installed to `dpvs/bin`.

```bash
% ls bin/
dpip dpvs ipvsadm keepalived
```

* `dpvs` is the main program.
* `dpip` is the tool to set IP address, route, vlan, neigh etc.
* `ipvsadm` and `keepalived` comes from LVS, both are modified.

## Launch DPVS

Now, `dpvs.conf` must be put at `/etc/dpvs.conf`, just copy it from `conf/dpvs.conf.single-nic.sample`.

```bash
% cp conf/dpvs.conf.single-nic.sample /etc/dpvs.conf
```

and start DPVS,

```bash
% cd <path-of-dpvs>/bin
% ./dpvs &
```

Check if it's get started ?

```bash
% ./dpip link show
1: dpdk0: socket 0 mtu 1500 rx-queue 8 tx-queue 8
UP 10000 Mbps full-duplex fixed-nego promisc-off
addr A0:36:9F:9D:61:F4 OF_RX_IP_CSUM OF_TX_IP_CSUM OF_TX_TCP_CSUM OF_TX_UDP_CSUM
```

If you see this message. Well done, `DPVS` is working with NIC `dpdk0`!

> Don't worry if you see this error,
```
EAL: Error - exiting with code: 1
Cause: ports in DPDK RTE (2) != ports in dpvs.conf(1)
```
it means the NIC used by DPVS is not match `/etc/dpvs.conf`. Pls use `dpdk-devbind` to adjust the NIC number or modify `dpvs.conf`. We'll improve this part to make DPVS more "clever" to avoid modify config file when NIC count is not match.


## Test Full-NAT Load Balance

The test topology looks like,

![fnat-single-nic](./pic/fnat-single-nic.png)

Set VIP and Local IP (LIP, needed by Full-NAT mode) on DPVS. Let's put commands into `setup.sh`. You do some check by `./ipvsadm -ln`, `./dpip addr show`.

```bash
% cat setup.sh
VIP=192.168.100.100
LIP=192.168.100.200
RS=192.168.100.2

./dpip addr add ${VIP}/24 dev dpdk0
./ipvsadm -A -t ${VIP}:80 -s rr
./ipvsadm -a -t ${VIP}:80 -r ${RS} -b

./ipvsadm --add-laddr -z ${LIP} -t 192.168.100.100:80 -F dpdk0
%

% ./setup.sh
```

Access VIP from Client, it looks good!

```bash
client % curl 192.168.100.100
Your ip:port : 192.168.100.3:56890
```

# Performance Test

Our test shows the forwarding speed (pps) of DPVS is several times than LVS and as good as Google's [Maglev](https://research.google.com/pubs/pub44824.html).

![performance](./pic/performance.png)

# License

Pls see the [License](./LICENSE.md) file.

# Contact Us

`DPVS` is developing by [iQiYi](www.iqiyi.com) [QLB](mailto:[email protected]) team since Arpil 2016 and now open-sourced. It's already widely used in iQiYi IDC for L4 load balance and SNAT clusters, and we plan to replace all our LVS clusters with DPVS. We are very happy that **more people can get involved** in this project. Welcome to try, report issues and submit pull requests. And pls feel free to contact us through **Github** or [Email](mailto:[email protected]).
Loading

0 comments on commit ee2099a

Please sign in to comment.