Todis [ 中文 ]

0. Open Source Community Version

Now(2021-11-14) users can build todis by himself, such build is community version which is full fledged but lacks performance sideplugins(topling SST & MemTab & Distributed Compaction), even lacks such components, todis works well and performance outperforms competitors.

Open Source Community Version is disallowed to be used by bytedance, see LICENSE.

1. Introduction

Todis is a massive, persistent Redis server developed by Topling Inc.

Currently(2021-10-11), there are many massive, persistent Redis server based on RocksDB, and many big company developed their private or open sourced alternatives(massive, persistent Redis).

These alternative Redis servers are low performance and high cost, and lack of scalability, operationality, monitoring, ....

Todis are aimed on all such pain points, and accomplished these goals, users can experiencing Managed Todis based on aliyun in 10 minutes.

2. Features

1. Computing and Storage are seperated for elastic scaling
2. Auto scaling, no need for sharding/partitioning
3. Plenty monitoring metrics
4. Web view for DB internal stats

3. Performance

1. With elastic distributed compaction, kicked off write stalls, achieved sustained write speed over 100MB/s on a 2C16G cloud server
2. With searchable in memory compression technology(topling-zip), achieved very high read performance(saturate network bandwitdh)
3. With shared storage and engine level leader-follwer mechanics
   • Sync latency is lower to milliseconds
   • New follower(read only node) can be started and available in 10 seconds

4. Architeture

4.1. Redis layer of Todis

Redis layer is forked from pika with many improvements:

1. Strorage engine is replaced from RocksDB to ToplingDB
2. Rewrite performance critical code of pika
3. Add many monitoring metrics(server side latency histogram for all commands and data len histogram...)
4. Adapt for ToplingDB SidePlugin, and Web view for config and server stats
5. Adapt for ToplingDB distributed compaction

4.2. Storage Engine layer (ToplingDB)

ToplingDB is a storage engine developed by Topling Inc., which forked from RocksDB with many improvments:

1. SidePlugin, Engine WebView, Monitoring Metrics...
2. Distributed Compaction which offload compaction to a dedicated cluster
   • In Managed Todis, this cluster is shared by all users and all Todis instances
3. Topling SST and MemTab which based on topling-zip(forked from terark-zip)
   • Rockeet is the founder of Terark Inc. and the author of terark-zip
   • Terark Inc. which was acquired by bytedance at 2019
   • Bytedance open sourced terark-zip at the end of 2020
   • As the author of terark-zip, rockeet added many enhancements to topling-zip(his fork of terark-zip) since then

5. Compile

5.1 CentOS

sudo yum install epel-release -y
sudo yum config-manager --set-enabled powertools
sudo yum install git gcc-c++ glog-devel protobuf-devel libaio-devel -y
git clone https://github.com/topling/todis
cd todis
git submodule update --init --recursive
make -j`nproc`

If your protobuf-devel is too old, compilation maybe fail, you should update protobuf-devel, such as compile new protobuf from github source, or from vcpkg...

6. Deploy

Compiling generated dir output which included all files required at runtime, users need change config files(especially path items), there are two config files: todis-community.conf and todis-community.json, the former is compatible for pika, the later is ToplingDB's SidePlugin conf for todis, sideplugin-conf of todis-community.conf is the path of todis-community.json.

The most simple way is to change all conf items prefixed with /path/to/.

---

---

---

Introduction中文

Pika is a persistent huge storage service , compatible with the vast majority of redis interfaces (details), including string, hash, list, zset, set and management interfaces. With the huge amount of data stored, redis may suffer for a capacity bottleneck, and pika was born for solving it. Except huge storage capacity, pika also support master-slave mode by slaveof command, including full and partial synchronization. You can also use pika together with twemproxy or codis(pika has supported data migration in codis，thanks left2right and fancy-rabbit) for distributed Redis solution

UserList

More

Feature

• huge storage capacity
• compatible with redis interface, you can migrate to pika easily
• support master-slave mode (slaveof)
• various management interfaces

For developer

Releases

The User can download the binary release from releases or compile the source release.

Dependencies

• snappy - a library for fast data compression
• glog - google log library

Upgrade your gcc to version at least 4.8 to get C++11 support.

Supported platforms

• linux - CentOS 5&6

• linux - Ubuntu

If it comes to some missing libs, install them according to the prompts and retry it.

Compile

Upgrade your gcc to version at least 4.8 to get C++11 support.

Get the source code

git clone https://github.com/Qihoo360/pika.git

Then compile pika, all submodules will be updated automatically.

make

Usage

./output/bin/pika -c ./conf/pika.conf

Performance

More details on Performance.

Documents

1. Wiki

Contact Us

Users Mail Group: join in [email protected]

Developers Mail Group: join in [email protected]

QQ group: 294254078

For more information about Pika, Atlas and some other technology please pay attention to our Hulk platform official account