diff --git a/README.md b/README.md index 6bab313a..a16c9c1c 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,9 @@ please refer to [Redis Cluster Protocol](./docs/redis_cluster_protocol.md). ##### Metadata Storage Metadata storage stores all the metadata of the whole `undermoon` cluster, including existing Redis instances, proxies, and exposed Redis clusters. -Now it's an in-memory storage server called `Memory Broker` -and there's [another implementation backed by Etcd](https://github.com/doyoubi/overmoon) in the future. +Now it's an in-memory storage server called `Memory Broker`. +When using [undermoon-operator](https://github.com/doyoubi/undermoon-operator), +this `Memory Broker` will change to use `ConfigMap` to store the data. ##### Coordinator Coordinator will synchronize the metadata between broker and server proxy. @@ -46,171 +47,27 @@ The design of chunk makes it very easy to build a cluster with a good topology f Using [undermoon-operator](https://github.com/doyoubi/undermoon-operator) is the easiest way to create Redis clusters if you have Kubernetes. -See the `README.md` of [undermoon-operator](https://github.com/doyoubi/undermoon-operator) -for more how to use it. - -### Run Undermoon using Docker Compose -The following examples use docker to create an `undermoon` cluster. - -Or you can set them up without docker following this docs: [setting up undermoon manually](docs/set_up_manually.md). - -Requirements: - -- docker-compose -- redis-cli -- [jq](https://stedolan.github.io/jq/) - -#### Run the cluster in docker-compose -Download and run the cluster directly: -```bash -$ make docker-mem-broker-example -``` - -Or build it yourself and run the `undermoon` docker image: -```bash -$ make docker-build-image -$ make docker-mem-broker -``` - -#### Register Proxies -After everything is up, run the initialize script to register the storage resources through HTTP API: -```bash -$ ./examples/mem-broker/init.sh -``` - -We have 6 available proxies. -```bash -$ curl http://localhost:7799/api/v2/proxies/addresses -``` - -#### Create Cluster -Since every proxy has 2 corresponding Redis nodes, we have 12 nodes in total. -Note that the number of a cluster could only be the multiples of 4. -Let's create a cluster with 4 nodes. -```bash -$ curl -XPOST -H 'Content-Type: application/json' \ - http://localhost:7799/api/v2/clusters/meta/mycluster -d '{"node_number": 4}' -``` - -Before connecting to the cluster, you need to add these hosts to you `/etc/hosts`: ``` -# /etc/hosts -127.0.0.1 server_proxy1 -127.0.0.1 server_proxy2 -127.0.0.1 server_proxy3 -127.0.0.1 server_proxy4 -127.0.0.1 server_proxy5 -127.0.0.1 server_proxy6 -``` - -Let's checkout our cluster. It's created by some randomly chosen proxies. -We need to find them out first. -Note that you need to install the `jq` command to parse json easily for the command below. +helm install my-undermoon-operator undermoon-operator-.tgz -```bash -# List the proxies of the our "mycluster`: -$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq -"server_proxy5:6005" -"server_proxy6:6006" +helm install \ + --set 'cluster.clusterName=my-cluster-name' \ + --set 'cluster.chunkNumber=2' \ + --set 'cluster.maxMemory=2048' \ + --set 'cluster.port=5299' \ + my-cluster \ + -n my-namespace \ + undermoon-cluster-.tgz ``` -Pickup one of the proxy address above (in my case it's `server_proxy5:6005`) for the cluster `mycluster` and connect to it. - -```bash -# Add `-c` to enable cluster mode: -$ redis-cli -h server_proxy5 -p 6005 -c -# List the proxies: -server_proxy5:6005> cluster nodes -mycluster___________d71bc00fbdddf89_____ server_proxy5:6005 myself,master - 0 0 7 connected 0-8191 -mycluster___________8de73f9146386295____ server_proxy6:6006 master - 0 0 7 connected 8192-16383 -# Send out some requests: -server_proxy5:6005> get a --> Redirected to slot [15495] located at server_proxy6:6006 -(nil) -server_proxy6:6006> get b --> Redirected to slot [3300] located at server_proxy5:6005 -(nil) -``` -Great! We can use our created cluster just like the official Redis Cluster. - -#### Scale Up -It actually has 4 Redis nodes under the hood. -```bash -# List the nodes of the our "mycluster`: -$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].address' -"redis9:6379" -"redis10:6379" -"redis11:6379" -"redis12:6379" -``` -Two of them are masters and the other two of them are replicas. - -Let's scale up to 8 nodes: -```bash -# Add 4 nodes -$ curl -XPATCH -H 'Content-Type: application/json' \ - http://localhost:7799/api/v2/clusters/nodes/mycluster -d '{"node_number": 4}' -# Start migrating the data -$ curl -XPOST http://localhost:7799/api/v2/clusters/migrations/expand/mycluster -``` - -Now we have 4 server proxies: -```bash -$ redis-cli -h server_proxy5 -p 6005 -c -server_proxy5:6005> cluster nodes -mycluster___________d71bc00fbdddf89_____ server_proxy5:6005 myself,master - 0 0 12 connected 0-4095 -mycluster___________8de73f9146386295____ server_proxy6:6006 master - 0 0 12 connected 8192-12287 -mycluster___________be40fe317baf2cf7____ server_proxy2:6002 master - 0 0 12 connected 4096-8191 -mycluster___________9434df4158f3c5a4____ server_proxy4:6004 master - 0 0 12 connected 12288-16383 -``` - -and 8 nodes: -```bash -# List the nodes of the our "mycluster`: -$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].address' -"redis9:6379" -"redis10:6379" -"redis11:6379" -"redis12:6379" -"redis3:6379" -"redis4:6379" -"redis7:6379" -"redis8:6379" -``` - -#### Failover -If you shutdown any proxy, the replica will be promoted to master. -And as long as the whole `undermoon` cluster has remaining free proxies, -it can automatically replace the failed proxy, - -```bash -# List the proxies of the our "mycluster`: -$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq -"server_proxy5:6005" -"server_proxy6:6006" -"server_proxy2:6002" -"server_proxy4:6004" -``` - -Let's shutdown one of the proxies like `server_proxy5:6005` here. -```bash -$ docker ps | grep server_proxy5 | awk '{print $1}' | xargs docker kill -``` +See the `README.md` of [undermoon-operator](https://github.com/doyoubi/undermoon-operator) +for how to use it. -The `undermoon` will detect the failure, replace the failed proxy, promote the new master, and add new replica to the new master. -```bash -# server_proxy5 is replaced by server_proxy3 -$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq -"server_proxy3:6003" -"server_proxy6:6006" -"server_proxy2:6002" -"server_proxy4:6004" -``` +### Run Undermoon Using Docker Compose +See [docker compose example](./docs/docker_compose_example.md). -And we can remove the server_proxy3 from the `undermoon` cluster now. -```bash -$ curl -XDELETE http://localhost:7799/api/v2/proxies/meta/server_proxy3:6003 -``` +### Setup Undermoon Manually +Or you can set them up without docker following this docs: [setting up undermoon manually](docs/set_up_manually.md). ## Development `undermoon` tries to avoid `unsafe` and some calls that could crash. diff --git a/conf/server-proxy.toml b/conf/server-proxy.toml index f9939fd3..c7bbd367 100644 --- a/conf/server-proxy.toml +++ b/conf/server-proxy.toml @@ -29,6 +29,7 @@ backend_high_flush_interval = 600000 # When active_redirection is enabled, # all the server proxies will handle the redirection inside. # Clients don't need to be a Redis Cluster Client. +# NOTICE: This is an experimental feature and don't use it in production. active_redirection = false # This is only useful when active_redirection is true. # Use 0 to disable limitation. diff --git a/docs/docker_compose_example.md b/docs/docker_compose_example.md new file mode 100644 index 00000000..c88c73a2 --- /dev/null +++ b/docs/docker_compose_example.md @@ -0,0 +1,160 @@ +# Run Undermoon using Docker Compose +The following examples use docker to create an `undermoon` cluster. + +Requirements: + +- docker-compose +- redis-cli +- [jq](https://stedolan.github.io/jq/) + +#### Run the cluster in docker-compose +Download and run the cluster directly: +```bash +$ make docker-mem-broker-example +``` + +Or build it yourself and run the `undermoon` docker image: +```bash +$ make docker-build-image +$ make docker-mem-broker +``` + +#### Register Proxies +After everything is up, run the initialize script to register the storage resources through HTTP API: +```bash +$ ./examples/mem-broker/init.sh +``` + +We have 6 available proxies. +```bash +$ curl http://localhost:7799/api/v2/proxies/addresses +``` + +#### Create Cluster +Since every proxy has 2 corresponding Redis nodes, we have 12 nodes in total. +Note that the number of a cluster could only be the multiples of 4. +Let's create a cluster with 4 nodes. +```bash +$ curl -XPOST -H 'Content-Type: application/json' \ + http://localhost:7799/api/v2/clusters/meta/mycluster -d '{"node_number": 4}' +``` + +Before connecting to the cluster, you need to add these hosts to you `/etc/hosts`: +``` +# /etc/hosts +127.0.0.1 server_proxy1 +127.0.0.1 server_proxy2 +127.0.0.1 server_proxy3 +127.0.0.1 server_proxy4 +127.0.0.1 server_proxy5 +127.0.0.1 server_proxy6 +``` + +Let's checkout our cluster. It's created by some randomly chosen proxies. +We need to find them out first. +Note that you need to install the `jq` command to parse json easily for the command below. + +```bash +# List the proxies of the our "mycluster`: +$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq +"server_proxy5:6005" +"server_proxy6:6006" +``` + +Pickup one of the proxy address above (in my case it's `server_proxy5:6005`) for the cluster `mycluster` and connect to it. + +```bash +# Add `-c` to enable cluster mode: +$ redis-cli -h server_proxy5 -p 6005 -c +# List the proxies: +server_proxy5:6005> cluster nodes +mycluster___________d71bc00fbdddf89_____ server_proxy5:6005 myself,master - 0 0 7 connected 0-8191 +mycluster___________8de73f9146386295____ server_proxy6:6006 master - 0 0 7 connected 8192-16383 +# Send out some requests: +server_proxy5:6005> get a +-> Redirected to slot [15495] located at server_proxy6:6006 +(nil) +server_proxy6:6006> get b +-> Redirected to slot [3300] located at server_proxy5:6005 +(nil) +``` +Great! We can use our created cluster just like the official Redis Cluster. + +#### Scale Up +It actually has 4 Redis nodes under the hood. +```bash +# List the nodes of the our "mycluster`: +$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].address' +"redis9:6379" +"redis10:6379" +"redis11:6379" +"redis12:6379" +``` +Two of them are masters and the other two of them are replicas. + +Let's scale up to 8 nodes: +```bash +# Add 4 nodes +$ curl -XPATCH -H 'Content-Type: application/json' \ + http://localhost:7799/api/v2/clusters/nodes/mycluster -d '{"node_number": 4}' +# Start migrating the data +$ curl -XPOST http://localhost:7799/api/v2/clusters/migrations/expand/mycluster +``` + +Now we have 4 server proxies: +```bash +$ redis-cli -h server_proxy5 -p 6005 -c +server_proxy5:6005> cluster nodes +mycluster___________d71bc00fbdddf89_____ server_proxy5:6005 myself,master - 0 0 12 connected 0-4095 +mycluster___________8de73f9146386295____ server_proxy6:6006 master - 0 0 12 connected 8192-12287 +mycluster___________be40fe317baf2cf7____ server_proxy2:6002 master - 0 0 12 connected 4096-8191 +mycluster___________9434df4158f3c5a4____ server_proxy4:6004 master - 0 0 12 connected 12288-16383 +``` + +and 8 nodes: +```bash +# List the nodes of the our "mycluster`: +$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].address' +"redis9:6379" +"redis10:6379" +"redis11:6379" +"redis12:6379" +"redis3:6379" +"redis4:6379" +"redis7:6379" +"redis8:6379" +``` + +#### Failover +If you shutdown any proxy, the replica will be promoted to master. +And as long as the whole `undermoon` cluster has remaining free proxies, +it can automatically replace the failed proxy, + +```bash +# List the proxies of the our "mycluster`: +$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq +"server_proxy5:6005" +"server_proxy6:6006" +"server_proxy2:6002" +"server_proxy4:6004" +``` + +Let's shutdown one of the proxies like `server_proxy5:6005` here. +```bash +$ docker ps | grep server_proxy5 | awk '{print $1}' | xargs docker kill +``` + +The `undermoon` will detect the failure, replace the failed proxy, promote the new master, and add new replica to the new master. +```bash +# server_proxy5 is replaced by server_proxy3 +$ curl -s http://localhost:7799/api/v2/clusters/meta/mycluster | jq '.cluster.nodes[].proxy_address' | uniq +"server_proxy3:6003" +"server_proxy6:6006" +"server_proxy2:6002" +"server_proxy4:6004" +``` + +And we can remove the server_proxy3 from the `undermoon` cluster now. +```bash +$ curl -XDELETE http://localhost:7799/api/v2/proxies/meta/server_proxy3:6003 +``` diff --git a/docs/mem_broker_replica.md b/docs/mem_broker_replica.md index 45ff9629..4e2fe642 100644 --- a/docs/mem_broker_replica.md +++ b/docs/mem_broker_replica.md @@ -1,8 +1,4 @@ # Setting Up Backup for Memory Broker -Even though the on going [broker backed by Etcd](https://github.com/doyoubi/overmoon) will be more reliable, -sometimes users may choose simplicity over reliability. -`Memory Broker` is still a good way to go at this point. -`Memory Broker` provides a naive solution for backing up the data. ## Setting Up Replica for Memory Broker Build the binaries: diff --git a/docs/memory_broker_api.md b/docs/memory_broker_api.md index bc5ad2b7..4ce28435 100644 --- a/docs/memory_broker_api.md +++ b/docs/memory_broker_api.md @@ -54,6 +54,7 @@ HTTP 200 ##### Error ``` HTTP 409 { "error": "INVALID_META_VERSION" } +HTTP 409 { "error": "RETRY" } ``` #### Get cluster info @@ -101,6 +102,7 @@ HTTP 400 { "error": "INVALID_CLUSTER_NAME" } HTTP 400 { "error": "INVALID_NODE_NUMBER" } HTTP 409 { "error": "ALREADY_EXISTED" } HTTP 409 { "error": "NO_AVAILABLE_RESOURCE" } +HTTP 409 { "error": "RETRY" } ``` #### Delete cluster @@ -115,6 +117,7 @@ HTTP 200 ``` HTTP 400 { "error": "INVALID_CLUSTER_NAME" } HTTP 404 { "error": "CLUSTER_NOT_FOUND" } +HTTP 409 { "error": "RETRY" } ``` #### Add nodes to cluster @@ -142,6 +145,7 @@ HTTP 409 { "error": "ALREADY_EXISTED" } HTTP 409 { "error": "NO_AVAILABLE_RESOURCE" } HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Add nodes to cluster if needed @@ -171,6 +175,7 @@ HTTP 409 { "error": "ALREADY_EXISTED" } HTTP 409 { "error": "NO_AVAILABLE_RESOURCE" } HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Delete Unused nodes in a cluster @@ -188,6 +193,7 @@ HTTP 404 { "error": "CLUSTER_NOT_FOUND" } HTTP 409 { "error": "FREE_NODE_NOT_FOUND" } HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Add or remove nodes and start migration @@ -212,6 +218,7 @@ HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 400 { "error": "INVALID_NODE_NUMBER" } HTTP 409 { "error": "NO_AVAILABLE_RESOURCE" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Start migration for scaling out @@ -231,6 +238,7 @@ HTTP 404 { "error": "CLUSTER_NOT_FOUND" } HTTP 409 { "error": "FREE_NODE_NOT_FOUND" } HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Start migration for scaling down @@ -252,6 +260,7 @@ HTTP 404 { "error": "CLUSTER_NOT_FOUND" } HTTP 409 { "error": "FREE_NODE_FOUND" } HTTP 409 { "error": "MIGRATION_RUNNING" } HTTP 409 { "error": "NODE_NUMBER_CHANGING" } +HTTP 409 { "error": "RETRY" } ``` #### Change cluster config @@ -279,6 +288,7 @@ HTTP 409 { "value": "xxxx", "message": "xxxx" } +HTTP 409 { "error": "RETRY" } ``` #### Add proxy @@ -302,6 +312,7 @@ HTTP 200 ``` HTTP 400 { "error": "INVALID_PROXY_ADDRESS" } HTTP 409 { "error": "ALREADY_EXISTED" } +HTTP 409 { "error": "RETRY" } ``` #### Delete proxy @@ -316,6 +327,7 @@ HTTP 200 ``` HTTP 404 { "error": "PROXY_NOT_FOUND" } HTTP 409 { "error": "IN_USE" } +HTTP 409 { "error": "RETRY" } ``` #### Balance Masters @@ -330,6 +342,7 @@ HTTP 200 ``` HTTP 400 { "error": "INVALID_CLUSTER_NAME" } HTTP 404 { "error": "CLUSTER_NOT_FOUND" } +HTTP 409 { "error": "RETRY" } ``` #### Get the current global epoch diff --git a/docs/meta_command.md b/docs/meta_command.md index 0ac2d3f1..8a319624 100644 --- a/docs/meta_command.md +++ b/docs/meta_command.md @@ -12,7 +12,9 @@ Sets the mapping relationship between the server-side proxy and its correspondin - `epoch` is the logical time of the configuration this command is sending used to decide which configuration is more up-to-date. Every running server-side proxy will store its epoch and will reject all the `UMCTL [SETCLUSTER|SETREPL]` requests which don't have higher epoch. -- `flags`: Currently it may be NOFLAG or FORCE. When it's `FORCE`, the server-side proxy will ignore the epoch rule above and will always accept the configuration +- `flags`: Currently it may be NOFLAG or combination of FORCE and COMPRESS("FORCE,COMPRESS"). +When it contains `FORCE`, the server-side proxy will ignore the epoch rule above and will always accept the configuration. +When it contains `COMPRESS`, later it only contains one element with gzip and base64 encoded data. - `slot_range` can be like - 1 0-1000 - 2 0-1000 2000-3000