add docs: set up es-cluster on k8s

huanuo-cis · Oct 12, 2018 · 681cf49 · 681cf49
1 parent e6edece
commit 681cf49
Show file tree

Hide file tree

Showing 19 changed files with 1,462 additions and 24 deletions.
diff --git a/README.md b/README.md
@@ -1,28 +1,26 @@
 # kubeasz
 
-`kubeasz`致力于提供快速部署高可用`k8s`集群的工具，并且也努力成为`k8s`实践、使用的参考书；基于二进制方式部署和利用`ansible-playbook`实现自动化：既提供一键安装脚本，也可以分步执行安装各个组件，同时讲解每一步主要参数配置和注意事项；二进制方式部署有助于理解系统各组件的交互原理和熟悉组件启动参数，有助于快速排查解决实际问题。
+`kubeasz`致力于提供快速部署高可用`k8s`集群的工具, 并且也努力成为`k8s`实践、使用的参考书；基于二进制方式部署和利用`ansible-playbook`实现自动化：即提供一键安装脚本, 也可以分步执行安装各个组件, 同时讲解每一步主要参数配置和注意事项。
 
-**集群特性：`TLS` 双向认证、`RBAC` 授权、多`Master`高可用、支持`Network Policy`**
+**集群特性：`TLS`双向认证、`RBAC`授权、多`Master`高可用、支持`Network Policy`、备份恢复**
 
-项目基于`Ubuntu 16.04/CentOS 7`，需要了解基础`kubernetes` `docker` `linux` `ansible`等知识。
-
-请阅读[项目TodoList](docs/mixes/TodoList.md)和[项目分支说明](docs/mixes/branch.md)，欢迎提[Issues](https://github.com/gjmzj/kubeasz/issues)和[PRs](docs/mixes/HowToContribute.md)参与维护项目。
-
-|组件|更新|支持|
-|:-|:-|:-|
-|OS|&nbsp;|Ubuntu 16.04+, CentOS 7|
-|k8s|v1.11.3|v1.8, v1.9, v1.10, v1.11|
-|etcd|v3.3.8|v3.1, v3.2, v3.3|
-|docker|17.03.1-ce|17.*.*-ce, 18.*.*-ce|
-|network|&nbsp;|calico, cilium, flannel, kube-router|
+|组件|支持|
+|:-|:-|
+|OS|Ubuntu 16.04+, CentOS 7|
+|k8s|v1.8, v1.9, v1.10, v1.11, v1.12|
+|etcd|v3.1, v3.2, v3.3|
+|docker|17.03.2-ce, 18.06.1-ce|
+|network|calico, cilium, flannel, kube-router|
 
 -注：集群用到的所有二进制文件已打包好供下载 [https://pan.baidu.com/s/1c4RFaA](https://pan.baidu.com/s/1c4RFaA)  
 
+请阅读[项目TodoList](docs/mixes/TodoList.md)和[项目分支说明](docs/mixes/branch.md), 欢迎提[Issues](https://github.com/gjmzj/kubeasz/issues)和[PRs](docs/mixes/HowToContribute.md)参与维护项目。
+
 ## 快速指南
 
 单机快速体验k8s集群的测试、开发环境--[AllinOne部署](docs/setup/quickStart.md)
 
-## 安装步骤
+## 安装指南
 
 <table border="0">
     <tr>
@@ -44,14 +42,14 @@
 <table border="0">
     <tr>
         <td><strong>常用插件</strong></td>
-        <td><a href="docs/guide/kubedns.md">kube-dns</a></td>
+        <td><a href="docs/guide/kubedns.md">DNS</a></td>
         <td><a href="docs/guide/dashboard.md">dashboard</a></td>
         <td><a href="docs/guide/metrics-server.md">metrics-server</a></td>
         <td><a href="docs/guide/prometheus.md">prometheus</a></td>
         <td><a href="docs/guide/index.md">更多...</a></td>
     </tr>
     <tr>
-        <td><strong>集群运维</strong></td>
+        <td><strong>集群管理</strong></td>
         <td><a href="docs/op/AddNode.md">增加node</a></td>
         <td><a href="docs/op/AddMaster.md">增加master</a></td>
         <td><a href="docs/op/upgrade.md">升级集群</a></td>
@@ -67,24 +65,32 @@
         <td><a href=""></a></td>
     </tr>
     <tr>
-        <td><strong>其他</strong></td>
-        <td><a href="docs/guide/harbor.md">harbor部署</a></td>
-        <td><a href="docs/guide/ingress.md">ingress入口</a></td>
-        <td><a href="docs/guide/helm.md">helm工具</a></td>
-        <td><a href="docs/guide/jenkins.md">jenkins集成</a></td>
+        <td><strong>周边生态</strong></td>
+        <td><a href="docs/guide/harbor.md">harbor</a></td>
+        <td><a href="docs/guide/helm.md">helm</a></td>
+        <td><a href="docs/guide/jenkins.md">jenkins</a></td>
+        <td><a href=""></a></td>
+        <td><a href=""></a></td>
+    </tr>
+    <tr>
+        <td><strong>应用实践</strong></td>
+        <td><a href="docs/practice/java_war_app.md">java应用部署</a></td>
+        <td><a href="docs/practice/es_cluster.md">elasticsearch部署</a></td>
+        <td><a href=""></a></td>
+        <td><a href=""></a></td>
         <td><a href=""></a></td>
     </tr>
 </table>
 
 ## 沟通交流
 
-- 微信群：k8s&kubeasz实践，搜索微信号`badtobone`，请备注（城市-github用户名），验证通过会加入群聊。
+- 微信群：k8s&kubeasz实践, 搜索微信号`badtobone`, 请备注（城市-github用户名）, 验证通过会加入群聊。
 - 推荐阅读：[rootsongjc-Kubernetes指南](https://github.com/rootsongjc/kubernetes-handbook) [feisky-Kubernetes指南](https://github.com/feiskyer/kubernetes-handbook/blob/master/zh/SUMMARY.md) [opsnull-安装教程](https://github.com/opsnull/follow-me-install-kubernetes-cluster)
 
 ## 贡献&致谢
 
 感谢所有为项目提交 `Issues`和`PRs` 的贡献者！
 
-- [如何贡献](docs/mixes/HowToContribute.md)
+- [如何 PR](docs/mixes/HowToContribute.md)
 
-Copyright 2017 gjmzj ([email protected]) Apache License 2.0，详情见 [LICENSE](docs/mixes/LICENSE) 文件。
+Copyright 2017 gjmzj ([email protected]) Apache License 2.0, 详情见 [LICENSE](docs/mixes/LICENSE) 文件。
diff --git a/docs/practice/es_cluster.md b/docs/practice/es_cluster.md
@@ -0,0 +1,168 @@
+# Elasticsearch 部署实践
+
+`Elasticsearch`是目前全文搜索引擎的首选，它可以快速地储存、搜索和分析海量数据；也可以看成是真正分布式的高效数据库集群；`Elastic`的底层是开源库`Lucene`；封装并提供了`REST API`的操作接口。
+
+## 单节点 docker 测试安装  
+``` bash
+cat > es-start.sh << EOF
+#!/bin/bash
+
+sysctl -w vm.max_map_count=262144
+
+docker run --detach \
+   --name es01 \
+   -p 9200:9200 -p 9300:9300 \
+   -e "discovery.type=single-node" \
+   -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 \
+   --ulimit nofile=65536:65536 \
+   --volume /srv/elasticsearch/data:/usr/share/elasticsearch/data \
+   --volume /srv/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
+   jmgao1983/elasticsearch:6.4.0
+EOF
+```
+
+执行`sh es-start.sh`后，就在本地运行了。
+
+- 验证 docker 镜像运行情况  
+``` bash
+root@docker-ts:~# docker ps -a
+CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                            NAMES
+171f3fecb596        jmgao1983/elasticsearch:6.4.0   "/usr/local/bin/do..."   2 hours ago         Up 2 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es01
+```
+- 验证 es 健康检查  
+``` bash
+root@docker-ts:~# curl http://127.0.0.1:9200/_cat/health
+epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
+1535523956 06:25:56  docker-es     green           1         1      0   0    0    0        0             0                  -                100.0%
+```
+## 在 k8s 上部署 Elasticsearch 集群
+
+在生产环境下，Elasticsearch 集群由不同的角色节点组成：
+
+- master 节点：参与主节点选举，不存储数据；建议3个以上，维护整个集群的稳定可靠状态
+- data 节点：不参与选主，负责存储数据；主要消耗磁盘，内存
+- client 节点：不参与选主，不存储数据；负责处理用户请求，实现请求转发，负载均衡等功能
+
+这里使用`helm chart`来部署 (https://github.com/helm/charts/tree/master/incubator/elasticsearch)
+
+- 1.安装 helm: 以本项目[安全安装helm](../guide/helm.md)为例
+- 2.准备 PV: 以本项目[K8S 集群存储](../setup/08-cluster-storage.md)创建`nfs`动态 PV 为例
+  - 编辑配置文件：roles/cluster-storage/defaults/main.yml  
+``` bash
+storage:
+  nfs:
+    enabled: "yes"
+    server: "192.168.1.8"
+    server_path: "/share"
+    storage_class: "nfs-es"
+    provisioner_name: "nfs-provisioner-01"
+```
+  - 创建 nfs provisioner  
+``` bash
+$ ansible-playbook /etc/ansible/roles/cluster-storage/cluster-storage.yml
+# 执行成功后验证
+$ kubectl get pod --all-namespaces |grep nfs-prov
+kube-system   nfs-provisioner-01-6b7fbbf9d4-bh8lh        1/1       Running   0          1d
+```
+- 3.安装 elasticsearch chart  
+``` bash
+$ cd /etc/ansible/manifests/es-cluster
+# 如果你的helm安装没有启用tls证书，请使用helm命令替换以下的helms命令
+$ helms install --name es-cluster --namespace elastic -f es-values.yaml elasticsearch
+```
+- 4.验证 es 集群  
+``` bash
+# 验证k8s上 es集群状态
+$ kubectl get pod,svc -n elastic 
+NAME                                                   READY   STATUS    RESTARTS   AGE
+pod/es-cluster-elasticsearch-client-778df74c8f-7fj4k   1/1     Running   0          2m17s
+pod/es-cluster-elasticsearch-client-778df74c8f-skh8l   1/1     Running   0          2m3s
+pod/es-cluster-elasticsearch-data-0                    1/1     Running   0          25m
+pod/es-cluster-elasticsearch-data-1                    1/1     Running   0          11m
+pod/es-cluster-elasticsearch-master-0                  1/1     Running   0          25m
+pod/es-cluster-elasticsearch-master-1                  1/1     Running   0          12m
+pod/es-cluster-elasticsearch-master-2                  1/1     Running   0          10m
+
+NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+service/es-cluster-elasticsearch-client      NodePort    10.68.157.105   <none>        9200:29200/TCP,9300:29300/TCP   25m
+service/es-cluster-elasticsearch-discovery   ClusterIP   None            <none>        9300/TCP                        25m
+
+# 验证 es集群本身状态
+$ curl $NODE_IP:29200/_cat/health
+1539335131 09:05:31 es-on-k8s green 7 2 0 0 0 0 0 0 - 100.0%
+
+$ curl $NODE_IP:29200/_cat/indices?v
+health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
+root@k8s401:/etc/ansible# curl 10.100.97.41:29200/_cat/nodes?
+172.31.2.4 27 80 5 0.09 0.11 0.21 mi - es-cluster-elasticsearch-master-0
+172.31.1.7 30 97 3 0.39 0.29 0.27 i  - es-cluster-elasticsearch-client-778df74c8f-skh8l
+172.31.3.7 20 97 3 0.11 0.17 0.18 i  - es-cluster-elasticsearch-client-778df74c8f-7fj4k
+172.31.1.5  8 97 5 0.39 0.29 0.27 di - es-cluster-elasticsearch-data-0
+172.31.2.5  8 80 3 0.09 0.11 0.21 di - es-cluster-elasticsearch-data-1
+172.31.1.6 18 97 4 0.39 0.29 0.27 mi - es-cluster-elasticsearch-master-2
+172.31.3.6 20 97 4 0.11 0.17 0.18 mi * es-cluster-elasticsearch-master-1
+```
+### es 性能压测
+
+如上已使用 chart 在 k8s上部署了 **7** 节点的 elasticsearch 集群；各位应该十分好奇性能怎么样；官方提供了压测工具[esrally](https://github.com/elastic/rally)可以方便的进行性能压测，这里省略安装和测试过程；压测机上执行：  
+`esrally --track=http_logs --target-hosts="$NODE_IP:29200" --pipeline=benchmark-only --report-file=report.md`  
+压测过程需要1-2个小时，部分压测结果如下：  
+``` bash
+------------------------------------------------------
+    _______             __   _____
+   / ____(_)___  ____ _/ /  / ___/_________  ________
+  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
+ / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
+/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
+------------------------------------------------------
+
+|   Lap |                               Metric |         Task |       Value |    Unit |
+|------:|-------------------------------------:|-------------:|------------:|--------:|
+...
+|   All |                       Min Throughput | index-append |     16903.2 |  docs/s |
+|   All |                    Median Throughput | index-append |     17624.4 |  docs/s |
+|   All |                       Max Throughput | index-append |     19382.8 |  docs/s |
+|   All |              50th percentile latency | index-append |     1865.74 |      ms |
+|   All |              90th percentile latency | index-append |     3708.04 |      ms |
+|   All |              99th percentile latency | index-append |     6379.49 |      ms |
+|   All |            99.9th percentile latency | index-append |     8389.74 |      ms |
+|   All |           99.99th percentile latency | index-append |     9612.84 |      ms |
+|   All |             100th percentile latency | index-append |     9861.02 |      ms |
+|   All |         50th percentile service time | index-append |     1865.74 |      ms |
+|   All |         90th percentile service time | index-append |     3708.04 |      ms |
+|   All |         99th percentile service time | index-append |     6379.49 |      ms |
+|   All |       99.9th percentile service time | index-append |     8389.74 |      ms |
+|   All |      99.99th percentile service time | index-append |     9612.84 |      ms |
+|   All |        100th percentile service time | index-append |     9861.02 |      ms |
+|   All |                           error rate | index-append |           0 |       % |
+|   All |                       Min Throughput |      default |        0.66 |   ops/s |
+|   All |                    Median Throughput |      default |        0.66 |   ops/s |
+|   All |                       Max Throughput |      default |        0.66 |   ops/s |
+|   All |              50th percentile latency |      default |      770131 |      ms |
+|   All |              90th percentile latency |      default |      825511 |      ms |
+|   All |              99th percentile latency |      default |      838030 |      ms |
+|   All |             100th percentile latency |      default |      839382 |      ms |
+|   All |         50th percentile service time |      default |      1539.4 |      ms |
+|   All |         90th percentile service time |      default |     1635.39 |      ms |
+|   All |         99th percentile service time |      default |     1728.02 |      ms |
+|   All |        100th percentile service time |      default |      1736.2 |      ms |
+|   All |                           error rate |      default |           0 |       % |
+...
+```  
+从测试结果看：集群的吞吐可以（k8s es-client pod还可以扩展）；延迟略高一些（因为使用了nfs共享存储）；整体效果不错。
+
+### 中文分词安装
+
+安装 ik 插件即可，可以自定义已安装ik插件的es docker镜像：创建如下 Dockerfile  
+``` bash
+FROM jmgao1983/elasticsearch:6.4.0
+
+RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install \
+  --batch https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.4.0/elasticsearch-analysis-ik-6.4.0.zip \
+  && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+```
+
+### 参考阅读
+
+1. [Elasticsearch 入门教程](http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html)
+2. [Elasticsearch 压测方案之 esrally 简介](https://segmentfault.com/a/1190000011174694)
diff --git a/manifests/es-cluster/elasticsearch/.helmignore b/manifests/es-cluster/elasticsearch/.helmignore
@@ -0,0 +1,3 @@
+.git
+# OWNERS file for Kubernetes
+OWNERS
diff --git a/manifests/es-cluster/elasticsearch/Chart.yaml b/manifests/es-cluster/elasticsearch/Chart.yaml
@@ -0,0 +1,21 @@
+name: elasticsearch
+home: https://www.elastic.co/products/elasticsearch
+version: 1.7.2
+appVersion: 6.4.0
+description: Flexible and powerful open source, distributed real-time search and analytics
+  engine.
+icon: https://static-www.elastic.co/assets/blteb1c97719574938d/logo-elastic-elasticsearch-lt.svg
+sources:
+- https://www.elastic.co/products/elasticsearch
+- https://github.com/jetstack/elasticsearch-pet
+- https://github.com/giantswarm/kubernetes-elastic-stack
+- https://github.com/GoogleCloudPlatform/elasticsearch-docker
+- https://github.com/clockworksoul/helm-elasticsearch
+- https://github.com/pires/kubernetes-elasticsearch-cluster
+maintainers:
+- name: simonswine
+  email: [email protected]
+- name: icereval
+  email: [email protected]
+- name: rendhalver
+  email: [email protected]
diff --git a/manifests/es-cluster/elasticsearch/OWNERS b/manifests/es-cluster/elasticsearch/OWNERS
@@ -0,0 +1,8 @@
+approvers:
+- simonswine
+- icereval
+- rendhalver
+reviewers:
+- simonswine
+- icereval
+- rendhalver