-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
207 lines (113 loc) · 97.4 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>Elasticsearch-索引处理</title>
<link href="//blog/Elasticsearch-%E7%B4%A2%E5%BC%95%E5%A4%84%E7%90%86.html"/>
<content type="html"><![CDATA[<p>使用 ELK 进行日志分析处理索引的过程中是否遇到索引数据过多;导致数据分析缓慢,以至于分析不准确.以下介绍如何处理日常生成的索引</p><a id="more"></a><h1 id="Elasticsearch-indices"><a href="#Elasticsearch-indices" class="headerlink" title="Elasticsearch indices"></a>Elasticsearch indices</h1><h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><blockquote><p>通过 ELK 收集 Nginx/Tomcat日志进行分析产生大量的索引文件,随着时间的增加索引速度随之变慢/监控数据显示变慢 并且增加磁盘使用量</p></blockquote><p>##分析</p><p>通过以上问题分析需要解决以下几点问题:</p><ol><li>减少 Elasticsearch 处理的索引量(提高监控实时准确性)</li><li>减少 Elasticsearch 索引存储量(减少磁盘使用量)</li><li>关闭 Elasticsearch 无用索引数据</li></ol><h2 id="方案"><a href="#方案" class="headerlink" title="方案"></a>方案</h2><p>使用官方推荐工具Curator处理以上问题</p><table><thead><tr><th style="text-align:center">日志名称</th><th style="text-align:center">保留天数</th><th style="text-align:center">关闭天数</th><th style="text-align:center">定时任务</th><th style="text-align:center">备注</th></tr></thead><tbody><tr><td style="text-align:center">logstash-wapnginx-access-</td><td style="text-align:center">60</td><td style="text-align:center">7天前</td><td style="text-align:center">每隔7天0:00执行</td><td style="text-align:center">7天做周报使用</td></tr><tr><td style="text-align:center">logstash-restapi-access-</td><td style="text-align:center">60</td><td style="text-align:center">7天前</td><td style="text-align:center">每隔7天0:00执行</td><td style="text-align:center">7天做周报使用</td></tr><tr><td style="text-align:center">……</td><td style="text-align:center">……</td><td style="text-align:center">……</td><td style="text-align:center">……</td><td style="text-align:center">……</td><td>…….</td></tr></tbody></table><p><strong>说明:</strong></p><ol><li>索引保存60</li><li>关闭索引为7天(提高 ELK 响应速度)</li><li>运维人员手动打开需要使用的时间长度的索引</li></ol><blockquote><p>Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your indices.<br>像博物馆馆长一样，管理展览和收藏品，Elasticsearch Curator可以帮助您策划或管理您的索引。</p></blockquote><p><strong>Compatibility Matrix:</strong></p><table><thead><tr><th style="text-align:center">version</th><th style="text-align:center">ES 1.x</th><th style="text-align:center">AWS ES 1.x</th><th style="text-align:center">ES 2.x</th><th style="text-align:center">AWS ES 2.x</th><th style="text-align:center">ES 5.x</th><th style="text-align:center">AWS ES 5.x </th></tr></thead><tbody><tr><td style="text-align:center">3</td><td style="text-align:center">yes</td><td style="text-align:center">yes*</td><td style="text-align:center">yes</td><td style="text-align:center">yes*</td><td style="text-align:center">no</td><td style="text-align:center">no</td></tr><tr><td style="text-align:center">4</td><td style="text-align:center">no</td><td style="text-align:center">no</td><td style="text-align:center">yes</td><td style="text-align:center">no</td><td style="text-align:center">yes</td><td style="text-align:center">no</td></tr><tr><td style="text-align:center">5</td><td style="text-align:center">no</td><td style="text-align:center">no</td><td style="text-align:center">no</td><td style="text-align:center">no</td><td style="text-align:center">yes</td><td style="text-align:center">yes*</td></tr></tbody></table><h2 id="方案实施"><a href="#方案实施" class="headerlink" title="方案实施"></a>方案实施</h2><ul><li><p>Curator安装</p><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html" target="_blank" rel="noopener">安装连接</a></li></ul></li><li><p>本文使用 pip 进行安装</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 默认安装curator, version 5.1.2</span></span><br><span class="line">pip install elasticsearch-curator</span><br></pre></td></tr></table></figure></li></ul><ul><li><p>Curator配置</p><p> <strong>主配置文件:</strong></p><blockquote><p>Curator默认查找配置文件的的地址为：~/.curator/curator.yml 指定自己的文件位置</p></blockquote><p> <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/configfile.html" target="_blank" rel="noopener">配置详细介绍</a></p> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"># cat /software/elasticsearch/curator.yml</span><br><span class="line">client: </span><br><span class="line"> hosts: # elasticsearch 主机</span><br><span class="line"> - 10.200.200.77</span><br><span class="line"> - 10.200.200.80</span><br><span class="line"> port: 9200 # 端口</span><br><span class="line"> url_prefix:</span><br><span class="line"> use_ssl: False</span><br><span class="line"> certificate:</span><br><span class="line"> client_cert:</span><br><span class="line"> client_key:</span><br><span class="line"> aws_key:</span><br><span class="line"> aws_secret_key:</span><br><span class="line"> aws_region:</span><br><span class="line"> ssl_no_validate: False</span><br><span class="line"> http_auth:</span><br><span class="line"> timeout: 30</span><br><span class="line"> master_only: False</span><br><span class="line">logging: </span><br><span class="line"> loglevel: INFO</span><br><span class="line"> logfile:</span><br><span class="line"> logformat: default</span><br><span class="line"> blacklist: ['elasticsearch', 'urllib3']</span><br></pre></td></tr></table></figure></li></ul><p><strong>删除索引配置</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat /software/elasticsearch/delete_indeces.yml</span></span><br><span class="line">actions: </span><br><span class="line"> <span class="number">1</span>:</span><br><span class="line"> action: delete_indices</span><br><span class="line"> description: >-</span><br><span class="line"> Delete indices older than <span class="number">60</span> days (based on index name), <span class="keyword">for</span> logstash-wapnginx-access-</span><br><span class="line"> prefixed indices. Ignore the error <span class="keyword">if</span> the filter does <span class="keyword">not</span> result <span class="keyword">in</span> an</span><br><span class="line"> actionable list of indices (ignore_empty_list) <span class="keyword">and</span> exit cleanly.</span><br><span class="line"> options:</span><br><span class="line"> ignore_empty_list: <span class="keyword">True</span></span><br><span class="line"> timeout_override:</span><br><span class="line"> continue_if_exception: <span class="keyword">False</span></span><br><span class="line"> disable_action: false</span><br><span class="line"> filters:</span><br><span class="line"> - filtertype: pattern</span><br><span class="line"> kind: prefix</span><br><span class="line"> value: logstash-wapnginx-access-</span><br><span class="line"> exclude:</span><br><span class="line"> - filtertype: age</span><br><span class="line"> source: name</span><br><span class="line"> direction: older</span><br><span class="line"> timestring: <span class="string">'%Y.%m.%d'</span></span><br><span class="line"> unit: days</span><br><span class="line"> unit_count: <span class="number">60</span></span><br><span class="line"> exclude:</span><br><span class="line"> <span class="number">2</span>:</span><br><span class="line"> action: delete_indices</span><br><span class="line"> description: >-</span><br><span class="line"> Delete indices older than <span class="number">60</span> days (based on index name), <span class="keyword">for</span> logstash-restapi-access-</span><br><span class="line"> prefixed indices. Ignore the error <span class="keyword">if</span> the filter does <span class="keyword">not</span> result <span class="keyword">in</span> an</span><br><span class="line"> actionable list of indices (ignore_empty_list) <span class="keyword">and</span> exit cleanly.</span><br><span class="line"> options:</span><br><span class="line"> ignore_empty_list: <span class="keyword">True</span></span><br><span class="line"> timeout_override:</span><br><span class="line"> continue_if_exception: <span class="keyword">False</span></span><br><span class="line"> disable_action: false</span><br><span class="line"> filters:</span><br><span class="line"> - filtertype: pattern</span><br><span class="line"> kind: prefix</span><br><span class="line"> value: logstash-restapi-access-</span><br><span class="line"> exclude:</span><br><span class="line"> - filtertype: age</span><br><span class="line"> source: name</span><br><span class="line"> direction: older</span><br><span class="line"> timestring: <span class="string">'%Y.%m.%d'</span></span><br><span class="line"> unit: days</span><br><span class="line"> unit_count: <span class="number">60</span></span><br><span class="line"> exclude:</span><br></pre></td></tr></table></figure><p><strong>关闭索引配置文件:</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"># cat /software/elasticsearch/close_indeces.yml</span><br><span class="line">---</span><br><span class="line">actions:</span><br><span class="line"> 1:</span><br><span class="line"> action: close</span><br><span class="line"> description: >-</span><br><span class="line"> Close indices older than 7 days (based on index name), for logstash-</span><br><span class="line"> prefixed indices.</span><br><span class="line"> options:</span><br><span class="line"> delete_aliases: False</span><br><span class="line"> disable_action: False</span><br><span class="line"> filters:</span><br><span class="line"> - filtertype: pattern</span><br><span class="line"> kind: prefix</span><br><span class="line"> value: logstash-</span><br><span class="line"> - filtertype: age</span><br><span class="line"> source: name</span><br><span class="line"> direction: older</span><br><span class="line"> timestring: '%Y.%m.%d'</span><br><span class="line"> unit: days</span><br><span class="line"> unit_count: 7</span><br></pre></td></tr></table></figure><p><strong>打开索引配置文件:</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line">---</span><br><span class="line">actions:</span><br><span class="line"> 1:</span><br><span class="line"> action: open</span><br><span class="line"> description: >-</span><br><span class="line"> Open indices older than 7 days but younger than 30 days (based on index</span><br><span class="line"> name), for logstash- prefixed indices.</span><br><span class="line"> options:</span><br><span class="line"> disable_action: False</span><br><span class="line"> filters:</span><br><span class="line"> - filtertype: pattern</span><br><span class="line"> kind: prefix</span><br><span class="line"> value: logstash-</span><br><span class="line"> exclude:</span><br><span class="line"> - filtertype: age</span><br><span class="line"> source: name</span><br><span class="line"> direction: older</span><br><span class="line"> timestring: '%Y.%m.%d'</span><br><span class="line"> unit: days</span><br><span class="line"> unit_count: 7</span><br><span class="line"> - filtertype: age</span><br><span class="line"> source: name</span><br><span class="line"> direction: younger</span><br><span class="line"> timestring: '%Y.%m.%d'</span><br><span class="line"> unit: days</span><br><span class="line"> unit_count: 30</span><br></pre></td></tr></table></figure><pre><code>**定时任务:**<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"># cat /software/script/schedule_script.sh</span><br><span class="line"># !/bin/sh</span><br><span class="line"></span><br><span class="line"># 定时删除大于一定时间的日志数据</span><br><span class="line"></span><br><span class="line">date=`date "+%Y-%m-%d %H:%M"` </span><br><span class="line">echo "============================= begin at: $date ===============================" >> /software/elasticsearch/logs/delete_indeces.log</span><br><span class="line"></span><br><span class="line">/usr/bin/curator --config /path/to/curator.yml /path/to/delete_indeces.yml >> /software/elasticsearch/logs/delete_indeces.log</span><br><span class="line"></span><br><span class="line">echo "==================================== end =====================================" >> /software/elasticsearch/logs/delete_indeces.log</span><br><span class="line"></span><br><span class="line"># 定时关闭一定时间的索引</span><br><span class="line"></span><br><span class="line">echo "============================= begin at: $date ===============================" >> /software/elasticsearch/logs/close_indeces.log</span><br><span class="line"></span><br><span class="line">/usr/bin/curator --config /software/elasticsearch/curator.yml /software/elasticsearch/close_indeces.yml >> /software/elasticsearch/logs/close_indeces.log</span><br><span class="line"></span><br><span class="line">echo "==================================== end =====================================" >> /software/elasticsearch/logs/close_indeces.log</span><br><span class="line"></span><br><span class="line"># crontab -l</span><br><span class="line">#For Auto elasticsearch indeces</span><br><span class="line">0 0 */7 * * /software/script/schedule_script.sh</span><br></pre></td></tr></table></figure></code></pre>]]></content>
<categories>
<category> Monitor </category>
</categories>
<tags>
<tag> Elasticsearch </tag>
</tags>
</entry>
<entry>
<title>Nginx_nginx-limit</title>
<link href="//blog/Nginx-nginx-limit.html"/>
<content type="html"><![CDATA[<p>典型的电商在负载均衡端进行简单的流量限制.但是应该如何配置? 下文详细介绍</p><a id="more"></a><h1 id="限流配置"><a href="#限流配置" class="headerlink" title="限流配置"></a>限流配置</h1><p>##流量限制</p><h3 id="说明"><a href="#说明" class="headerlink" title="说明"></a>说明</h3><blockquote><p>流量限制(rate-limiting)，是Nginx中一个非常实用，却经常被错误理解和错误配置的功能。我们可以用来限制用户在给定时间内HTTP请求的数量。请求，可以是一个简单网站首页的GET请求，也可以是登录表单的POST请求。</p></blockquote><h3 id="如何限流"><a href="#如何限流" class="headerlink" title="如何限流"></a>如何限流</h3><blockquote><p>Nginx的”流量限制”使用漏桶算法(leaky bucket algorithm)，该算法在通讯和分组交换计算机网络中广泛使用，用以处理带宽有限时的突发情况。就好比，一个桶口在倒水，桶底在漏水的水桶。如果桶口倒水的速率大于桶底的漏水速率，桶里面的水将会溢出；同样，在请求处理方面，水代表来自客户端的请求，水桶代表根据”先进先出调度算法”(FIFO)等待被处理的请求队列，桶底漏出的水代表离开缓冲区被服务器处理的请求，桶口溢出的水代表被丢弃和不被处理的请求。</p></blockquote><h3 id="配置基本的限流"><a href="#配置基本的限流" class="headerlink" title="配置基本的限流"></a>配置基本的限流</h3><p>流量限制”配置两个主要的指令，<code>limit_req_zone</code>和<code>limit_req</code>，如下所示：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;</span><br><span class="line"></span><br><span class="line">server {</span><br><span class="line">location /login/ {</span><br><span class="line">limit_req zone=mylimit;</span><br><span class="line"></span><br><span class="line">proxy_pass http://my_upstream;</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><code>limit_req_zone</code>指令定义了流量限制相关的参数,而<code>limit_req</code>指令在出现的上下文中启用流量限制<br><code>limit_req_zone</code>指令通常在HTTP块中定义，使其可在多个上下文中使用，它需要以下三个参数：</p><ul><li><p>Key - 定义应用限制的请求特性。示例中的Nginx变量<code>$binary_remote_addr</code>，保存客户端IP地址的二进制形式。这意味着，我们可以将每个不同的IP地址限制到，通过第三个参数设置的请求速率。(使用该变量是因为比字符串形式的客户端IP地址<code>$remote_addr</code>，占用更少的空间)</p></li><li><p>Zone - 定义用于存储每个IP地址状态以及被限制请求URL访问频率的共享内存区域。保存在内存共享区域的信息，意味着可以在Nginx的worker进程之间共享。定义分为两个部分：通过<code>zone=keyword</code>标识区域的名字，以及冒号后面跟区域大小。16000个IP地址的状态信息，大约需要1MB，所以示例中区域可以存储160000个IP地址。</p></li><li><p>Rate - 定义最大请求速率。在示例中，速率不能超过每秒10个请求。Nginx实际上以毫秒的粒度来跟踪请求，所以速率限制相当于每100毫秒1个请求。因为不允许”突发情况”(见下一章节)，这意味着在前一个请求100毫秒内到达的请求将被拒绝。</p></li></ul><p><code>limit_req_zone</code>指令设置流量限制和共享内存区域的参数，但实际上并不限制请求速率。所以需要通过添加<code>limit_req</code>指令，将流量限制应用在特定的<code>location</code>或者<code>server</code>块。在上面示例中，我们对<code>/login/</code>请求进行流量限制。</p><p>现在每个IP地址被限制为每秒只能请求10次<code>/login/</code>，更准确地说，在前一个请求的100毫秒内不能请求该URL。</p><h3 id="处理突发"><a href="#处理突发" class="headerlink" title="处理突发"></a>处理突发</h3><p>如果我们在100毫秒内接收到2个请求，怎么办？对于第二个请求，Nginx将给客户端返回状态码503。这可能并不是我们想要的结果，因为应用本质上趋向于突发性。相反地，我们希望缓冲任何超额的请求，然后及时地处理它们。我们更新下配置，在<code>limit_req</code>中使用<code>burst</code>参数：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">location /login/ {</span><br><span class="line">limit_req zone=mylimit burst=20;</span><br><span class="line">proxy_pass http://my_upstream;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><code>burst</code>参数定义了超出zone指定速率的情况下(示例中的<code>mylimit</code>区域，速率限制在每秒10个请求，或每100毫秒一个请求)，客户端还能发起多少请求。上一个请求100毫秒内到达的请求将会被放入队列，我们将队列大小设置为20。</p><p>这意味着，如果从一个给定IP地址发送21个请求，Nginx会立即将第一个请求发送到上游服务器群，然后将余下20个请求放在队列中。然后每100毫秒转发一个排队的请求，只有当传入请求使队列中排队的请求数超过20时，Nginx才会向客户端返回503。</p><h3 id="无延迟的排队"><a href="#无延迟的排队" class="headerlink" title="无延迟的排队"></a>无延迟的排队</h3><p>配置<code>burst</code>参数将会使通讯更流畅，但是可能会不太实用，因为该配置会使站点看起来很慢。在上面的示例中，队列中的第20个包需要等待2秒才能被转发，此时返回给客户端的响应可能不再有用。要解决这个情况，可以在<code>burst</code>参数后添加nodelay参数：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">location /login/ {</span><br><span class="line">limit_req zone=mylimit burst=20 nodelay;</span><br><span class="line"></span><br><span class="line">proxy_pass http://my_upstream;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>使用<code>nodelay</code>参数，Nginx仍将根据<code>burst</code>参数分配队列中的位置，并应用已配置的速率限制，而不是清理队列中等待转发的请求。相反地，当一个请求到达“太早”时，只要在队列中能分配位置，Nginx将立即转发这个请求。将队列中的该位置标记为”taken”(占据)，并且不会被释放以供另一个请求使用，直到一段时间后才会被释放(在这个示例中是，100毫秒后)。</p><p>假设如前所述，队列中有20个空位，从给定的IP地址发出的21个请求同时到达。Nginx会立即转发这个21个请求，并且标记队列中占据的20个位置，然后每100毫秒释放一个位置。如果是25个请求同时到达，Nginx将会立即转发其中的21个请求，标记队列中占据的20个位置，并且返回503状态码来拒绝剩下的4个请求。</p><p>现在假设，第一组请求被转发后101毫秒，另20个请求同时到达。队列中只会有一个位置被释放，所以Nginx转发一个请求并返回503状态码来拒绝其他19个请求。如果在20个新请求到达之前已经过去了501毫秒，5个位置被释放，所以Nginx立即转发5个请求并拒绝另外15个。</p><p>效果相当于每秒10个请求的“流量限制”。如果希望不限制两个请求间允许间隔的情况下实施“流量限制”，nodelay参数是很实用的。</p><p>注意： 对于大部分部署，我们建议使用<code>burst</code>和<code>nodelay</code>参数来配置<code>limit_req</code>指令。</p>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Nginx </tag>
</tags>
</entry>
<entry>
<title>Nginx_Tcp-Loadbalancing</title>
<link href="//blog/Nginx-Tcp-Loadbalance.html"/>
<content type="html"><![CDATA[<p>日常使用过程中大多使用 <code>upstream</code> 进行 http 的负载均衡, 那么 <code>TCP</code> 协议的负载均衡如何配置呢? 请看下文</p><a id="more"></a><h1 id="Nginx-TCP-Load-Balance"><a href="#Nginx-TCP-Load-Balance" class="headerlink" title="Nginx TCP Load Balance"></a>Nginx TCP Load Balance</h1><h2 id="配置-TCP-负载均衡"><a href="#配置-TCP-负载均衡" class="headerlink" title="配置 TCP 负载均衡"></a>配置 TCP 负载均衡</h2><blockquote><p>The ngx_stream_core_module module is available since version 1.9.0. This module is not built by default, it should be enabled with the –with-stream configuration parameter.</p></blockquote><p><em><strong>安装使用 stream 模块:</strong></em></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line">1. 安装 stream (编译 Nginx 是增加--with-stream选项)</span><br><span class="line">2. 配置 Nginx stream 模块(与 http 同级) TCP 代理</span><br><span class="line">worker_processes auto;</span><br><span class="line"></span><br><span class="line">error_log /var/<span class="built_in">log</span>/nginx/error.log info;</span><br><span class="line"></span><br><span class="line">events {</span><br><span class="line"> worker_connections 1024;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">stream {</span><br><span class="line"> upstream backend {</span><br><span class="line"> <span class="built_in">hash</span> <span class="variable">$remote_addr</span> consistent;</span><br><span class="line"></span><br><span class="line"> server backend1.example.com:12345 weight=5;</span><br><span class="line"> server 127.0.0.1:12345 max_fails=3 fail_timeout=30s;</span><br><span class="line"> server unix:/tmp/backend3;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> upstream dns {</span><br><span class="line"> server 192.168.0.1:53535;</span><br><span class="line"> server dns.example.com:53;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen 12345;</span><br><span class="line"> proxy_connect_timeout 1s;</span><br><span class="line"> proxy_timeout 3s;</span><br><span class="line"> proxy_pass backend;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen 127.0.0.1:53 udp;</span><br><span class="line"> proxy_responses 1;</span><br><span class="line"> proxy_timeout 20s;</span><br><span class="line"> proxy_pass dns;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> server {</span><br><span class="line"> listen [::1]:12345;</span><br><span class="line"> proxy_pass unix:/tmp/stream.socket;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>##使用实例</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">邮箱代理服务器</span><br><span class="line">stream {</span><br><span class="line"> upstream mail {</span><br><span class="line"> server smtp.163.com:25;</span><br><span class="line"> }</span><br><span class="line"> server {</span><br><span class="line"> listen 25;</span><br><span class="line"> proxy_pass mail;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Nginx </tag>
</tags>
</entry>
<entry>
<title>Nginx会话保持</title>
<link href="//blog/Nginx%E4%BC%9A%E8%AF%9D%E4%BF%9D%E6%8C%81.html"/>
<content type="html"><![CDATA[<p>在整个网络前端没有 F5的情况下,运维同事们该如何进行后端服务的集群化? 实现了集群化该如何实现集群内的服务进行会话保持? 接下来进行详细讲解</p><a id="more"></a><h1 id="Nginx"><a href="#Nginx" class="headerlink" title="Nginx"></a>Nginx</h1><h2 id="配置会话保持"><a href="#配置会话保持" class="headerlink" title="配置会话保持"></a><em><strong>配置会话保持</strong></em></h2><p><em><strong>Sticky工作原理 ：</strong></em></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">Sticky是nginx的一个模块，它是基于cookie的一种nginx的负载均衡解决方案，通过分发和识别cookie，来使同一个客户端的请求落在同一台服务器上，默认标识名为route</span><br><span class="line">1.客户端首次发起访问请求，nginx接收后，发现请求头没有cookie，则以轮询方式将请求分发给后端服务器。</span><br><span class="line">2.后端服务器处理完请求，将响应数据返回给nginx。</span><br><span class="line">3.此时nginx生成带route的cookie，返回给客户端。route的值与后端服务器对应，可能是明文，也可能是md5、sha1等Hash值</span><br><span class="line">4.客户端接收请求，并保存带route的cookie。</span><br><span class="line">5.当客户端下一次发送请求时，会带上route，nginx根据接收到的cookie中的route值，转发给对应的后端服务器</span><br></pre></td></tr></table></figure><h3 id="Sticky-模块使用"><a href="#Sticky-模块使用" class="headerlink" title="Sticky 模块使用:"></a><em><strong>Sticky 模块使用:</strong></em></h3><blockquote><p>sticky [name=route] [domain=.foo.bar] [path=/] [expires=1h] [hash=index|md5|sha1] [no_fallback] [secure] [httponly];</p></blockquote><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[name=route] 设置用来记录会话的cookie名称</span><br><span class="line">[domain=.foo.bar] 设置cookie作用的域名</span><br><span class="line">[path=/] 设置cookie作用的URL路径，默认根目录</span><br><span class="line">[expires=1h] 设置cookie的生存期，默认不设置，浏览器关闭即失效，需要是大于1秒的值</span><br><span class="line">[hash=index|md5|sha1] 设置cookie中服务器的标识是用明文还是使用md5值，默认使用md5</span><br><span class="line">[no_fallback] 设置该项，当sticky的后端机器挂了以后，nginx返回502 (Bad Gateway or Proxy Error) ，而不转发到其他服务器，不建议设置</span><br><span class="line">[secure] 设置启用安全的cookie，需要HTTPS支持</span><br><span class="line">[httponly] 允许cookie不通过JS泄漏</span><br></pre></td></tr></table></figure><h3 id="Sticky-其他的参数-语法："><a href="#Sticky-其他的参数-语法：" class="headerlink" title="Sticky 其他的参数-语法："></a><em><strong>Sticky 其他的参数-语法：</strong></em></h3><blockquote><p>session_sticky [cookie=name] [domain=your_domain] [path=your_path] [maxage=time][mode=insert|rewrite|prefix] [option=indirect] [maxidle=time] [maxlife=time] [fallback=on|off] [hash=plain|md5]</p></blockquote><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">mode设置cookie的模式:</span><br><span class="line"> insert: 在回复中本模块通过Set-Cookie头直接插入相应名称的cookie。</span><br><span class="line"> prefix: 不会生成新的cookie，但会在响应的cookie值前面加上特定的前缀，当浏览器带着这个有特定标识的cookie再次请求时，模块在传给后端服务前先删除加入的前缀，后端服务拿到的还是原来的cookie值，这些动作对后端透明。如：”Cookie: NAME=SRV~VALUE”。</span><br><span class="line"> rewrite: 使用服务端标识覆盖后端设置的用于session sticky的cookie。如果后端服务在响应头中没有设置该cookie，则认为该请求不需要进行session sticky，使用这种模式，后端服务可以控制哪些请求需要sesstion sticky，哪些请求不需要。</span><br><span class="line">option设置用于session sticky的cookie的选项，可设置成indirect或direct。indirect不会将session sticky的cookie传送给后端服务，该cookie对后端应用完全透明。direct则与indirect相反。</span><br><span class="line">maxidle设置session cookie的最长空闲的超时时间</span><br><span class="line">maxlife设置session cookie的最长生存期</span><br><span class="line">maxage是cookie的生存期。不设置时，浏览器或App关闭后就失效。下次启动时，又会随机分配后端服务器。所以如果希望该客户端的请求长期落在同一台后端服务器上，可以设置maxage。</span><br><span class="line">hash不论是明文还是hash值，都有固定的数目。因为hash是server的标识，所以有多少个server，就有等同数量的hash值。</span><br></pre></td></tr></table></figure><h3 id="平滑升级nginx，安装nginx-sticky-module"><a href="#平滑升级nginx，安装nginx-sticky-module" class="headerlink" title="平滑升级nginx，安装nginx-sticky-module"></a><em><strong>平滑升级nginx，安装nginx-sticky-module</strong></em></h3><blockquote><p>模块下载地址：<a href="https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/get/master.tar.gz" target="_blank" rel="noopener">https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/get/master.tar.gz</a></p></blockquote><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># tar zxf nginx-goodies-nginx-sticky-module-ng-1e96371de59f.tar.gz</span></span><br><span class="line"><span class="comment"># mv nginx-goodies-nginx-sticky-module-ng-1e96371de59f nginx-sticky</span></span><br><span class="line"><span class="comment"># tar zxf nginx-1.10.0.tar.gz</span></span><br><span class="line"><span class="comment"># cd nginx-1.10.0</span></span><br><span class="line"><span class="comment"># ./configure --prefix=/software/nginx/ --with-pcre=/software/packages/pcre-8.39 --with-zlib=/software/packages/zlib-1.2.8 --user=nginx --group=nginx --with-http_stub_status_module --with-http_gzip_static_module --with-http_sub_module --with-http_ssl_module --add-module=/software/packages/nginx-sticky/</span></span><br><span class="line"><span class="comment"># make</span></span><br><span class="line"><span class="comment"># cd /software/nginx/sbin</span></span><br><span class="line"><span class="comment"># mv nginx nginx.old</span></span><br><span class="line"><span class="comment"># cp /software/packages/nginx-1.10.0/objs/nginx ./</span></span><br><span class="line"><span class="comment"># cd -</span></span><br><span class="line"><span class="comment"># make upgrade</span></span><br></pre></td></tr></table></figure><h3 id="服务器增加配置"><a href="#服务器增加配置" class="headerlink" title="服务器增加配置"></a><em><strong>服务器增加配置</strong></em></h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">upstream backend-merchat {</span><br><span class="line"> sticky;</span><br><span class="line"> server 192.168.0.15:8080 max_fails=0;</span><br><span class="line"> server 192.168.0.22:8080 max_fails=0;</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Nginx </tag>
</tags>
</entry>
<entry>
<title>Tomcat配置分享</title>
<link href="//blog/Tomcat%E9%85%8D%E7%BD%AE%E5%88%86%E4%BA%AB.html"/>
<content type="html"><![CDATA[<p>Tomcat 日常使用过程中遇到问题及解决分享…</p><a id="more"></a><h1 id="Tomcat8-启动异常"><a href="#Tomcat8-启动异常" class="headerlink" title="Tomcat8 启动异常"></a>Tomcat8 启动异常</h1><h2 id="问题"><a href="#问题" class="headerlink" title="问题:"></a><em><strong>问题:</strong></em></h2><p> tomcat7/8 启动慢</p><h2 id="问题原因"><a href="#问题原因" class="headerlink" title="问题原因:"></a><em><strong>问题原因:</strong></em></h2><p><strong><code>SHA1PRNG</code>算法是基于<code>SHA-1</code>算法实现且保密性较强的伪随机数生成器</strong>。</p><p><strong>在<code>SHA1PRNG</code>中，有一个种子产生器，它根据配置执行各种操作。</strong></p><p>1）如果<code>Java.security.egd</code>属性或<code>securerandom.source</code>属性指定的是<code>"file:/dev/random"</code>或<code>"file:/dev/urandom"</code>，那么JVM会使用本地种子产生器<code>NativeSeedGenerator</code>，它会调用<code>super()</code>方法，即调用<code>SeedGenerator.URLSeedGenerator(/dev/random)</code>方法进行初始化。</p><p>2）如果<code>java.security.egd</code>属性或<code>securerandom.source</code>属性指定的是其它已存在的URL，那么会调用<code>SeedGenerator.URLSeedGenerator(url)</code>方法进行初始化。</p><p>这就是为什么我们设置值为<code>"file:///dev/urandom"</code>或者值为<code>"file:/./dev/random"</code>都会起作用的原因。</p><p>在这个实现中，产生器会评估熵池<code>（entropy pool）</code>中的噪声数量。随机数是从熵池中进行创建的。当读操作时，<code>/dev/random</code>设备会只返回熵池中噪声的随机字节。<code>/dev/random</code>非常适合那些需要非常高质量随机性的场景，比如一次性的支付或生成密钥的场景。</p><p>当熵池为空时，来自<code>/dev/random</code>的读操作将被阻塞，直到熵池收集到足够的环境噪声数据。这么做的目的是成为一个密码安全的伪随机数发生器，熵池要有尽可能大的输出。对于生成高质量的加密密钥或者是需要长期保护的场景，一定要这么做。</p><p>那么什么是环境噪声？</p><p>随机数产生器会手机来自设备驱动器和其它源的环境噪声数据，并放入熵池中。产生器会评估熵池中的噪声数据的数量。当熵池为空时，这个噪声数据的收集是比较花时间的。这就意味着，Tomcat在生产环境中使用熵池时，会被阻塞较长的时间。</p><h2 id="解决办法"><a href="#解决办法" class="headerlink" title="解决办法:"></a><strong>解决办法:</strong></h2><p>有两种解决办法：</p><p>1）在<code>Tomcat</code>环境中解决</p><p> 可以通过配置JRE使用非阻塞的<code>Entropy Source</code>。</p><p> 在catalina.sh中CATALINA_OPTS加入这么一行：-Djava.security.egd=file:/dev/./urandom 即可。</p><p> 加入后再启动Tomcat</p><p>2）在JVM环境中解决</p><p> 打开$JAVA_PATH/jre/lib/security/java.security这个文件，找到下面的内容：</p><p> securerandom.source=file:/dev/urandom </p><p> 替换成</p><p> securerandom.source=file:/dev/./urandom</p><hr><h1 id="Tomcat-报错"><a href="#Tomcat-报错" class="headerlink" title="Tomcat 报错"></a>Tomcat 报错</h1><h2 id="错误信息"><a href="#错误信息" class="headerlink" title="错误信息"></a><strong>错误信息</strong></h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Error occurred during initialization of VM</span><br><span class="line">Too small initial heap <span class="keyword">for</span> new size specified</span><br></pre></td></tr></table></figure><h3 id="错误详解"><a href="#错误详解" class="headerlink" title="错误详解"></a><strong>错误详解</strong></h3><ol><li>错误信息有误导，不一定是堆内存大小设置不够</li><li>Xms可以设置与-Xmx相同，但是最好别把三个都设置一样</li></ol><h3 id="报错原因"><a href="#报错原因" class="headerlink" title="报错原因"></a><strong>报错原因</strong></h3><p><strong>对xmn理解错误(Xmn不能大于等于Xmx)</strong></p><blockquote><p>-Xmn2g ：设置年轻代大小为2G。整个堆大小=年轻代大小 + 年老代大小 + 持久代大小 。持久代一般固定大小为64m，所以增大年轻代后，将会减小年老代大小。此值对系统性能影响较大，Sun官方推荐配置为整个堆的3/8。</p></blockquote><hr><h1 id="Tomcat-报错相信信息排查"><a href="#Tomcat-报错相信信息排查" class="headerlink" title="Tomcat 报错相信信息排查"></a>Tomcat 报错相信信息排查</h1><h2 id="开启-Debug"><a href="#开启-Debug" class="headerlink" title="开启 Debug"></a>开启 Debug</h2><p><strong>开启 debug 模式进行深层次的排错</strong></p><ol><li>${tomcat_home}/lib/logging.properties # 创建该文件</li><li><p>添加如下内容</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler</span><br><span class="line"><span class="comment">############################################################</span></span><br><span class="line"><span class="comment"># Handler specific properties.</span></span><br><span class="line"><span class="comment"># Describes specific configuration info for Handlers.</span></span><br><span class="line"><span class="comment">############################################################</span></span><br><span class="line"></span><br><span class="line">org.apache.juli.FileHandler.level = FINE</span><br><span class="line">org.apache.juli.FileHandler.directory = <span class="variable">${catalina.base}</span>/logs</span><br><span class="line">org.apache.juli.FileHandler.prefix = error-debug.</span><br><span class="line"></span><br><span class="line">java.util.logging.ConsoleHandler.level = FINE</span><br><span class="line">java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter</span><br></pre></td></tr></table></figure></li><li><p>查看${tomcat_home}/logs/error……</p></li></ol><hr><h1 id="Tomcat-黑白名单"><a href="#Tomcat-黑白名单" class="headerlink" title="Tomcat 黑白名单"></a>Tomcat 黑白名单</h1><h2 id="Tomcat-开启访问限制"><a href="#Tomcat-开启访问限制" class="headerlink" title="Tomcat 开启访问限制"></a>Tomcat 开启访问限制</h2><p><em><strong>Tomcat 配置文件更改</strong></em></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat ${tomcat_home}/config/server.conf</span></span><br><span class="line"><Context crossContext=<span class="string">"true"</span> docBase=<span class="string">"/software/deploy_jenkins/jenkins"</span> reloadable=<span class="string">"true"</span> allowLinking=<span class="string">"true"</span> path=<span class="string">""</span>></span><br><span class="line"> <Valve className=<span class="string">"org.apache.catalina.valves.RemoteAddrValve"</span> allow=<span class="string">"10.200.200.91"</span> deny=<span class="string">""</span>/></span><br><span class="line"></Context></span><br></pre></td></tr></table></figure><h1 id="Tomcat-流量转换"><a href="#Tomcat-流量转换" class="headerlink" title="Tomcat 流量转换"></a>Tomcat 流量转换</h1><blockquote><p>HTTP 转 HTTPS( 在证书存在负载均衡端)</p></blockquote><h2 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">Make Tomcat Understand X-Forwarded-Proto by adding the following text in the Tomcat server.xml <Engine> section. You only need to change this if you are using the shared Jelastic SSL certificate or if you are using a load balancer.</span><br><span class="line"></span><br><span class="line"><Valve className="org.apache.catalina.valves.RemoteIpValve"</span><br><span class="line">remoteIpHeader="x-forwarded-for"</span><br><span class="line">protocolHeader="x-forwarded-proto"</span><br><span class="line">protocolHeaderHttpsValue="https" /></span><br></pre></td></tr></table></figure><h1 id="Tomcat代理配置"><a href="#Tomcat代理配置" class="headerlink" title="Tomcat代理配置"></a>Tomcat代理配置</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># catalina.sh JAVA_OPT=添加如下</span></span><br><span class="line">-Dhttp.proxyHost=192.168.0.9 -Dhttp.proxyPort=37128</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Tomcat </tag>
</tags>
</entry>
<entry>
<title>Mongodb日志清理</title>
<link href="//blog/Mongodb%E6%97%A5%E5%BF%97%E6%B8%85%E7%90%86.html"/>
<content type="html"><![CDATA[<p>日常生产环境或者测试环境 Mongodb 日志无线增大该怎么处理? 一下介绍 Mongodb 日志清理</p><a id="more"></a><h1 id="Mongodb-日志清理"><a href="#Mongodb-日志清理" class="headerlink" title="Mongodb 日志清理"></a>Mongodb 日志清理</h1><h2 id="手动清理日志-不重启"><a href="#手动清理日志-不重启" class="headerlink" title="手动清理日志(不重启)"></a><em><strong>手动清理日志(不重启)</strong></em></h2><ol><li>连接 mongodb</li><li>使用 admin (use admin)</li><li>运行 db.runCommand({logRotate:1}) 阻断当前日志重新生成新的日志</li></ol><h2 id="自动定时任务-切割日志"><a href="#自动定时任务-切割日志" class="headerlink" title="自动定时任务(切割日志)"></a>自动定时任务(切割日志)</h2><pre><code><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="comment"># 程序名称</span></span><br><span class="line">app=<span class="string">"master.conf"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 日志存放路径</span></span><br><span class="line">logfile=/software/mongodb/<span class="built_in">log</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 保留日志天数</span></span><br><span class="line">days=7</span><br><span class="line"></span><br><span class="line"><span class="comment"># 程序PID</span></span><br><span class="line">mongodb_pid=`ps aux | grep mongodb | grep <span class="variable">${app}</span> | awk <span class="string">'{print $2}'</span>`</span><br><span class="line"></span><br><span class="line"><span class="comment"># 切割日志</span></span><br><span class="line">/bin/<span class="built_in">kill</span> -SIGUSR1 <span class="variable">${mongodb_pid}</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 删除几天前的日志</span></span><br><span class="line">find <span class="variable">$logfile</span>/ -mtime +<span class="variable">${days}</span> -delete</span><br></pre></td></tr></table></figure></code></pre><hr>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Mongodb </tag>
</tags>
</entry>
<entry>
<title>Nexus安装与迁移</title>
<link href="//blog/Nexus%E5%AE%89%E8%A3%85%E4%B8%8E%E8%BF%81%E7%A7%BB.html"/>
<content type="html"><![CDATA[<p>Maven registry 私有仓库搭建方法</p><a id="more"></a><h1 id="Maven-registry-maven私有仓库）"><a href="#Maven-registry-maven私有仓库）" class="headerlink" title="Maven registry(maven私有仓库）"></a>Maven registry(maven私有仓库）</h1><h3 id="安装配置-Nexus"><a href="#安装配置-Nexus" class="headerlink" title="安装配置 Nexus"></a>安装配置 Nexus</h3><ul><li><strong>JAVA 配置</strong></li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> JAVA_HOME=/software/jdk1.7.0_79</span><br><span class="line"><span class="built_in">export</span> JRE_HOME=<span class="variable">${JAVA_HOME}</span>/jre</span><br><span class="line"><span class="built_in">export</span> CLASSPATH=.:<span class="variable">${JAVA_HOME}</span>/lib:<span class="variable">${JRE_HOME}</span>/lib</span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">${JAVA_HOME}</span>/bin:<span class="variable">$PATH</span></span><br></pre></td></tr></table></figure><ul><li><p><strong>下载nexus</strong></p><p> <a href="http://www.sonatype.org/nexus/go" target="_blank" rel="noopener">http://www.sonatype.org/nexus/go</a></p></li><li><p><strong>安装nexus</strong></p></li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ tar xzf nexus.x.x.x.tar.gz</span><br><span class="line">$ mv nexus.x.x.x</span><br><span class="line">$ <span class="built_in">cd</span> nexus.x.x.x/bin</span><br></pre></td></tr></table></figure><ul><li><strong>Nexus配置</strong></li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">application-port=端口</span><br><span class="line">application-host=0.0.0.0</span><br><span class="line">nexus-webapp=<span class="variable">${bundleBasedir}</span>/nexus 程序目录</span><br><span class="line">nexus-webapp-context-path=/ 访问路径</span><br><span class="line">nexus-work=数据存放路径</span><br></pre></td></tr></table></figure><ul><li><strong>访问</strong><br><img src="http://p5m69ggo7.bkt.clouddn.com/nuxus_view.jpg" alt="nuxus_view"></li></ul><h3 id="迁移nexus数据"><a href="#迁移nexus数据" class="headerlink" title="迁移nexus数据"></a>迁移nexus数据</h3><ol><li>拷贝原nexus数据目录（sonatype-work）</li><li>还原nexus数据目录到新环境</li><li>启动新环境</li><li>访问新地址</li></ol>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Linux </tag>
</tags>
</entry>
<entry>
<title>Ansible模块编写</title>
<link href="//blog/Custom_Moudles.html"/>
<content type="html"><![CDATA[<p>使用 Ansible 自定义模块功能编写日常使用功能.</p><a id="more"></a><h1 id="Ansible-远程服务器连接-Internet-测试"><a href="#Ansible-远程服务器连接-Internet-测试" class="headerlink" title="Ansible 远程服务器连接 Internet 测试"></a>Ansible 远程服务器连接 Internet 测试</h1><p>Email 连接: <a href="mailto:lonnyliu@126.com" target="_blank" rel="noopener">lonnyliu@126.com</a></p><hr><p><strong>需求</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">在使用Ansible过程中不可避免需要了解远端服务器是否能够联通外网,以便进行在线安装软件或者其他.对于运维人员来说普遍的办法有</span><br><span class="line"></span><br><span class="line">1. 登录到每台服务器进行 ping 测试(基本上这个时候 `www.baidu.com` 派上用场了)</span><br><span class="line">2. 使用Ansible直接对以定义好的主机进行远端执行shell命令</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">以上两种第一种有可能是普遍的方式,第二种有一部分人在使用但也不否定有其他更好的办法</span><br></pre></td></tr></table></figure><p><strong>目标</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">快速获取远端服务器连接 Internet 情况</span><br></pre></td></tr></table></figure><p><strong>实现</strong></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">1. 配置Ansible 关闭或者使用缓存进行 facts 收集</span><br><span class="line">2. 进行自定义模块进行远端服务器连接 Internet 测试</span><br><span class="line">3. 收集结果结果做后续使用</span><br></pre></td></tr></table></figure><p><strong>代码</strong></p><ul><li>ansible 配置文件</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># cat /etc/ansible.cfg</span></span><br><span class="line">[defaults]</span><br><span class="line">hostfile = path <span class="comment"># hosts 文件存放位置</span></span><br><span class="line">roles_path = path <span class="comment"># roles 角色目录存放位置</span></span><br><span class="line">library = path <span class="comment"># 自定义模块存放问题</span></span><br><span class="line">remote_user = root <span class="comment"># 远程连接用户</span></span><br><span class="line">private_key_file = path <span class="comment"># 公钥存放位置</span></span><br><span class="line">host_key_checking = False</span><br><span class="line">deprecation_warnings=False</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 配置使用本地 redis 缓存--自行配置</span></span><br><span class="line">gathering = smart</span><br><span class="line">fact_caching = redis</span><br><span class="line">fact_caching_timeout = 86400</span><br><span class="line">fact_caching_connection = 127.0.0.1:6379:0:ansiblefact</span><br></pre></td></tr></table></figure><ul><li>自定义模块</li></ul><p><a href="http://docs.ansible.com/ansible/latest/dev_guide/developing_modules.html" target="_blank" rel="noopener">具体模块介绍请查看官网链接</a></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/env python</span></span><br><span class="line"><span class="comment"># -*- coding: utf-8 -*- </span></span><br><span class="line"><span class="comment"># Author EdwardLiu</span></span><br><span class="line"></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">检测是否可以连通外网, 使用 url方式, url 为可变参数</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> ansible.module_utils.basic <span class="keyword">import</span> *</span><br><span class="line"><span class="keyword">import</span> urllib2</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">internet_on</span><span class="params">(url)</span>:</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"></span><br><span class="line"> urllib2.urlopen(url, timeout=<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> results = {</span><br><span class="line"> <span class="string">'status'</span>: <span class="number">0</span>,</span><br><span class="line"> <span class="string">'messages'</span>: <span class="string">"Connection"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> results</span><br><span class="line"></span><br><span class="line"> <span class="keyword">except</span> urllib2.URLError <span class="keyword">as</span> err:</span><br><span class="line"></span><br><span class="line"> results = {</span><br><span class="line"> <span class="string">'status'</span>: <span class="number">1</span>,</span><br><span class="line"> <span class="string">'messages'</span>: <span class="string">"Failed"</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> results</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> refactor_module = AnsibleModule(</span><br><span class="line"> argument_spec=dict(</span><br><span class="line"> url=dict(required=<span class="keyword">True</span>),</span><br><span class="line"> ),</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"> url = <span class="string">"http://"</span> + refactor_module.params[<span class="string">'url'</span>]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> internet_on(url)[<span class="string">'status'</span>] == <span class="number">0</span>:</span><br><span class="line"></span><br><span class="line"> result = dict(module=<span class="string">'timezone'</span>, stdotut=internet_on(url)[<span class="string">'messages'</span>], changed=<span class="keyword">False</span>, rc=<span class="number">0</span>)</span><br><span class="line"></span><br><span class="line"> refactor_module.exit_json(**result)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"></span><br><span class="line"> result = dict(msg=<span class="string">'execute failed'</span>, rc=internet_on(url)[<span class="string">'status'</span>])</span><br></pre></td></tr></table></figure><p><strong>使用介绍</strong></p><p> ansible–Ad-Hoc Commands(使用命令行)</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">ansible pipeline -m checknetwork -a <span class="string">"url=www.baidu.com"</span></span><br><span class="line">pipeline: hosts 文件中组名称</span><br><span class="line">checknetwork: 自定义模块名称</span><br><span class="line">url=www.baidu.com 测试外网连接的 URL</span><br></pre></td></tr></table></figure><p>效果:<br><img src="http://p5m69ggo7.bkt.clouddn.com/ansible_custom-moudle.jpg" alt=""></p><p><strong>附ansible playbooks使用案例</strong></p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">---</span><br><span class="line"><span class="comment"># 判断是否有外网</span></span><br><span class="line">- name: checkne internet connection</span><br><span class="line"> checknetwork: url={{ check_url }}</span><br><span class="line"> register: result</span><br><span class="line"></span><br><span class="line">- debug:</span><br><span class="line"> msg: <span class="string">"System {{ inventory_hostname }} has Connection internet!!!"</span></span><br><span class="line"> when: result.stdotut == <span class="string">'Connection'</span></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Ansible </tag>
</tags>
</entry>
<entry>
<title>Git 钩子</title>
<link href="//blog/Git_Hooks.html"/>
<content type="html"><![CDATA[<p>使用 Git Hooks 功能, 进行 Linux 配置文件管理</p><a id="more"></a><h1 id="git-hooks-使用"><a href="#git-hooks-使用" class="headerlink" title="git hooks 使用"></a>git hooks 使用</h1><h2 id="配置管理示意图"><a href="#配置管理示意图" class="headerlink" title="配置管理示意图"></a>配置管理示意图</h2><p><img src="http://p5m69ggo7.bkt.clouddn.com/config_setup.png" alt="配置文件管理示意图"></p><h2 id="git-hooks-使用说明"><a href="#git-hooks-使用说明" class="headerlink" title="git hooks 使用说明"></a>git hooks 使用说明</h2><h3 id="客户端-hooks"><a href="#客户端-hooks" class="headerlink" title="客户端 hooks"></a>客户端 hooks</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">1. pre-commit 钩子在键入提交信息前运行。</span><br><span class="line">2. prepare-commit-msg 钩子在启动提交信息编辑器之前，默认信息被创建之后运行。</span><br><span class="line">3. post-commit 钩子在整个提交过程完成后运行。</span><br><span class="line">4. applypatch-msg 你可以用该脚本来确保提交信息符合格式，或直接用脚本修正格式错误。</span><br><span class="line">5. pre-applypatch 在 git am 运行期间被调用</span><br><span class="line">6. post-applypatch 运行于提交产生之后，是在 git am 运行期间最后被调用的钩子。</span><br><span class="line">7. pre-rebase 钩子运行于变基之前，以非零值退出可以中止变基的过程。</span><br><span class="line">8. post-rewrite 钩子被那些会替换提交记录的命令调用。</span><br><span class="line">9. post-checkout 在 git checkout 成功运行后调用。</span><br><span class="line">10. post-merge 在 git merge 成功运行后调用。</span><br><span class="line">11. pre-push 在 git push 运行期间， 更新了远程引用但尚未传送对象时被调用。</span><br><span class="line">12. pre-auto-gc 会在垃圾回收开始之前被调用，可以用它来提醒你现在要回收垃圾了，或者依情形判断是否要中断</span><br></pre></td></tr></table></figure><h3 id="服务端-Hooks"><a href="#服务端-Hooks" class="headerlink" title="服务端 Hooks"></a>服务端 Hooks</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">1. pre-receive 处理来自客户端的推送操作时最先被调用。 </span><br><span class="line">2. update 它会为每一个准备更新的分支各运行一次。 </span><br><span class="line">3. post-receive 在整个过程完结以后运行，可以用来更新其他系统服务或者通知用户。</span><br></pre></td></tr></table></figure><h3 id="测试验证"><a href="#测试验证" class="headerlink" title="测试验证"></a>测试验证</h3><table><thead><tr><th style="text-align:center">服务器</th><th style="text-align:center">功能</th><th style="text-align:center">目录结构</th></tr></thead><tbody><tr><td style="text-align:center">10.90.11.227</td><td style="text-align:center">系统管理员本地仓库</td><td style="text-align:center">/software/process_config</td></tr><tr><td style="text-align:center">172.31.4.123</td><td style="text-align:center">中转仓库/数据存放仓库</td><td style="text-align:center">中转仓库: /software/workspace/project 数据存放仓库: /software/workspace/project-local/project</td></tr></tbody></table><ul><li><p>初始化中转仓库</p><blockquote><p>git init 和 git init –bare 的区别<br>初始化出来的仓库是不一样的，前者初始化的是一个普通的仓库，其中 .git 文件夹是隐藏的，并且能看见该仓库下所有的源码。而后者初始化出来的仓库中的文件，就是 .git 中的文件夹，但不能像前者那样直接浏览或修改仓库中的代码。</p></blockquote> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir /software/workspace/project</span><br><span class="line">$ <span class="built_in">cd</span> /software/workspace/project</span><br><span class="line">$ git init --bare</span><br><span class="line">**结果:**</span><br><span class="line">-rw-r--r-- 1 root root 23 Apr 25 06:10 HEAD</span><br><span class="line">drwxr-xr-x 2 root root 6 Apr 25 06:10 branches</span><br><span class="line">-rw-r--r-- 1 root root 66 Apr 25 06:10 config</span><br><span class="line">-rw-r--r-- 1 root root 73 Apr 25 06:10 description</span><br><span class="line">drwxr-xr-x 2 root root 4096 Apr 25 10:27 hooks</span><br><span class="line">drwxr-xr-x 2 root root 20 Apr 25 06:10 info</span><br><span class="line">drwxr-xr-x 41 root root 4096 Apr 25 10:26 objects</span><br><span class="line">drwxr-xr-x 4 root root 29 Apr 25 06:10 refs</span><br></pre></td></tr></table></figure></li><li><p>初始化数据存储仓库–其实就是做中转仓的检出</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /software/workspace/project-local</span><br><span class="line">$ git <span class="built_in">clone</span> /software/workspace/project</span><br><span class="line">drwxr-xr-x 6 root root 72 Apr 26 00:08 project</span><br></pre></td></tr></table></figure></li><li><p>配置中转仓– hooks</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> /software/workspace/project/hooks</span><br><span class="line">$ cat post-receive <span class="comment"># 系统管理员每次提交时指定 remote 仓库 后自动触发 hooks 执行以下代码 必须有执行权限</span></span><br><span class="line"><span class="meta">#!/bin/sh</span></span><br><span class="line"><span class="built_in">cd</span> /software/workspace/project-local/project || <span class="built_in">exit</span></span><br><span class="line"><span class="built_in">echo</span> <span class="variable">${GIT_DIR}</span></span><br><span class="line"><span class="built_in">unset</span> GIT_DIR</span><br><span class="line">/usr/bin/git pull</span><br><span class="line"><span class="built_in">echo</span> <span class="string">"pull successful 172.31.4.123"</span></span><br></pre></td></tr></table></figure></li><li><p>系统管理员本地仓库添加remote 源</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 克隆版本库的时候，所使用的远程主机自动被Git命名为origin。如果想用其他的主机名，需要用git clone命令的-o选项指定。</span></span><br><span class="line">$ git <span class="built_in">clone</span> -o <span class="built_in">source</span> http://gitlab.product.co-mall:10080/youname/process_configure.git</span><br><span class="line">$ git remote add deploy [email protected]:/software/workspace/project <span class="comment"># root为系统账户</span></span><br><span class="line">$ git remote -v</span><br><span class="line">[email protected]:/software/workspace/project (fetch)</span><br><span class="line">[email protected]:/software/workspace/project (push)</span><br><span class="line"><span class="built_in">source</span>http://gitlab.product.co-mall:10080/youname/process_configure.git (fetch)</span><br><span class="line"><span class="built_in">source</span>http://gitlab.product.co-mall:10080/youname/process_configure.git (push)</span><br></pre></td></tr></table></figure></li><li><p>系统管理员本地仓库提交测试</p> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">$ vim nginx.conf</span><br><span class="line">$ git add .</span><br><span class="line">$ git commit -m <span class="string">"test hooks"</span></span><br><span class="line">[master 7b05314] <span class="built_in">test</span> hooks</span><br><span class="line"> 1 file changed, 1 insertion(+)</span><br><span class="line">$ git push deploy <span class="comment"># 此处如果想直接全部提交到远程仓库进行如下配置</span></span><br><span class="line"><span class="comment">########</span></span><br><span class="line"> git remote <span class="built_in">set</span>-url --add --push origin http://gitlab.product.co-mall:10080/youname/process_configure.git</span><br><span class="line"> git remote add both [email protected]:/software/workspace/project</span><br><span class="line"> git remote <span class="built_in">set</span>-url --add --push both http://gitlab.product.co-mall:10080/youname/process_configure.git</span><br><span class="line"> git remote <span class="built_in">set</span>-url --add --push both [email protected]:/software/workspace/project</span><br><span class="line">git push both</span><br><span class="line"><span class="comment">########</span></span><br><span class="line">Counting objects: 4, <span class="keyword">done</span>.</span><br><span class="line">Delta compression using up to 4 threads.</span><br><span class="line">Compressing objects: 100% (4/4), <span class="keyword">done</span>.</span><br><span class="line">Writing objects: 100% (4/4), 457 bytes | 0 bytes/s, <span class="keyword">done</span>.</span><br><span class="line">Total 4 (delta 1), reused 0 (delta 0)</span><br><span class="line"><span class="comment"># 以下都是自定义 hooks 脚本返回的信息</span></span><br><span class="line">remote: .</span><br><span class="line">remote: 来自 /software/workspace/project</span><br><span class="line">remote: 7c07381..7b05314 master -> origin/master</span><br><span class="line">remote: 更新 7c07381..7b05314</span><br><span class="line">remote: Fast-forward</span><br><span class="line">remote: nginx/nginx.conf | 1 +</span><br><span class="line">remote: 1 file changed, 1 insertion(+)</span><br><span class="line">remote: pull successful 172.31.4.123</span><br><span class="line">To [email protected]:/software/workspace/project</span><br><span class="line"> 7c07381..7b05314 master -> master</span><br><span class="line">lonny@LonnyLiuMacPro ~/Documents/process_configure/nginx (git)-[master] %</span><br></pre></td></tr></table></figure></li><li><p>配置文件管理完毕</p></li></ul>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Git </tag>
</tags>
</entry>
<entry>
<title>Mfs Install</title>
<link href="//blog/MFS_Setup.html"/>
<content type="html"><![CDATA[<p>详细介绍 MFS(moosefs) 安装与配置</p><a id="more"></a><h1 id="MFS安装"><a href="#MFS安装" class="headerlink" title="MFS安装"></a>MFS安装</h1><h2 id="服务器列表"><a href="#服务器列表" class="headerlink" title="服务器列表"></a>服务器列表</h2><table><thead><tr><th>角色</th><th>IPAdress</th></tr></thead><tbody><tr><td>主控服务器（Master server）</td><td>10.90.0.10</td></tr><tr><td>主控备份服务器（Metalogger server）</td><td>10.90.0.20</td></tr><tr><td>存储块服务器(Chunk server）</td><td>10.90.0.30</td></tr><tr><td>客户端主机（Clients）</td><td>10.90.0.40</td></tr></tbody></table><ul><li>主控服务器</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line">1. 添加用户和组</span><br><span class="line"> [root@mfsmaster software]<span class="comment"># groupadd mfs</span></span><br><span class="line"> [root@mfsmaster software]<span class="comment">#useradd -s /sbin/nologin -g mfs -M mfs</span></span><br><span class="line">2. 下载软件包并解压安装</span><br><span class="line"> [root@mfsmaster software]<span class="comment">#wget http://ppa.moosefs.com/src/moosefs-3.0.88-1.tar.gz</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment"># tar xzf moosefs-3.0.88-1.tar.gz</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment">#cd moosefs-3.0.88-1</span></span><br><span class="line"> 3. 安装master</span><br><span class="line"> [root@mfsmaster packages]<span class="comment">#./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment"># make && make install</span></span><br><span class="line"> 成功安装master后，系统会在/usr/<span class="built_in">local</span>/mfs/etc 目录自动生成样例配置文件，这些样例配置文件是以.dist后缀命名，这里将借用这些样例文件作为MooseFS主控服务器的目标配置文件</span><br><span class="line"> [root@mfsmaster packages]<span class="comment"># cd /usr/local/mfs/etc/mfs</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment">#cp mfsmaster.cfg.sample mfsmaster.cfg</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment"># cp mfsmetalogger.cfg.sample mfsmetalogger.cfg</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment">#cp mfsexports.cfg.sample mfsexports.cfg</span></span><br><span class="line"> [root@mfsmaster packages]<span class="comment">#cp mfstopology.cfg.sample mfstopology.cfg</span></span><br><span class="line"> 在配置文件mfsexports.cfg文件的第一行，先取消注释，然后把星号(*)改成10.10.11.0/8，以便我们可以得到下面的文本行</span><br><span class="line"> 10.90.0.0/16 / rw,alldirs,admin,maproot=0:0</span><br><span class="line">启动master服务</span><br><span class="line">[root@mfsmaster packages]<span class="comment"># /usr/local/mfs/sbin/mfsmaster</span></span><br><span class="line">启动报错及解决</span><br><span class="line">[root@mfsmaster etc]<span class="comment"># /usr/local/mfs/sbin/mfsmaster </span></span><br><span class="line"> open files <span class="built_in">limit</span> has been <span class="built_in">set</span> to: 16384</span><br><span class="line"> working directory: /usr/<span class="built_in">local</span>/mfs/var/mfs</span><br><span class="line"> lockfile created and locked</span><br><span class="line"> initializing mfsmaster modules ...</span><br><span class="line"> exports file has been loaded</span><br><span class="line"> topology file has been loaded</span><br><span class="line"> loading metadata ...</span><br><span class="line"> can<span class="string">'t find metadata.mfs - try using option '</span>-a<span class="string">'</span></span><br><span class="line"><span class="string"> init: metadata manager failed !!!</span></span><br><span class="line"><span class="string"> error occurred during initialization - exiting</span></span><br><span class="line"><span class="string">[root@mfsmaster etc]cd /usr/local/mfs/var/&& mv metadata.mfs.empty metadata.mfs</span></span><br><span class="line"><span class="string">运行CGI监控服务，这样就可以用浏览器查看整个MooseFS的运行情况</span></span><br><span class="line"><span class="string">[root@mfsmaster packages]#/usr/local/mfs/sbin/mfscgiserv</span></span><br><span class="line"><span class="string">在浏览器中输入：http://10.10.0.10:9425 就能看到master的运行情况，效果如下图所示</span></span><br><span class="line"><span class="string">插图等待。。。。。</span></span><br></pre></td></tr></table></figure><ul><li>备份服务器Backup server（Metalogger server）安装</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[root@backup_server ~]<span class="comment"># groupadd mfs && useradd -s /sbin/nologin -g mfs -M mfs</span></span><br><span class="line">[root@backup_server pacakges]<span class="comment"># tar xzf moosefs-3.0.88-1.tar.gz </span></span><br><span class="line">[root@backup_server pacakges]<span class="comment"># cd moosefs-3.0.88</span></span><br><span class="line">[root@backup_server moosefs-3.0.88]<span class="comment"># ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --sysconfdir=/usr/local/mfs --localstatedir=/usr/local/mfs/data --enable-mfsmount </span></span><br><span class="line">[root@backup_server moosefs-3.0.88]<span class="comment"># make && make install</span></span><br><span class="line">[root@backup_server etc]<span class="comment"># cd /usr/local/mfs/etc/mfs</span></span><br><span class="line">[root@backup_server mfs]<span class="comment"># cp mfsmetalogger.cfg.sample mfsmetalogger.cfg</span></span><br><span class="line">[root@backup_server mfs]<span class="comment"># vim /etc/hosts #添加10.90.0.10 mfsmaster</span></span><br><span class="line">[root@backup_server mfs]<span class="comment"># /usr/local/mfs/sbin/mfsmetalogger</span></span><br></pre></td></tr></table></figure><ul><li>存储块服务器Chunk servers 安装</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">[root@chunk_server ~]<span class="comment"># groupadd mfs && useradd -s /sbin/nologin -g mfs -M mfs</span></span><br><span class="line">[root@chunk_server pacakges]<span class="comment"># tar xzf moosefs-3.0.88-1.tar.gz </span></span><br><span class="line">[root@chunk_server pacakges]<span class="comment"># cd moosefs-3.0.88</span></span><br><span class="line">[root@chunk_server moosefs-3.0.88]<span class="comment"># ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --sysconfdir=/usr/local/mfs --localstatedir=/usr/local/mfs/data --enable-mfsmount --disable-mfsmaster </span></span><br><span class="line">[root@chunk_server moosefs-3.0.88]<span class="comment"># make && make install</span></span><br><span class="line">[root@chunk_server mfs]<span class="comment"># vim /etc/hosts #添加10.90.0.10 mfsmaster</span></span><br><span class="line">[root@chunk_server etc]<span class="comment"># cd /usr/local/mfs/etc/mfs</span></span><br><span class="line">[root@chunk_server mfs]<span class="comment"># cp mfschunkserver.cfg.sample mfschunkserver.cfg</span></span><br><span class="line">[root@chunk_server mfs]<span class="comment">#cp mfshdd.cfg.sample mfshdd.cfg</span></span><br><span class="line"> [root@chunk_server mfs]<span class="comment">#vim mfshdd.cfg#添加共享空间</span></span><br><span class="line">在配置文件mfshdd.cfg中，我们给出了用于客户端挂接MooseFS分布式文件系统根分区所使用的共享空间位置。建议在Chunk server上划分单独的空间给MooseFS使用。这样做的好处是便于管理剩余的空间，此处我们假定要使用两个贡献点/mnt/mfschunks1 和/mnt/mfschunks2</span><br><span class="line"> [root@chunk_server mfs]<span class="comment">#mkdir /mnt/{mfschunks1,mfschunks2}</span></span><br><span class="line"> [root@chunk_server mfs]<span class="comment">#chown mfs.mfs /mnt/{mfschunks1,mfschunks2}</span></span><br><span class="line"> 启动 chunk serve</span><br><span class="line"> [root@chunk_server mfs]<span class="comment"># /usr/local/mfs/sbin/mfschunkserver</span></span><br></pre></td></tr></table></figure><ul><li>客户端安装</li></ul><blockquote><p>为了挂接基于MooseFS 分布式文件，客户端主机必须安装FUSE软件包（FUSE版本号至少2.6，推荐版本号大于2.7的fuse）。如果系统没安装fuse，你必须手动对其进行安装。在所以客户端上执行如下命令：</p></blockquote><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#安装FUSE</span></span><br><span class="line">[root@client_001 ~]<span class="comment">#yum -y install fuse fuse-devel</span></span><br><span class="line">[root@client_001 ~]<span class="comment"># groupadd mfs && useradd -s /sbin/nologin -g mfs -M mfs</span></span><br><span class="line">[root@client_001 pacakges]<span class="comment"># tar xzf moosefs-3.0.88-1.tar.gz </span></span><br><span class="line">[root@client_001 pacakges]<span class="comment"># cd moosefs-3.0.88</span></span><br><span class="line">[root@client_001 moosefs-3.0.88]<span class="comment"># ./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --enable-mfsmount --disable-mfsmaster --disable-mfschunkserver</span></span><br><span class="line">[root@client_001 moosefs-3.0.88]<span class="comment"># make && make install</span></span><br><span class="line">[root@client_001 mfs]<span class="comment"># vim /etc/hosts #添加10.90.0.10 mfsmaster</span></span><br><span class="line">测试挂载</span><br><span class="line">[root@client_001 mfs]<span class="comment">#mkdir /software/mfs_test</span></span><br><span class="line">/usr/<span class="built_in">local</span>/mfs/sbin/mount.moosefs -H msmaster /software/mfs_test</span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category> 技术分享 </category>
</categories>
<tags>
<tag> Linux </tag>
</tags>
</entry>
</search>