-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
4222 lines (3690 loc) · 678 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>[Babel] Babel 使用筆記</title>
<url>/posts/22d5a4ab/</url>
<content><![CDATA[<p><a href="https://github.com/babel/babel"><img src="https://raw.githubusercontent.com/babel/logo/master/babel.png" alt="Babel"></a></p>
<h1 id="簡介"><a href="#簡介" class="headerlink" title="簡介"></a>簡介</h1><p><a href="https://babeljs.io/" title="Babel">Babel</a> 是一個JavaScript compiler，可以將新的 JS 語法轉譯為瀏覽器支援的 ES5，因為 JavaScript 幾乎是每年會提出一個新的規格草案，但是瀏覽器沒辦法很快就能夠支援新的 JS，所以我們需要先轉譯成瀏覽器支援的 ES5。</p>
<span id="more"></span>
<h1 id="安裝"><a href="#安裝" class="headerlink" title="安裝"></a>安裝</h1><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># babel-loader 是給webpack設定loader使用。</span></span><br><span class="line"><span class="comment"># babel-core 主要是babel編譯的部分。</span></span><br><span class="line"><span class="comment"># babel-preset-es2015 是設定可以編譯ES6語法，轉譯成ES5普遍瀏覽器可以讀懂的Javascript版本。</span></span><br><span class="line"><span class="comment"># babel-preset-react 編譯react語法。</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># loader</span></span><br><span class="line">$ npm install --save-dev babel-loader</span><br><span class="line"></span><br><span class="line"><span class="comment"># core</span></span><br><span class="line">$ npm install --save-dev babel-core</span><br><span class="line"></span><br><span class="line"><span class="comment"># ES2017轉碼規則</span></span><br><span class="line">$ npm install --save-dev babel-preset-es2017</span><br><span class="line"></span><br><span class="line"><span class="comment"># command line執行babel</span></span><br><span class="line"><span class="comment">#$ npm install --save-dev babel-cli</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># react轉碼規則</span></span><br><span class="line"><span class="comment">#$ npm install --save-dev babel-preset-react</span></span><br></pre></td></tr></table></figure>
<h1 id="設定"><a href="#設定" class="headerlink" title="設定"></a>設定</h1><p>設定的方式有幾種：設定 <code>.babelrc</code>、透過 <code>package.json</code> 設定，或是搭配 <code>Webpack</code> 來設定。</p>
<ul>
<li>設定 <code>.babelrc</code></li>
</ul>
<figure class="highlight js"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="string">"presets"</span>: [<span class="string">"es2017"</span>, <span class="string">"react"</span>]</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<ul>
<li>透過 <code>package.json</code> 設定</li>
</ul>
<figure class="highlight js"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="string">"name"</span>: <span class="string">"app"</span>,</span><br><span class="line"> <span class="string">"version"</span>: <span class="string">"1.0.0"</span>,</span><br><span class="line"> <span class="string">"scripts"</span>: {</span><br><span class="line"> <span class="string">"build"</span>: <span class="string">"babel code -d build"</span></span><br><span class="line"> },</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<ul>
<li>搭配 <code>Webpack</code> 來使用，參考: <a href="/posts/f0b15985/">[Webpack] Webpack 設定及使用</a></li>
</ul>
<h1 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h1><ul>
<li><a href="https://babeljs.io/" title="Babel">Babel</a></li>
</ul>
]]></content>
<categories>
<category>Babel</category>
</categories>
<tags>
<tag>Node.js</tag>
<tag>Babel</tag>
</tags>
</entry>
<entry>
<title>[Cookie] Cross site cookie</title>
<url>/posts/cbecdf44/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>這篇文章主要是記錄一下使用跨域 cookie 需要設定及注意的地方，以下範例程式後端是使用 Python/Flask, 前端則是用 Axios 來送出 request 到後端。</p>
<span id="more"></span>
<h2 id="Cookies"><a href="#Cookies" class="headerlink" title="Cookies"></a>Cookies</h2><p>首先，先來介紹一下 Cookie 幾個比較重要的屬性：</p>
<ul>
<li>HttpOnly: 只能被用在 HTTP 傳輸，不可被 JavaScript 存取</li>
<li>Domain: 可使用的 domain, 如果沒有設定，則只能被設定此 cookie 的 domain 所使用，如果希望設定子網域都可使用此 cookie, 則可以設為 <code>.domain.com</code></li>
<li>Path: 指定可存取 cookie 的路徑</li>
<li>Max-Age: cookie 的期限，單位為秒，或者可以用 <code>Expires</code> 來設定，格式是 Unix time</li>
<li>SameSite: 用來防止瀏覽器將 cookie 跨網站傳送，可以幫助避免 CSRF 攻擊，以下是 <code>SameSite</code> 的三種設定值:<ul>
<li>Strict: 最嚴謹，request 的網域與目前網址相同才會發送 cookie</li>
<li>Lax: 除了 same site request 能夠使用 cookie之外，部分的 cross-site request 也能夠使用 cookie, 包含 HTML 中的 <code><a></code> , <code><link rel="prerender"></code>, <code><form method="GET"></code></li>
<li>None: Cross site 的 request 都能夠使用 cookie, <strong>注意在 Chrome 80 之後，若使用這個選項，則必須要設定 Secure</strong></li>
</ul>
</li>
<li>Secure: 若設為 True, 表示只能透過 HTTPS 傳輸</li>
</ul>
<p><strong>注意</strong>: 在 Chrome 80 之後，如果沒有設定 SameSite, 則預設為 Lax, 如果 SameSite 設為 None, 則必須要有 Secure.</p>
<h2 id="Backend"><a href="#Backend" class="headerlink" title="Backend"></a>Backend</h2><p>後端需要設定 CORS (Cross Origin Resource Sharing), 因為會使用到 Cross site cookie, 所以也記得要設定 <code>Access-Control-Allow-Credentials = true</code> , for example:</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># In app.py</span></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask, jsonify</span><br><span class="line"><span class="keyword">from</span> flask_cors <span class="keyword">import</span> CORS</span><br><span class="line"></span><br><span class="line">app = Flask(__name__)</span><br><span class="line">whitelist = [<span class="string">'http://localhost:8080'</span>, <span class="string">'https://domain1.com'</span>, <span class="string">'https://domain2.com'</span>]</span><br><span class="line">resource = {</span><br><span class="line"> <span class="string">r'/.*'</span>: {</span><br><span class="line"> <span class="string">'origins'</span>: whitelist</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line">CORS(app, resources=resource, supports_credentials=<span class="literal">True</span>, methods=<span class="string">'GET,POST,OPTIONS'</span>, allow_headers=<span class="string">'*'</span>)</span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/login'</span>, methods=[<span class="string">'POST'</span>]</span>)</span></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">login</span>():</span><br><span class="line"> <span class="comment"># User authentication</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Set cookie</span></span><br><span class="line"> response = jsonify(status=<span class="string">'success'</span>)</span><br><span class="line"> response.set_cookie(</span><br><span class="line"> <span class="string">'access_token'</span>, <span class="string">'your_access_token'</span>, </span><br><span class="line"> domain=<span class="string">'.your_domain.com'</span>, </span><br><span class="line"> path=<span class="string">'/'</span>, </span><br><span class="line"> max_age=<span class="number">600</span>,</span><br><span class="line"> secure=<span class="literal">True</span>, </span><br><span class="line"> httponly=<span class="literal">True</span>,</span><br><span class="line"> samesite=<span class="literal">None</span></span><br><span class="line"> )</span><br><span class="line"> response.set_cookie(</span><br><span class="line"> <span class="string">'refresh_token'</span>, <span class="string">'your_refresh_token'</span>, </span><br><span class="line"> domain=<span class="string">'.your_domain.com'</span>, </span><br><span class="line"> path=<span class="string">'/refresh'</span>, </span><br><span class="line"> max_age=<span class="number">86400</span>,</span><br><span class="line"> secure=<span class="literal">True</span>, </span><br><span class="line"> httponly=<span class="literal">True</span>,</span><br><span class="line"> samesite=<span class="literal">None</span></span><br><span class="line"> )</span><br><span class="line"> <span class="keyword">return</span> response</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> app.run()</span><br></pre></td></tr></table></figure>
<p><strong>注意</strong>: 設定 <code>supports_credentials</code> 的話， <code>origins</code> 就不能用 <code>*</code></p>
<h2 id="Frontend"><a href="#Frontend" class="headerlink" title="Frontend"></a>Frontend</h2><p>前端的部分，我們是使用 Axios 來送出 request 到後端，這邊記得要設定 <code>withCredentials: true</code>, for example:</p>
<figure class="highlight jsx"><table><tr><td class="code"><pre><span class="line">axios.<span class="title function_">post</span>(<span class="string">'/login'</span>, data, {</span><br><span class="line"> <span class="attr">withCredentials</span>: <span class="literal">true</span></span><br><span class="line">}).<span class="title function_">then</span>(<span class="function">(<span class="params">response</span>) =></span> {</span><br><span class="line"> <span class="variable language_">console</span>.<span class="title function_">log</span>(response);</span><br><span class="line">});</span><br></pre></td></tr></table></figure>
<p>如果順利的話，會在在網頁的 Cookie 中看到新增了兩個 Cookie (access_token, refresh_token).</p>
<p>但是，在 localhost 測試時會發現無法設定 cookie, 這是因為要跨域使用 cookie, SameSite 就必須設為 None, 而 SameSite 設為 None 就必須要設定 Secure, 但我們目前的 localhost 是沒有設定 SSL 的，所以會無法設定 cookie. 解決方式是把 localhost 加上 SSL.</p>
<p>Localhost 加上 SSL 可參考以下文章:</p>
<ul>
<li>Windows: <a href="https://pvencs.blogspot.com/2019/01/windows-localhost-https.html">讓 Windows 環境的 localhost 啟用 HTTPS 連線</a></li>
<li>MacOS: <a href="https://manglekuo.medium.com/%E8%A8%AD%E5%AE%9Amacos%E6%9C%AC%E5%9C%B0%E7%AB%AFhttps-ssl%E8%AD%89%E6%9B%B8-b2f79bcdedf0">設定macOS本地端HTTPs/SSL證書</a> (MacOS 的部分沒測試過，可能需要大家自行測試)</li>
</ul>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://jcbaey.com/authentication-in-spa-reactjs-and-vuejs-the-right-way?utm_source=medium&utm_campaign=spa-authentication">Authentication in SPA (ReactJS and VueJS) the right way - Part 1</a></li>
<li><a href="https://ianhung0529.medium.com/chrome-80-%E5%BE%8C%E9%87%9D%E5%B0%8D%E7%AC%AC%E4%B8%89%E6%96%B9-cookie-%E7%9A%84%E8%A6%8F%E5%89%87%E8%AA%BF%E6%95%B4-default-samesite-lax-aaba0bc785a3">Chrome 80 後針對第三方 Cookie 的規則調整 (default SameSite=Lax)</a></li>
</ul>
]]></content>
<categories>
<category>Cookie</category>
</categories>
<tags>
<tag>Cookie</tag>
</tags>
</entry>
<entry>
<title>[DevOps] 使用 supervisor 管理 process</title>
<url>/posts/53302290/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>Supervisor 是一個使用 Python 開發的 process 管理工具，主要用在 process 後台維護、即時監控 process 狀態、自動重啟服務…等。</p>
<span id="more"></span>
<h2 id="安裝"><a href="#安裝" class="headerlink" title="安裝"></a>安裝</h2><p>使用 pip 來安裝:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ pip install supervisor</span><br></pre></td></tr></table></figure>
<h2 id="設定"><a href="#設定" class="headerlink" title="設定"></a>設定</h2><p>安裝完成後，可以使用 <code>echo_supervisord_conf</code> 來取得預設的 config file:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ echo_supervisord_conf > supervisord.conf</span><br></pre></td></tr></table></figure>
<p>以下是預設的 config file 中的主要設定內容:</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[unix_http_server]</span><br><span class="line">file=/tmp/supervisor.sock ; the path to the socket file</span><br><span class="line">;chmod=0700 ; socket file mode (default 0700)</span><br><span class="line">;chown=nobody:nogroup ; socket file uid:gid owner</span><br><span class="line">;username=user ; default is no username (open server)</span><br><span class="line">;password=123 ; default is no password (open server)</span><br><span class="line"></span><br><span class="line">[inet_http_server] ; Web 管理介面</span><br><span class="line">port=127.0.0.1:9001 ; Web 管理介面的 IP 和 port, 如果設為 public 須注意安全性問題</span><br><span class="line">username=user ; 登入管理介面的 user</span><br><span class="line">password=123 ; 登入管理介面的密碼</span><br><span class="line"></span><br><span class="line">[supervisord]</span><br><span class="line">logfile=/tmp/supervisord.log ; 日誌文件，默認是 $CWD/supervisord.log</span><br><span class="line">logfile_maxbytes=50MB ; 日誌文件大小，超出會rotate，默認50MB</span><br><span class="line">logfile_backups=10 ; 日誌文件保留備份數量, 默認10</span><br><span class="line">loglevel=info ; 日誌級別，默認info，其它: debug,warn,trace</span><br><span class="line">pidfile=/tmp/supervisord.pid ; pid 文件</span><br><span class="line">nodaemon= false ; 是否在前台啟動，默認是false，即以daemon的方式啟動</span><br><span class="line">silent=false ; no logs to stdout if true; 預設是 false</span><br><span class="line">minfds=1024 ; 可以打開的文件描述符的最小值，默認1024</span><br><span class="line">minprocs=200 ; 可以打開的進程數的最小值，默認200</span><br><span class="line"></span><br><span class="line">; The rpcinterface:supervisor section must remain in the config file for</span><br><span class="line">; RPC (supervisorctl/web interface) to work. Additional interfaces may be</span><br><span class="line">; added by defining them in separate [rpcinterface:x] sections.</span><br><span class="line"></span><br><span class="line">[rpcinterface:supervisor]</span><br><span class="line">supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface</span><br><span class="line"></span><br><span class="line">; The supervisorctl section configures how supervisorctl will connect to</span><br><span class="line">; supervisord. configure it match the settings in either the unix_http_server</span><br><span class="line">; or inet_http_server section.</span><br><span class="line"></span><br><span class="line">[supervisorctl]</span><br><span class="line">serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket, 透過 unix socket 連接 supervisord, 路徑和 unix_http_server 的 file 一致</span><br><span class="line">serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket, 透過 HTTP 的方式連接 supervisord</span><br><span class="line">username=admin ; should be same as in [*_http_server] if set</span><br><span class="line">password=admin2020@dwave ; should be same as in [*_http_server] if set</span><br><span class="line">;prompt=mysupervisor ; cmd line prompt (default "supervisor")</span><br><span class="line">;history_file=~/.sc_history ; use readline history if available</span><br><span class="line"></span><br><span class="line">; The [include] section can just contain the "files" setting. This</span><br><span class="line">; setting can list multiple files (separated by whitespace or</span><br><span class="line">; newlines). It can also contain wildcards. The filenames are</span><br><span class="line">; interpreted as relative to this file. Included files *cannot*</span><br><span class="line">; include files themselves.</span><br><span class="line"></span><br><span class="line">[include]</span><br><span class="line">files = /etc/supervisord/service-enabled/*.conf ; 可以是 *.conf 或 *.ini</span><br></pre></td></tr></table></figure>
<p>如果有很多個服務，可以將各個服務的 config 寫在不同檔案，最後在 supervisord.conf 中 include 即可，以下範例為 program config file:</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[program:program_name] ; program_name 須為唯一值, 不可重複</span><br><span class="line">directory=<project_dir> ; directory to cwd to before exec (def no cwd)</span><br><span class="line">environment=PATH="/home/<user>/anaconda3/envs/api/bin" ; process environment additions (def no adds)</span><br><span class="line">command=/home/<user>/anaconda3/envs/api/bin/uwsgi --ini <project_dir>/conf/uwsgi.ini ; the program (relative uses PATH, can take args)</span><br><span class="line">autostart=true ; start at supervisord start (default: true)</span><br><span class="line">autorestart=true ; when to restart if exited after running (def: unexpected)</span><br><span class="line">stdout_logfile=/stdout/logfile/path ; stdout log path, NONE for none; default AUTO</span><br><span class="line">stderr_logfile=/stderr/logfile/path ; stderr log path, NONE for none; default AUTO</span><br><span class="line"></span><br><span class="line">;process_name=%(program_name)s ; process_name expr (default %(program_name)s)</span><br><span class="line">;numprocs=1 ; number of processes copies to start (def 1)</span><br><span class="line">;umask=022 ; umask for process (default None)</span><br><span class="line">;priority=999 ; the relative start priority (default 999)</span><br><span class="line">;startsecs=1 ; # of secs prog must stay up to be running (def. 1)</span><br><span class="line">;startretries=3 ; max # of serial start failures when starting (default 3)</span><br><span class="line">;exitcodes=0 ; 'expected' exit codes used with autorestart (default 0)</span><br><span class="line">;stopsignal=QUIT ; signal used to kill process (default TERM)</span><br><span class="line">;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)</span><br><span class="line">;stopasgroup=false ; send stop signal to the UNIX process group (default false)</span><br><span class="line">;killasgroup=false ; SIGKILL the UNIX process group (def false)</span><br><span class="line">;user=chrism ; setuid to this UNIX account to run the program</span><br><span class="line">;redirect_stderr=true ; redirect proc stderr to stdout (default false)</span><br><span class="line">;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)</span><br><span class="line">;stdout_logfile_backups=10 ; # of stdout logfile backups (0 means none, default 10)</span><br><span class="line">;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)</span><br><span class="line">;stdout_events_enabled=false ; emit events on stdout writes (default false)</span><br><span class="line">;stdout_syslog=false ; send stdout to syslog with process name (default false)</span><br><span class="line">;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)</span><br><span class="line">;stderr_logfile_backups=10 ; # of stderr logfile backups (0 means none, default 10)</span><br><span class="line">;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)</span><br><span class="line">;stderr_events_enabled=false ; emit events on stderr writes (default false)</span><br><span class="line">;stderr_syslog=false ; send stderr to syslog with process name (default false)</span><br><span class="line">;serverurl=AUTO ; override serverurl computation (childutils)</span><br></pre></td></tr></table></figure>
<h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><h3 id="啟動-supervisord"><a href="#啟動-supervisord" class="headerlink" title="啟動 supervisord"></a>啟動 supervisord</h3><p>撰寫好 config file 之後，使用以下指令來啟動 supervisord:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -c: config file</span></span><br><span class="line">$ supervisord -c supervisord.conf</span><br></pre></td></tr></table></figure>
<p>啟動之後可以搭配 supervisorctl 監控狀態, 或是可以連線至 <a href="http://127.0.0.1:9001/">http://127.0.0.1:9001</a>, 登入管理者帳號密碼之後，就可以看到目前各個服務的狀態。</p>
<h3 id="supervisorctl-指令"><a href="#supervisorctl-指令" class="headerlink" title="supervisorctl 指令"></a>supervisorctl 指令</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -c: config file</span></span><br><span class="line">$ supervisorctl -c supervisord.conf</span><br><span class="line"></span><br><span class="line"><span class="comment"># 檢查 process 狀態</span></span><br><span class="line">$ supervisorctl status</span><br><span class="line"></span><br><span class="line"><span class="comment"># 啟動 process</span></span><br><span class="line">$ supervisorctl start <process_name></span><br><span class="line"></span><br><span class="line"><span class="comment"># 關閉 process</span></span><br><span class="line">$ supervisorctl stop <process_name></span><br><span class="line"></span><br><span class="line"><span class="comment"># Restart process</span></span><br><span class="line">$ supervisorctl restart <process_name></span><br><span class="line"></span><br><span class="line"><span class="comment"># 重新載入配置檔案</span></span><br><span class="line">$ upervisordctl update</span><br><span class="line"></span><br><span class="line"><span class="comment"># 關閉 supervisord</span></span><br><span class="line">$ supervisorctl shutdown</span><br><span class="line"></span><br><span class="line"><span class="comment"># 清空 log file</span></span><br><span class="line">$ supervisorctl clear <process_name></span><br><span class="line"></span><br><span class="line"><span class="comment"># 進入互動模式, 可使用 help 查看所有指令</span></span><br><span class="line">$ supervisorctl</span><br></pre></td></tr></table></figure>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="http://supervisord.org/index.html">Supervisor 官方文件</a></li>
<li><a href="http://liyangliang.me/posts/2015/06/using-supervisor/">使用supervisor 管理進程</a></li>
</ul>
]]></content>
<categories>
<category>DevOps</category>
</categories>
<tags>
<tag>supervisor</tag>
<tag>DevOps</tag>
</tags>
</entry>
<entry>
<title>[Docker] Health Check and Restart Unhealthy Container</title>
<url>/posts/b8226bad/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>在沒有 <code>HEALTHCHECK</code> 指令之前，Docker 只能透過 process 是否退出來判斷 container 的狀態，不過有時候是服務已經無法正常運作了，但 process 沒有退出，這樣會導致該服務仍然可以接收用戶請求，但是無法正常回應。</p>
<span id="more"></span>
<h2 id="Health-Check"><a href="#Health-Check" class="headerlink" title="Health Check"></a>Health Check</h2><p>在 Docker 版本 1.12 之後提供了 <code>HEALTHCHECK</code> 指令，可以設定一行 command 用來判斷服務的狀態是否正常，這樣可以更準確地判斷服務狀態。</p>
<p>Container 啟動後的初始狀態為 <code>starting</code>, 在 <code>HEALTHCHECK</code> 指令檢查成功後，狀態會更改為 <code>healthy</code>，如果連續失敗超過指定次數則會改為 <code>unhealthy</code>.</p>
<p><code>HEALTHCHECK</code> options:</p>
<ul>
<li><code>--interval</code>: Health check 時間間隔，預設為 30 秒</li>
<li><code>--timeout</code>: 當 Health check 超過此設定的時間，則會視為失敗，預設為 30 秒</li>
<li><code>--retries</code>: 當 Health check 連續失敗次數超過此設定時，則會將狀態更改為 <code>unhealthy</code>，預設為 3 次</li>
<li><code>--start-period</code>: 啟動時間，預設為 0 秒</li>
</ul>
<p><code>HEALTHCHECK</code> 可以透過 Dockerfile 或是 docker-compose file 做設定:</p>
<h3 id="Dockerfile-example"><a href="#Dockerfile-example" class="headerlink" title="Dockerfile example"></a>Dockerfile example</h3><p>在 Dockerfile 中，<code>HEALTHCHECK</code> 指令格式為 <code>HEALTHCHECK [options] CMD <command></code>, <code><command></code> 可以是 shell 指令或是 exec 格式 (和其他 Dockerfile 指令相同，可以參考 <code>ENTRYPOINT</code>)。而一個 Dockerfile 中只能有一個 <code>HEALTHCHECK</code> 指令，如果同時有多個 <code>HEALTHCHECK</code> 指令，則只有最後一個有效。</p>
<p><code><command></code> 的返回值代表 container 的狀態:</p>
<ul>
<li>0: 成功，container is healthy</li>
<li>1: 失敗，如果失敗超過指定次數，則 container 為 unhealthy</li>
<li>2: reserved, 不要使用這個值</li>
</ul>
<p>假設我們的 container 服務是 web 服務，我們可以使用 <code>curl</code> 來檢查服務是否正常運作，例如: 每 30 秒檢查一次 <code>http://localhost:3000</code> 是否可在 5 秒內回應請求:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="comment"># ...</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">HEALTHCHECK</span><span class="language-bash"> --interval=30s --<span class="built_in">timeout</span>=5s --retries=5 --start_period=30s \</span></span><br><span class="line"><span class="language-bash"> CMD curl -fs http://localhost:3000/ || <span class="built_in">exit</span> 1</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># ...</span></span><br></pre></td></tr></table></figure>
<h3 id="Docker-compose-example"><a href="#Docker-compose-example" class="headerlink" title="Docker-compose example"></a>Docker-compose example</h3><p>在 <code>docker-compose.yml</code> 中，<code>healthcheck</code> 範例如下:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">"3.7"</span></span><br><span class="line"></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"> <span class="attr">api:</span></span><br><span class="line"> <span class="attr">restart:</span> <span class="string">always</span></span><br><span class="line"> <span class="attr">image:</span> <span class="string">api</span></span><br><span class="line"> <span class="attr">container_name:</span> <span class="string">api</span></span><br><span class="line"> <span class="attr">ports:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="number">3000</span><span class="string">:3000</span></span><br><span class="line"> <span class="attr">build:</span></span><br><span class="line"> <span class="attr">context:</span> <span class="string">./api</span></span><br><span class="line"> <span class="attr">healthcheck:</span></span><br><span class="line"> <span class="attr">test:</span> <span class="string">curl</span> <span class="string">-fs</span> <span class="string">http://localhost:3000/</span> <span class="string">||</span> <span class="string">exit</span> <span class="number">1</span></span><br><span class="line"> <span class="attr">interval:</span> <span class="string">30s</span></span><br><span class="line"> <span class="attr">timeout:</span> <span class="string">5s</span></span><br><span class="line"> <span class="attr">retries:</span> <span class="number">5</span></span><br><span class="line"> <span class="attr">start_period:</span> <span class="string">30s</span></span><br><span class="line"> <span class="attr">networks:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">net</span></span><br><span class="line"><span class="attr">networks:</span></span><br><span class="line"> <span class="attr">net:</span></span><br><span class="line"> <span class="attr">name:</span> <span class="string">net</span></span><br><span class="line"> <span class="attr">driver:</span> <span class="string">bridge</span></span><br></pre></td></tr></table></figure>
<p>其中 <code>test</code> 必須是 string 或 list. 如果是 list, 第一個 item 必須是 <code>NONE</code>, <code>CMD</code> 或 <code>CMD-SHELL</code>。如果是 string, 則等同於 <code>CMD-SHELL</code>。</p>
<h3 id="確認健康狀態"><a href="#確認健康狀態" class="headerlink" title="確認健康狀態"></a>確認健康狀態</h3><p>在設定好 health check 指令之後，接著啟動 container，檢查 container 狀態時可以看到初始狀態是 <code>health: starting</code>:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker ps</span><br><span class="line">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</span><br><span class="line">3c7b9ca321d2 api:1.0.0 <span class="string">"uwsgi --ini /home/d…"</span> 5 seconds ago Up 2 seconds (health: starting) 0.0.0.0:3000->3000/tcp api</span><br></pre></td></tr></table></figure>
<p>過 30 秒之後再執行一次 <code>docker ps</code> 可以看到 container 的狀態變成 <code>healthy</code>:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker ps</span><br><span class="line">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</span><br><span class="line">3c7b9ca321d2 api:1.0.0 <span class="string">"uwsgi --ini /home/d…"</span> 35 seconds ago Up 32 seconds (healthy) 0.0.0.0:3000->3000/tcp api</span><br></pre></td></tr></table></figure>
<p>而如果連續失敗超過指定次數，狀態會變成 <code>unhealthy</code>。</p>
<p>在 <code>HEALTHCHECK</code> command 的任何 output (包含 <code>stdout</code> 和 <code>stderr</code>) 都會被儲存在健康狀態中，可以使用 <code>docker inspect</code> 來查看:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker inspect --format <span class="string">'{{json .State.Health}}'</span> api</span><br><span class="line">{</span><br><span class="line"> <span class="string">"Status"</span>:<span class="string">"healthy"</span>,</span><br><span class="line"> <span class="string">"FailingStreak"</span>:0,</span><br><span class="line"> <span class="string">"Log"</span>:[{</span><br><span class="line"> <span class="string">"Start"</span>:<span class="string">"2021-04-25T15:08:05.579571483+08:00"</span>,</span><br><span class="line"> <span class="string">"End"</span>:<span class="string">"2021-04-25T15:08:05.871891851+08:00"</span>,</span><br><span class="line"> <span class="string">"ExitCode"</span>:0,</span><br><span class="line"> <span class="string">"Output"</span>:<span class="string">"store HEALTHCHECK output here"</span></span><br><span class="line"> }]</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h2 id="Restart-Unhealthy-Container"><a href="#Restart-Unhealthy-Container" class="headerlink" title="Restart Unhealthy Container"></a>Restart Unhealthy Container</h2><p>以上的步驟只有檢查 container 的健康狀態，但沒有針對 unhealthy container 做任何處理，這部分我們可以搭配 <a href="https://github.com/willfarrell/docker-autoheal" title="docker-autoheal">docker-autoheal</a> 來重啟 unhealthy container.</p>
<p>這部分可以直接使用 docker 執行，或是寫在 docker-compose file 中:</p>
<ul>
<li><p>使用 docker 指令:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker run -d \</span><br><span class="line"> --name autoheal \</span><br><span class="line"> --restart=always \</span><br><span class="line"> -e AUTOHEAL_CONTAINER_LABEL=all \</span><br><span class="line"> -v /var/run/docker.sock:/var/run/docker.sock \</span><br><span class="line"> willfarrell/autoheal</span><br></pre></td></tr></table></figure>
</li>
<li><p>透過 docker-compose file 設定:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">"3.7"</span></span><br><span class="line"></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"> <span class="attr">autoheal:</span></span><br><span class="line"> <span class="attr">restart:</span> <span class="string">always</span></span><br><span class="line"> <span class="attr">image:</span> <span class="string">willfarrell/autoheal</span></span><br><span class="line"> <span class="attr">container_name:</span> <span class="string">autoheal</span></span><br><span class="line"> <span class="attr">environment:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">AUTOHEAL_CONTAINER_LABEL=all</span></span><br><span class="line"> <span class="attr">volumes:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/var/run/docker.sock:/var/run/docker.sock</span></span><br></pre></td></tr></table></figure>
<p>接著執行 <code>docker-compose up --build -d autoheal</code> 即可。</p>
</li>
</ul>
<p>最後就可以確認一下 unhealthy container 是否有自動重啟~</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://docs.docker.com/engine/reference/builder/#healthcheck">Docker HEALTHCHECK</a></li>
<li><a href="https://yeasy.gitbook.io/docker_practice/image/dockerfile/healthcheck">Docker —— 从入门到实践</a></li>
<li><a href="https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck">Docker-compose healthcheck</a></li>
<li><a href="https://github.com/willfarrell/docker-autoheal" title="docker-autoheal">docker-autoheal</a></li>
</ul>
]]></content>
<categories>
<category>Docker</category>
</categories>
<tags>
<tag>Docker</tag>
<tag>Docker-compose</tag>
</tags>
</entry>
<entry>
<title>[Docker] Use non-root user in docker container</title>
<url>/posts/fe688d83/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>在 docker 中，預設是使用 root user, 但這樣非常不安全，所以通常會在 docker 中另外建立 user，可以透過 <code>--build-args</code> 傳入當前使用者的 User ID 和 Group ID，將新增的 user 的 User ID 和 Group ID 設為當前使用者，最後使用 <code>USER new_user</code> 切換為 non-root user 再執行所需指令。</p>
<span id="more"></span>
<h2 id="Dockerfile-example"><a href="#Dockerfile-example" class="headerlink" title="Dockerfile example"></a>Dockerfile example</h2><figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">ARG</span> PYTHON_VERSION</span><br><span class="line"></span><br><span class="line"><span class="keyword">FROM</span> python:${PYTHON_VERSION}-slim-buster</span><br><span class="line"></span><br><span class="line"><span class="keyword">ARG</span> USER_ID</span><br><span class="line"><span class="keyword">ARG</span> GID</span><br><span class="line"><span class="keyword">ARG</span> PROJECT_DIR=/home/docker/api</span><br><span class="line"></span><br><span class="line"><span class="comment"># Set environment variable</span></span><br><span class="line"><span class="keyword">ENV</span> TZ=Asia/Taipei</span><br><span class="line"><span class="keyword">ENV</span> PATH=<span class="string">"/home/docker:/home/docker/.local/bin:${PATH}"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Create user</span></span><br><span class="line"><span class="keyword">RUN</span><span class="language-bash"> groupadd -g <span class="variable">$GID</span> docker-users && \</span></span><br><span class="line"><span class="language-bash"> useradd -m --no-log-init -s /bin/bash -u <span class="variable">$USER_ID</span> -g <span class="variable">$GID</span> docker && \</span></span><br><span class="line"><span class="language-bash"> <span class="built_in">echo</span> <span class="string">"docker:docker"</span> | chpasswd && \</span></span><br><span class="line"><span class="language-bash"> adduser docker sudo</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Set timezone and install packages </span></span><br><span class="line"><span class="keyword">RUN</span><span class="language-bash"> <span class="built_in">ln</span> -snf /usr/share/zoneinfo/<span class="variable">$TZ</span> /etc/localtime && <span class="built_in">echo</span> <span class="variable">$TZ</span> > /etc/timezone && \</span></span><br><span class="line"><span class="language-bash"> apt-get update && \</span></span><br><span class="line"><span class="language-bash"> apt-get install -y --no-install-recommends build-essential sudo curl locales tzdata && \</span></span><br><span class="line"><span class="language-bash"> apt-get clean && \</span></span><br><span class="line"><span class="language-bash"> apt-get autoremove</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Set locale</span></span><br><span class="line"><span class="keyword">ENV</span> LC_ALL=C.UTF-<span class="number">8</span></span><br><span class="line"><span class="keyword">ENV</span> LANG=C.UTF-<span class="number">8</span></span><br><span class="line"><span class="keyword">ENV</span> LANGUAGE=C.UTF-<span class="number">8</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">WORKDIR</span><span class="language-bash"> <span class="variable">$PROJECT_DIR</span></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Copy code, install required packages, create log directory and change permission</span></span><br><span class="line"><span class="keyword">COPY</span><span class="language-bash"> . .</span></span><br><span class="line"><span class="keyword">RUN</span><span class="language-bash"> pip install -r ./requirements.txt && \</span></span><br><span class="line"><span class="language-bash"> <span class="built_in">mkdir</span> -p ./logs && \</span></span><br><span class="line"><span class="language-bash"> <span class="built_in">chown</span> -R <span class="variable">$USER_ID</span>:<span class="variable">$GID</span> .</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">USER</span> docker</span><br><span class="line"><span class="keyword">CMD</span><span class="language-bash"> [<span class="string">"uwsgi"</span>, <span class="string">"--ini"</span>, <span class="string">"/home/docker/api/config/uwsgi.ini"</span>]</span></span><br></pre></td></tr></table></figure>
<h2 id="Build-image-and-run-container"><a href="#Build-image-and-run-container" class="headerlink" title="Build image and run container"></a>Build image and run container</h2><h3 id="Use-docker-command"><a href="#Use-docker-command" class="headerlink" title="Use docker command"></a>Use <code>docker</code> command</h3><p>Build image: </p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker build -t <image_name>:<version> \</span><br><span class="line"> --build-arg USER_ID=$(<span class="built_in">id</span> -u) \</span><br><span class="line"> --build-arg GID=$(<span class="built_in">id</span> -g) \</span><br><span class="line"> --build-arg PYTHON_VERSION=<span class="string">"3.8.7"</span> .</span><br></pre></td></tr></table></figure>
<p>Run container: </p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker run -d \</span><br><span class="line"> [--name container_name] \</span><br><span class="line"> [-v host_path:container_path] \</span><br><span class="line"> <image_name>:<version></span><br></pre></td></tr></table></figure>
<p>ps. 須注意 volumes 的資料夾權限 </p>
<h3 id="User-docker-compose"><a href="#User-docker-compose" class="headerlink" title="User docker-compose"></a>User <code>docker-compose</code></h3><p><code>docker-compose.yml</code> example:</p>
<figure class="highlight yaml"><table><tr><td class="code"><pre><span class="line"><span class="attr">version:</span> <span class="string">"3.7"</span></span><br><span class="line"></span><br><span class="line"><span class="attr">services:</span></span><br><span class="line"> <span class="attr">api:</span></span><br><span class="line"> <span class="attr">restart:</span> <span class="string">always</span></span><br><span class="line"> <span class="attr">image:</span> <span class="string">api:${VERSION}</span></span><br><span class="line"> <span class="attr">container_name:</span> <span class="string">api</span></span><br><span class="line"> <span class="attr">user:</span> <span class="string">"${USER_ID}:${GID}"</span></span><br><span class="line"> <span class="attr">ports:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="number">9999</span><span class="string">:9999</span></span><br><span class="line"> <span class="attr">build:</span></span><br><span class="line"> <span class="attr">context:</span> <span class="string">/home/user/workspace/api</span></span><br><span class="line"> <span class="attr">args:</span></span><br><span class="line"> <span class="attr">USER_ID:</span> <span class="string">"${USER_ID}"</span></span><br><span class="line"> <span class="attr">GID:</span> <span class="string">"${GID}"</span></span><br><span class="line"> <span class="attr">PYTHON_VERSION:</span> <span class="string">"3.8.7"</span></span><br><span class="line"> <span class="attr">volumes:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/home/user/workspace/api/config:/home/docker/api/config</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">/home/user/logs/api:/home/docker/api/logs</span></span><br><span class="line"> <span class="attr">networks:</span></span><br><span class="line"> <span class="bullet">-</span> <span class="string">net</span></span><br><span class="line"><span class="attr">networks:</span></span><br><span class="line"> <span class="attr">net:</span></span><br><span class="line"> <span class="attr">name:</span> <span class="string">net</span></span><br><span class="line"> <span class="attr">driver:</span> <span class="string">bridge</span></span><br></pre></td></tr></table></figure>
<p>Build image and run container:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ USER_ID=$(<span class="built_in">id</span> -u) GID=$(<span class="built_in">id</span> -g) VERSION=<span class="string">"1.0.0"</span> docker-compose up --build -d api</span><br></pre></td></tr></table></figure>
<p>ps. 須注意 volumes 的資料夾權限</p>
<h2 id="References"><a href="#References" class="headerlink" title="References"></a>References</h2><ul>
<li><a href="https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b">Processes In Containers Should Not Run As Root</a></li>
</ul>
]]></content>
<categories>
<category>Docker</category>
</categories>
<tags>
<tag>Docker</tag>
</tags>
</entry>
<entry>
<title>[Docker] 安裝及使用 Docker</title>
<url>/posts/ba763ef1/</url>
<content><![CDATA[<p><a href="https://www.docker.com/sites/default/files/social/docker_facebook_share.png"><img src="https://www.docker.com/sites/default/files/social/docker_facebook_share.png" alt="Docker"></a></p>
<h2 id="簡介"><a href="#簡介" class="headerlink" title="簡介"></a>簡介</h2><p><a href="https://www.docker.com/" title="Docker">Docker</a> 是一種虛擬化技術，透過 container 的方式將應用程式和所需執行環境打包起來，方便佈署到其他伺服器上，避免因為系統環境不同而無法正確執行，也可以節省建立環境的繁瑣步驟。此外，Docker 也可以用來模擬不同環境下，程式是否能正確執行。</p>
<p>和 Virtual Machine 相比，Docker 的利用率更高，容量小、高效能、啟動速度快，一台伺服器上(Host)可以執行多個 container, 而每個 container 是獨立的，互不影響。</p>
<p>接下來紀錄一下如何在 Linux 環境上安裝及使用 Docker，並且使用 Docker 將我們的應用程式及執行環境打包。</p>
<span id="more"></span>
<h2 id="安裝"><a href="#安裝" class="headerlink" title="安裝"></a>安裝</h2><p>這裡我們使用的系統環境是: Ubuntu 16.04</p>
<h3 id="設定-Repository"><a href="#設定-Repository" class="headerlink" title="設定 Repository"></a>設定 Repository</h3><p>Update:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get update</span><br></pre></td></tr></table></figure>
<p>安裝所需套件:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common</span><br></pre></td></tr></table></figure>
<p>新增 Docker GPG key:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</span><br></pre></td></tr></table></figure>
<p>Add repository:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo add-apt-repository \</span><br><span class="line"> <span class="string">"deb [arch=amd64] https://download.docker.com/linux/ubuntu \</span></span><br><span class="line"><span class="string"> <span class="subst">$(lsb_release -cs)</span> \</span></span><br><span class="line"><span class="string"> stable"</span></span><br></pre></td></tr></table></figure>
<h3 id="Install-Docker-CE"><a href="#Install-Docker-CE" class="headerlink" title="Install Docker CE"></a>Install Docker CE</h3><p>Update:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get update</span><br></pre></td></tr></table></figure>
<p>Install:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get install docker-ce</span><br></pre></td></tr></table></figure>
<p>裝好之後，確認一下是否安裝成功:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">docker version</span><br></pre></td></tr></table></figure>
<p>正常運作的話，可以看到以下畫面:</p>
<p><img src="https://i.imgur.com/2PnQTFu.png" alt="Docker version"></p>
<p>如果有出現 <code>Permission denied</code> 的錯誤訊息，將當前使用者加入 <code>docker</code> 群組即可解決:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo gpasswd -a <span class="variable">${USER}</span> docker</span><br></pre></td></tr></table></figure>
<p>設定好之後，記得要重新登入才會生效。</p>
<h2 id="基本概念"><a href="#基本概念" class="headerlink" title="基本概念"></a>基本概念</h2><h3 id="Image"><a href="#Image" class="headerlink" title="Image"></a>Image</h3><p>映像檔，可以透過 Docker 將應用程式和執行環境打包成 image, 方便之後直接安裝到其他伺服器上。</p>
<h3 id="Container"><a href="#Container" class="headerlink" title="Container"></a>Container</h3><p>Container 是利用 image 建立出來的，一個 Image 可以 create 多個 container, 每個 container 是獨立的，互不影響。</p>
<h2 id="基本使用"><a href="#基本使用" class="headerlink" title="基本使用"></a>基本使用</h2><h3 id="搜尋及下載-Image"><a href="#搜尋及下載-Image" class="headerlink" title="搜尋及下載 Image"></a>搜尋及下載 Image</h3><p>在 <a href="https://hub.docker.com/" title="Docker Hub">Docker Hub</a> 上有許多公開的 image, 可以使用以下指令來搜尋所需要的 image:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker search ubuntu</span><br></pre></td></tr></table></figure>
<p><img src="https://i.imgur.com/b7Yqlhk.png" alt="search"></p>
<p>下載 image:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker pull ubuntu</span><br></pre></td></tr></table></figure>
<p><img src="https://i.imgur.com/v78BxLO.png" alt="pull"></p>
<h3 id="查看所有-Image"><a href="#查看所有-Image" class="headerlink" title="查看所有 Image"></a>查看所有 Image</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker images</span><br></pre></td></tr></table></figure>
<h3 id="刪除-Image"><a href="#刪除-Image" class="headerlink" title="刪除 Image"></a>刪除 Image</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker rmi [options] <image></span><br></pre></td></tr></table></figure>
<h3 id="建立並啟動-Container"><a href="#建立並啟動-Container" class="headerlink" title="建立並啟動 Container"></a>建立並啟動 Container</h3><p>下載好 image 之後，執行以下指令就可以建立並啟動 container:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker run -it --name <container_name> <image> bash</span><br></pre></td></tr></table></figure>
<p><code>-it</code> 是可以進入這個 container 的 shell 進行操作，如果要離開 container, 使用 <code>exit</code> 指令即可。</p>
<p><code>--name</code> 設定 container 名稱。</p>
<h3 id="查看-Container"><a href="#查看-Container" class="headerlink" title="查看 Container"></a>查看 Container</h3><p>查看目前運行的 container:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker ps</span><br></pre></td></tr></table></figure>
<p>查看所有 container</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker ps -a</span><br></pre></td></tr></table></figure>
<h3 id="進入-Container"><a href="#進入-Container" class="headerlink" title="進入 Container"></a>進入 Container</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker <span class="built_in">exec</span> -it <container_id> bash</span><br></pre></td></tr></table></figure>
<h3 id="停止-Container"><a href="#停止-Container" class="headerlink" title="停止 Container"></a>停止 Container</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker stop [options] <container_id></span><br></pre></td></tr></table></figure>
<h3 id="重新啟動-Container"><a href="#重新啟動-Container" class="headerlink" title="重新啟動 Container"></a>重新啟動 Container</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker restart [options] <container_id></span><br></pre></td></tr></table></figure>
<h3 id="刪除-Container"><a href="#刪除-Container" class="headerlink" title="刪除 Container"></a>刪除 Container</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker <span class="built_in">rm</span> [options] <container_id></span><br></pre></td></tr></table></figure>
<h2 id="Dockerize-應用程式"><a href="#Dockerize-應用程式" class="headerlink" title="Dockerize 應用程式"></a>Dockerize 應用程式</h2><p>了解基本概念以及簡單的操作指令之後，我們就可以將應用程式和執行環境打包，產生一個 image, 方便之後佈署到其他機器上。</p>
<h3 id="建立並撰寫-Dockerfile"><a href="#建立並撰寫-Dockerfile" class="headerlink" title="建立並撰寫 Dockerfile"></a>建立並撰寫 Dockerfile</h3><p>Dockerfile 是用來記錄打包的步驟，我們先在專案目錄底下建立 <code>Dockerfile</code>，專案目錄架構如下:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">application/</span><br><span class="line">├── Dockerfile</span><br><span class="line">├── README.md</span><br><span class="line">├── app.js</span><br><span class="line">├── node_modules/</span><br><span class="line">└── package.json</span><br></pre></td></tr></table></figure>
<h4 id="選定-Base-image"><a href="#選定-Base-image" class="headerlink" title="選定 Base image"></a>選定 Base image</h4><p>Docker 的 image 是一層一層疊加，所以我們需要先從 <a href="https://hub.docker.com/" title="Docker Hub">Docker Hub</a> 上找到適合的 base image, 再慢慢加上我們需要的東西。</p>
<p><a href="https://larrylu.blog/step-by-step-dockerize-your-app-ecd8940696f4"><img src="https://cdn-images-1.medium.com/max/1600/1*bKRHfz7unRA35WHd9SBOCA.png" alt="Docker"></a></p>
<p>因為我們的專案是用 Node.js 寫的，所以這裡我們選擇 node 作為 base image，如果是 Python 或其他語言也可以找到對應的 image 作為 base image.</p>
<p>使用 <code>FROM</code> 設定 base image:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">FROM</span> node:<span class="number">12.2</span>.<span class="number">0</span></span><br></pre></td></tr></table></figure>
<p>指定 node 版本為 12.2.0，如果要用最新版也可以寫成 <code>node:latest</code>.</p>
<h4 id="設定-Working-directory"><a href="#設定-Working-directory" class="headerlink" title="設定 Working directory"></a>設定 Working directory</h4><figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">WORKDIR</span><span class="language-bash"> /app</span></span><br></pre></td></tr></table></figure>
<h4 id="Copy-code"><a href="#Copy-code" class="headerlink" title="Copy code"></a>Copy code</h4><p>使用 copy 指令將程式碼複製到 <code>WORKDIR</code>, 這裡我們假設 <code>WORKDIR</code> 是 <code>/app/</code>:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">COPY</span><span class="language-bash"> app.js package.json /app/</span></span><br></pre></td></tr></table></figure>
<p>如果想要複製所有檔案，只排除特定檔案，可以建立 <code>.dockerignore</code>，裡面寫要排除檔案:</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line"># Ignore files</span><br><span class="line">node_modules</span><br><span class="line">.git</span><br><span class="line">.circleci</span><br><span class="line">test</span><br><span class="line">*.txt</span><br></pre></td></tr></table></figure>
<p>然後在 <code>Dockerfile</code> 就可以直接寫:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">COPY</span><span class="language-bash"> . /app/</span></span><br></pre></td></tr></table></figure>
<h4 id="安裝所需套件"><a href="#安裝所需套件" class="headerlink" title="安裝所需套件"></a>安裝所需套件</h4><p>使用 <code>RUN</code> 執行安裝指令，清理 npm cache 是為了讓 build 出來的 image 小一點。</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">RUN</span><span class="language-bash"> npm i && npm cache clean --force</span></span><br></pre></td></tr></table></figure>
<h4 id="設定-Port"><a href="#設定-Port" class="headerlink" title="設定 Port"></a>設定 Port</h4><p>設定 container 開放的 port:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">EXPOSE</span> <span class="number">8080</span></span><br></pre></td></tr></table></figure>
<h4 id="設定環境變數"><a href="#設定環境變數" class="headerlink" title="設定環境變數"></a>設定環境變數</h4><figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">ENV</span> NAME HelloWord</span><br></pre></td></tr></table></figure>
<h4 id="設定-Initial-command"><a href="#設定-Initial-command" class="headerlink" title="設定 Initial command"></a>設定 Initial command</h4><p>最後就是執行程式的指令，使用 <code>CMD</code> 來設定這個 image run 起來之後要執行的預設指令:</p>
<figure class="highlight dockerfile"><table><tr><td class="code"><pre><span class="line"><span class="keyword">CMD</span><span class="language-bash"> node app.js</span></span><br></pre></td></tr></table></figure>
<h3 id="Build"><a href="#Build" class="headerlink" title="Build"></a>Build</h3><p>完成 Dockerfile 之後就可以開始建立 image, 執行:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ docker build -t <image_name>[:tag] <path></span><br></pre></td></tr></table></figure>
<p>建立完之後，執行 <code>docker images</code> 就可以看到多了剛才建立的 image 囉!</p>
<h3 id="執行打包的應用程式"><a href="#執行打包的應用程式" class="headerlink" title="執行打包的應用程式"></a>執行打包的應用程式</h3><p>直接使用 <code>docker run <image></code> 就可以啟動應用程式，並且預設會執行 <code>CMD</code> 設定的指令，這樣就成功了!</p>
<p>如果想要在背景執行，可以加上 <code>-d</code> 參數，讓 container 在背景執行，如果想要看 log, 可以執行 <code>docker logs <container_id></code>.</p>
<p>另外可以使用 <code>-p</code> 將外部 Host 的 port 轉到 container 的 port, 例如: 將伺服器的 8080 port 轉到 container 的 8080 port:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">docker run -d -p 8080:8080 <image></span><br></pre></td></tr></table></figure>
<h3 id="佈署到遠端-server"><a href="#佈署到遠端-server" class="headerlink" title="佈署到遠端 server"></a>佈署到遠端 server</h3><p>如果想要佈署到遠端 server, 有以下幾種方式:</p>
<ol>
<li>將 image 傳到遠端 server</li>
<li>將 Dockerfile 和所有程式碼放在 git repo, 直接在遠端 pull repo 之後 build image</li>
<li>將自己的 image push 到 Docker Hub</li>
</ol>
<p>第一點要將 image 傳到遠端 server 有點麻煩，第三點因為 Docker Hub 的 private repository 只能有一個，而 image 又包含自己的程式碼或是一些重要的 key, 不能把它設為 public repository, 所以建議用第二個方法，在遠端 server 重新 build image, 因為都是依照 Dockerfile 的步驟執行，所以不用擔心執行結果不同。</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://larrylu.blog/step-by-step-dockerize-your-app-ecd8940696f4">Docker 實戰系列（一）：一步一步帶你 dockerize 你的應用</a></li>
<li><a href="https://github.com/twtrubiks/docker-tutorial">docker-tutorial</a></li>
</ul>
]]></content>
<categories>
<category>Docker</category>
</categories>
<tags>
<tag>Docker</tag>
</tags>
</entry>
<entry>
<title>[ELK] 利用 ELK 架構分析 log</title>
<url>/posts/5e2804d6/</url>
<content><![CDATA[<p><a href="https://medium.com/oneclicklabs-io/streaming-spring-boot-application-logs-to-elk-stack-part-1-a68bd7cccaeb"><img src="https://cdn-images-1.medium.com/max/748/1*mwSvtVy_qGz0nTjaYbvwpw.png" alt="ELK"></a></p>
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>通常我們架設網站或服務時，會把 Log 寫在檔案中，方便追蹤系統狀態，當發生問題時也能夠查看紀錄，但是當我們的服務是分散在許多台機器時，每次發生問題就要到各個機器開啟 Log file 查看，這樣的過程很繁瑣又很沒效率，所以這一篇文章主要紀錄如何利用 ELK 架構來分析 Log，透過 ELK 架構可以更方便地追蹤系統狀態、分析 Log.</p>
<span id="more"></span>
<p>ELK 是由 <a href="https://www.elastic.co/cn/products/elasticsearch" title="Elasticsearch">Elasticsearch</a>、<a href="https://www.elastic.co/cn/products/logstash" title="Logstash">Logstash</a> 和 <a href="https://www.elastic.co/cn/products/kibana" title="Kibana">Kibana</a> 所組成的 Log 收集、分析和查詢的架構:</p>
<ul>
<li>Elasticsearch: 是一個以 <a href="https://lucene.apache.org/core/">Apache Lucene</a> 為核心，分散式的 RESTful 風格的搜尋和數據分析引擎。它是以 JSON 的形式儲存資料，並提供即時的分析及搜尋。在此架構中, 主要作為儲存和查詢 Log 的搜尋引擎。</li>
<li>Logstash: 是開源的 Log 收集、處理的工具，並將處理後的 Log 資料儲存到 Elasticsearch.</li>
<li>Kibana: 將 Elasticsearch 中的資料以視覺化的方式呈現，並提供操作 Elastic Stack 的 UI 介面。</li>
</ul>
<h2 id="安裝及設定"><a href="#安裝及設定" class="headerlink" title="安裝及設定"></a>安裝及設定</h2><h3 id="Elasticsearch-amp-Kibana"><a href="#Elasticsearch-amp-Kibana" class="headerlink" title="Elasticsearch & Kibana"></a>Elasticsearch & Kibana</h3><p>可以參考 <a href="/posts/e0cb3249/"><a href="https://www.elastic.co/cn/products/elasticsearch" title="Elasticsearch">Elasticsearch</a> 使用 Elasticsearch + Kibana 實現中文全文檢索</a> 來安裝和設定 Elasticsearch 和 Kibana.</p>
<h3 id="Logstash"><a href="#Logstash" class="headerlink" title="Logstash"></a>Logstash</h3><p>直接從 <a href="https://www.elastic.co/cn/products/logstash" title="Logstash">Logstash</a> 網站中下載並解壓縮:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz</span><br><span class="line">$ tar zxvf logstash-6.2.3.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> logstash-6.2.3</span><br></pre></td></tr></table></figure>
<h2 id="使用-ELK-架構"><a href="#使用-ELK-架構" class="headerlink" title="使用 ELK 架構"></a>使用 ELK 架構</h2><p>依照上面步驟安裝完 Elasticsearch、Kibana 和 Logstash 之後，可以開始來設定 Logstash 蒐集 Log 資料，將資料處理過後儲存到 Elasticsearch, 最後透過 Kibana 呈現 Log 內容，流程如下:</p>
<p><a href="https://oranwind.org/dv-elk-an-zhuang-ji-she-ding-jiao-xue/"><img src="http://upload-images.jianshu.io/upload_images/5342565-8df4892e85e1e904.png" alt="ELK流程"></a></p>
<h3 id="設定-Logstash"><a href="#設定-Logstash" class="headerlink" title="設定 Logstash"></a>設定 Logstash</h3><p>Logstash 處理資料的流程: <code>輸入資料 → 過濾/處理資料 → 輸出資料</code>, 設定檔格式如下:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">input</span> {</span><br><span class="line"></span><br><span class="line">}</span><br><span class="line"><span class="string">filter</span> {</span><br><span class="line"></span><br><span class="line">}</span><br><span class="line"><span class="string">output</span> {</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>這裡稍微介紹一下 Logstash 常用的 plugins:</p>
<h4 id="Input-plugins"><a href="#Input-plugins" class="headerlink" title="Input plugins"></a>Input plugins</h4><p>我們主要透過 Input plugins 來讀取 Log 資料，以下是幾個常用的 Input plugins 及使用方式，更多 Input plugins 可以參考<a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html">官方網站 - Input plugins</a>.</p>
<h5 id="tcp"><a href="#tcp" class="headerlink" title="tcp"></a>tcp</h5><p>tcp plugin 是可以接收來自 TCP socket 的資料，將各個機器的 Log 送到指定的 port 就可以蒐集不同來源的資料了。</p>
<p>範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">input</span> {</span><br><span class="line"> <span class="string">tcp</span> {</span><br><span class="line"> <span class="string">port</span> <span class="string">=></span> <span class="number">9250</span></span><br><span class="line"> <span class="string">codec</span> <span class="string">=></span> <span class="string">json</span> <span class="comment"># vaule type, default is "plain"</span></span><br><span class="line"> <span class="string">type</span> <span class="string">=></span> <span class="string">dev_log</span> <span class="comment"># 分類, 可以用在 filter 進行資料處理</span></span><br><span class="line"> }</span><br><span class="line"> <span class="string">tcp</span> {</span><br><span class="line"> <span class="string">port</span> <span class="string">=></span> <span class="number">9260</span></span><br><span class="line"> <span class="string">codec</span> <span class="string">=></span> <span class="string">json</span></span><br><span class="line"> <span class="string">type</span> <span class="string">=></span> <span class="string">sys_log</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h5 id="file"><a href="#file" class="headerlink" title="file"></a>file</h5><p>file plugin 主要是用來讀取已經寫成純文字檔的 Log file, 預設會持續觀察指定檔案，如果檔案有更新則會觸發 Logstash 進行資料擷取。</p>
<p>範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">input</span> {</span><br><span class="line"> <span class="string">file</span> {</span><br><span class="line"> <span class="string">path</span> <span class="string">=></span> [<span class="string">"/var/log/*.log"</span>] <span class="comment"># path can be an array</span></span><br><span class="line"> <span class="string">type</span> <span class="string">=></span> <span class="string">"sys_log"</span></span><br><span class="line"> <span class="string">start_position</span> <span class="string">=></span> <span class="string">"beginning"</span> <span class="comment"># 如果是第一次啟動，可以設定讀取全部檔案內容</span></span><br><span class="line"> <span class="string">discover_interval</span> <span class="string">=></span> <span class="number">15</span> <span class="comment"># 多久察看一次指定檔案, 預設15秒</span></span><br><span class="line"> <span class="string">exclude</span> <span class="string">=></span> [<span class="string">"*.tar.gz"</span>] <span class="comment"># 排除的檔案</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h5 id="elasticsearch"><a href="#elasticsearch" class="headerlink" title="elasticsearch"></a>elasticsearch</h5><p>elasticsearch plugin 可以用來讀取 Elasticsearch 的 query result, 也可以排程定期執行一次 query.</p>
<p>範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">input</span> {</span><br><span class="line"> <span class="string">elasticsearch</span> {</span><br><span class="line"> <span class="string">hosts</span> <span class="string">=></span> [<span class="string">"localhost:9200"</span>] <span class="comment"># hosts can be an array</span></span><br><span class="line"> <span class="string">index</span> <span class="string">=></span> <span class="string">"mylog"</span> <span class="comment"># default is "logstash-*"</span></span><br><span class="line"> <span class="string">query</span> <span class="string">=></span> <span class="string">'{ \"query\": { "match": { "statuscode": 200 } }, "sort": [ "_doc" ] }'</span></span><br><span class="line"> <span class="string">schedule</span> <span class="string">=></span> <span class="string">"0 * * * *"</span> <span class="comment"># cron format, 每小時執行一次</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>上面範例等同於每小時執行一次以下指令:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl <span class="string">'http://localhost:9200/mylog/_search?&scroll=1m&size=1000'</span> -d <span class="string">'{</span></span><br><span class="line"><span class="string"> "query": {</span></span><br><span class="line"><span class="string"> "match": {</span></span><br><span class="line"><span class="string"> "statuscode": 200</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> },</span></span><br><span class="line"><span class="string"> "sort": [ "_doc" ]</span></span><br><span class="line"><span class="string">}'</span></span><br></pre></td></tr></table></figure>
<h4 id="Filter-plugins"><a href="#Filter-plugins" class="headerlink" title="Filter plugins"></a>Filter plugins</h4><p>透過 Filter plugins 可以過濾資料，這裡介紹一些 Filter plugins 的使用方式，更多 Filter plugins 可以參考<a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html">官方網站 - Filter plugins</a>:</p>
<h5 id="grok"><a href="#grok" class="headerlink" title="grok"></a>grok</h5><p>我們所要過濾的資料都在 <code>message</code> 這個欄位，也就是 Log file 的完整內容，而 <code>grok</code> 這個 plugin 有方便的字串處理功能，可以使用正規表達式及 grok 語法來處理資料。範例:</p>
<p>test.log:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">[2018-04-16 13:51:17.482] INFO MyServer - client:127.0.0.1, Server start.</span><br></pre></td></tr></table></figure>
<p>使用範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">grok</span> {</span><br><span class="line"> <span class="string">match</span> <span class="string">=></span> {</span><br><span class="line"> <span class="string">"message"</span> <span class="string">=></span> <span class="string">"\[(?<date>.+?)\] <span class="template-variable">%{LOGLEVEL:level}</span> <span class="template-variable">%{DATA:logger}</span> - client:<span class="template-variable">%{IPV4:client_ip}</span>, <span class="template-variable">%{DATA:message}</span>"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>上面的範例使用正規表達式切割出日期 <code>date</code>，剩下的則是使用 grok 語法來切割。grok 語法的格式如: <code>%{grok patterns:自訂屬性名稱}</code>，透過以上設定就可以將 Log 資料處理成:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"date"</span><span class="punctuation">:</span> <span class="string">"2018-04-16 13:51:17.482"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"level"</span><span class="punctuation">:</span> <span class="string">"INFO"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"logger"</span><span class="punctuation">:</span> <span class="string">"MyServer"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"client_ip"</span><span class="punctuation">:</span> <span class="string">"127.0.0.1"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"message"</span><span class="punctuation">:</span> <span class="string">"Server start."</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<p>詳細的 grok patterns 可以參考: <a href="https://grokdebug.herokuapp.com/patterns#">grok patterns</a>, 這裡也提供一個方便的工具 <a href="https://grokdebug.herokuapp.com/">grok debugger</a>, 可以用來測試 grok 語法是否正確。</p>
<h5 id="json"><a href="#json" class="headerlink" title="json"></a>json</h5><p>透過 json plugin, 可以處理 json 格式的資料，範例:</p>
<p>test.log:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">{"date":"2018-04-16 13:51:17.482","loglevel":"INFO","message":"Server start.","more":["test1", "test2"]}</span><br></pre></td></tr></table></figure>
<p>filter 設定:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">json</span> {</span><br><span class="line"> <span class="string">source</span> <span class="string">=></span> <span class="string">"message"</span></span><br><span class="line"> <span class="string">target</span> <span class="string">=></span> <span class="string">"jsoncontent"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>上面的範例是將 json 格式的資料處理後，儲存到 <code>jsoncontent</code> 屬性，結果如下:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"jsoncontent"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"date"</span><span class="punctuation">:</span> <span class="string">"2018-04-16 13:51:17.482"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"loglevel"</span><span class="punctuation">:</span> <span class="string">"INFO"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"message"</span><span class="punctuation">:</span> <span class="string">"Server start."</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"more"</span><span class="punctuation">:</span> <span class="punctuation">[</span><span class="string">"test1"</span><span class="punctuation">,</span> <span class="string">"test2"</span><span class="punctuation">]</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<h5 id="mutate"><a href="#mutate" class="headerlink" title="mutate"></a>mutate</h5><p>mutate 這個 plugin 除了有字串切割的功能之外，還有型態轉換和基本運算的功能，使用範例如下:</p>
<p>test.log:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">2018-04-16 13:51:17.482,INFO,MyServer,client:127.0.0.1,Server start.</span><br></pre></td></tr></table></figure>
<p>我們可以將 grok 處理後的資料取出來，再透過 mutate convert 轉換至其他型態:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">mutate</span> {</span><br><span class="line"> <span class="string">convert</span> <span class="string">=></span> [<span class="string">"exec_time"</span>, <span class="string">"float"</span>]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>也可以將 log file 內容以指定符號切割:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">mutate</span> {</span><br><span class="line"> <span class="string">split</span> <span class="string">=></span> [<span class="string">"message"</span>, <span class="string">","</span>]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>也可以取代特定字串:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">mutate</span> {</span><br><span class="line"> <span class="string">gsub</span> <span class="string">=></span> [<span class="string">"message"</span>, <span class="string">" "</span>, <span class="string">"_"</span>]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>另外還可以重新命名屬性欄位:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">mutate</span> {</span><br><span class="line"> <span class="string">rename</span> <span class="string">=></span> [<span class="string">"exec_time"</span>, <span class="string">"sys_exec_time"</span>]</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h5 id="geoip"><a href="#geoip" class="headerlink" title="geoip"></a>geoip</h5><p>geoip 可以分析 IPv4 或 IPv6, 提供 IP 的所在位置及相關資訊。</p>
<p>使用範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">grok</span> {</span><br><span class="line"> <span class="string">"match"</span> <span class="string">=></span> {</span><br><span class="line"> <span class="string">"message"</span> <span class="string">=></span> <span class="string">"<span class="template-variable">%{IPV4:ip}</span>"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="string">geoip</span> {</span><br><span class="line"> <span class="string">source</span> <span class="string">=></span> <span class="string">"ip"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>會產生 geoip 屬性，裡面包含 IP 的相關資訊:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="attr">"geoip"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"city_name"</span><span class="punctuation">:</span> <span class="string">"Hanoi"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"timezone"</span><span class="punctuation">:</span> <span class="string">"Asia/Ho_Chi_Minh"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"ip"</span><span class="punctuation">:</span> <span class="string">"123.30.238.16"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"latitude"</span><span class="punctuation">:</span> <span class="number">21.0333</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"country_code2"</span><span class="punctuation">:</span> <span class="string">"VN"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"country_name"</span><span class="punctuation">:</span> <span class="string">"Vietnam"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"continent_code"</span><span class="punctuation">:</span> <span class="string">"AS"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"country_code3"</span><span class="punctuation">:</span> <span class="string">"VN"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"region_name"</span><span class="punctuation">:</span> <span class="string">"Thanh Pho Ha Noi"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"location"</span><span class="punctuation">:</span> <span class="punctuation">[</span> <span class="number">105.85</span><span class="punctuation">,</span> <span class="number">21.0333</span> <span class="punctuation">]</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"longitude"</span><span class="punctuation">:</span> <span class="number">105.85</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"region_code"</span><span class="punctuation">:</span> <span class="string">"64"</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<h4 id="Output-plugins"><a href="#Output-plugins" class="headerlink" title="Output plugins"></a>Output plugins</h4><p>最後我們透過 Output plugins 來將處理後的資料輸出，在 ELK 架構中，我們搭配 elasticsearch plugin 來將資料輸出到 Elasticsearch, 其他的 Output plugins 可以參考: <a href="https://www.elastic.co/guide/en/logstash/current/output-plugins.html">官方網站 - Output plugins</a>.</p>
<p>完整的 Logstash config 範例:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">input</span> {</span><br><span class="line"> <span class="string">file</span> {</span><br><span class="line"> <span class="string">path</span> <span class="string">=></span> [<span class="string">"/home/user/test.log"</span>]</span><br><span class="line"> <span class="string">type</span> <span class="string">=></span> <span class="string">"test_log"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"><span class="string">filter</span> {</span><br><span class="line"> <span class="string">grok</span> {</span><br><span class="line"> <span class="string">match</span> <span class="string">=></span> {</span><br><span class="line"> <span class="string">"message"</span> <span class="string">=></span> <span class="string">"\[(?<date>.+?)\] <span class="template-variable">%{LOGLEVEL:level}</span> <span class="template-variable">%{DATA:logger}</span> - client:<span class="template-variable">%{IPV4:client_ip}</span>, <span class="template-variable">%{DATA:detail}</span>"</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"><span class="string">output</span> {</span><br><span class="line"> <span class="string">elasticsearch</span> {</span><br><span class="line"> <span class="string">hosts</span> <span class="string">=></span> [<span class="string">"localhost:9200"</span>] <span class="comment"># hosts can be an array</span></span><br><span class="line"> <span class="string">index</span> <span class="string">=></span> <span class="string">"mylog"</span> <span class="comment"># Elasticsearch index name</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>寫好設定檔之後，啟動 Logstash:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ./bin/logstash -f ./config/log.conf</span><br></pre></td></tr></table></figure>
<h3 id="瀏覽-Log-資料"><a href="#瀏覽-Log-資料" class="headerlink" title="瀏覽 Log 資料"></a>瀏覽 Log 資料</h3><p>接著我們開啟 Kibana 的 Discover 頁面, 就可以看到由 Logstash 輸出的 Log 資料囉!<br>如果是第一次使用，記得要先到 Management 建立 Index pattern, 就能在 Discover 中看到 Elasticsearch 中的資料了!</p>
<p><img src="https://i.imgur.com/yHY2f9f.png" alt="Kibana UI"></p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://hk.saowen.com/a/78d90e1799838b5e28369e9be35655de59c2495f554561dc9de5cda335403058">ELK入門級介紹–打造實時日誌查詢系統</a></li>
<li><a href="https://blog.johnwu.cc/article/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-red-hat.html">ELK 教學 - 從無到有安裝 ELK (CentOS/Red Hat)</a></li>
<li><a href="https://oranwind.org/dv-elk-an-zhuang-ji-she-ding-jiao-xue/">[DV] ELK 安裝及設定教學</a></li>
<li><a href="https://ithelp.ithome.com.tw/articles/10186786">Day5 - 便利的 logstash plugin (filter)</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
</categories>
<tags>
<tag>Elasticsearch</tag>
<tag>Logstash</tag>
<tag>Kibana</tag>
<tag>ELK</tag>
</tags>
</entry>
<entry>
<title>[Elasticsearch] Elasticsearch + Kibana 設定使用者認證</title>
<url>/posts/9cefa9a7/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>參考上一篇文章: <a href="/posts/e0cb3249/">[Elasticsearch] 使用 Elasticsearch + Kibana 實現中文全文檢索</a> 開始使用 Elasticsearch 和 Kibana 之後就會發現一個問題: 沒有使用者帳號密碼嗎? 因為像是 MySQL、MongoDB 等許多資料庫都有使用者認證的機制，需要有帳號密碼才可存取資料庫，但 Elasticsearch 和 Kibana 本身是沒有提供使用者驗證的機制，如果希望是內部使用，只能透過設定防火牆的方式限制特定網域使用，但是如果我們希望讓外部也能夠存取資源，綁定 IP 之後，任何人只要知道 IP 和 port 就都可以存取資料，這樣非常不安全，因此我們希望能夠設定使用者認證。</p>
<p>雖然 Elasticsearch 和 Kibana 本身沒有提供使用者認證機制，但 Elastic 有提供 X-Pack 工具，它可以用來做 Elasticsearch 和 Kibana 的安全防護、即時監控和產生報表等等，但是它是要付費的，試用期過了之後就無法使用。如果除了安全防護之外，也需要有即時監控、報表紀錄等等功能，也是可以考慮這個方案，詳細設定方式可以參考 <a href="https://www.elastic.co/">Elastic 官網</a>。</p>
<p>因為我們主要的需求是希望讓外部也能夠存取資源，但需要有使用者驗證機制，不要讓任何人都能夠存取資料，所以我們選擇使用 Nginx 作為 proxy，接收外部的請求，再透過 Nginx 轉發給 Elasticsearch 或 Kibana, 將 Elasticsearch 和 Kibana 設定為只能本機存取 (host 設為 <code>127.0.0.1</code>)，並針對 Nginx 設定 HTTP Basic Auth, 這樣就能夠達到需要輸入使用者帳號密碼才能夠存取 Elasticsearch 和 Kibana 的需求。</p>
<span id="more"></span>
<h2 id="安裝"><a href="#安裝" class="headerlink" title="安裝"></a>安裝</h2><p>首先，我們先來安裝所需要的套件:</p>
<h3 id="安裝-Nginx-相依套件"><a href="#安裝-Nginx-相依套件" class="headerlink" title="安裝 Nginx 相依套件"></a>安裝 Nginx 相依套件</h3><p><a href="http://pcre.org/">PCRE</a>: Supports regular expressions. Required by the NGINX Core and Rewrite modules.</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.43.tar.gz</span><br><span class="line">$ tar -zxf pcre-8.43.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> pcre-8.43</span><br><span class="line">$ ./configure</span><br><span class="line">$ make</span><br><span class="line">$ sudo make install</span><br></pre></td></tr></table></figure>
<p><a href="http://www.zlib.net/">zlib</a>: Supports header compression. Required by the NGINX Gzip module.</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget http://zlib.net/zlib-1.2.11.tar.gz</span><br><span class="line">$ tar -zxf zlib-1.2.11.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> zlib-1.2.11</span><br><span class="line">$ ./configure</span><br><span class="line">$ make</span><br><span class="line">$ sudo make install</span><br></pre></td></tr></table></figure>
<p><a href="https://www.openssl.org/">OpenSSL</a>: Supports the HTTPS protocol. Required by the NGINX SSL module and others</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget https://www.openssl.org/source/openssl-1.0.2r.tar.gz</span><br><span class="line">$ tar -zxf openssl-1.0.2r.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> openssl-1.0.2r</span><br><span class="line">$ ./Configure darwin64-x86_64-cc --prefix=/usr</span><br><span class="line">$ make</span><br><span class="line">$ sudo make install</span><br></pre></td></tr></table></figure>
<h3 id="安裝-Nginx"><a href="#安裝-Nginx" class="headerlink" title="安裝 Nginx"></a>安裝 Nginx</h3><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget http://nginx.org/download/nginx-1.16.0.tar.gz</span><br><span class="line">$ tar zxf nginx-1.16.0.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> nginx-1.16.0</span><br></pre></td></tr></table></figure>
<p>接下來設定 Nginx 路徑, 這裡我們把路徑設定在 <code>~/nginx/</code>:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ./configure --prefix=/home/<user>/nginx <span class="comment"># DEFAULT: /usr/local/nginx</span></span><br></pre></td></tr></table></figure>
<p>Make & Install:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ make</span><br><span class="line">$ make install</span><br></pre></td></tr></table></figure>
<h3 id="安裝-Apache-工具"><a href="#安裝-Apache-工具" class="headerlink" title="安裝 Apache 工具"></a>安裝 Apache 工具</h3><p>後面步驟會使用到此工具來建立密碼檔，因此我們先安裝 <code>apache2-utils</code>:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get install apache2-utils</span><br></pre></td></tr></table></figure>
<h2 id="設定"><a href="#設定" class="headerlink" title="設定"></a>設定</h2><h3 id="設定-Elasticsearch"><a href="#設定-Elasticsearch" class="headerlink" title="設定 Elasticsearch"></a>設定 Elasticsearch</h3><p>將 Elasticsearch 設定為只能本機存取, 修改 <code><ES_DIR>/config/elasticsearch.yml</code>:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">network.host:</span> <span class="number">127.0</span><span class="number">.0</span><span class="number">.1</span></span><br><span class="line"><span class="attr">http.port:</span> <span class="number">9200</span></span><br></pre></td></tr></table></figure>
<h3 id="設定-Kibana"><a href="#設定-Kibana" class="headerlink" title="設定 Kibana"></a>設定 Kibana</h3><p>將 Kibana 設定為只能本機存取, 修改 <code><Kibana_DIR>/config/kibana.yml</code>:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">server.port:</span> <span class="number">5601</span></span><br><span class="line"><span class="attr">server.host:</span> <span class="string">"127.0.0.1"</span></span><br></pre></td></tr></table></figure>
<p>並設定 Elasticsearch url:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">elasticsearch.url:</span> <span class="string">"http://127.0.0.1:9200"</span></span><br></pre></td></tr></table></figure>
<p>如果 Elasticsearch 和 Kibana 在不同台機器上，則需要設定為:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">elasticsearch.url:</span> <span class="string">"http://<nginx_host>:<nginx_es_port>"</span></span><br><span class="line"></span><br><span class="line"><span class="attr">elasticsearch.username:</span> <span class="string">"username"</span></span><br><span class="line"><span class="attr">elasticsearch.password:</span> <span class="string">"password"</span></span><br></pre></td></tr></table></figure>
<h3 id="設定-Nginx-Server"><a href="#設定-Nginx-Server" class="headerlink" title="設定 Nginx Server"></a>設定 Nginx Server</h3><p>接著設定 Nginx, 修改 <code><NGINX_DIR>/conf/nginx.conf</code>, 將原本的 http server 註解, 並加上:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="string">http</span> {</span><br><span class="line"> <span class="comment"># Elasticsearch</span></span><br><span class="line"> <span class="string">server</span> {</span><br><span class="line"> <span class="string">listen</span> <span class="number">9201</span><span class="string">;</span></span><br><span class="line"> <span class="string">server_name</span> <span class="string">localhost;</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Authorize</span></span><br><span class="line"> <span class="string">auth_basic</span> <span class="string">"ES Auth"</span><span class="string">;</span></span><br><span class="line"> <span class="string">auth_basic_user_file</span> <span class="string">/home/<user>/nginx/passwd/elasticsearch.passwd;</span></span><br><span class="line"></span><br><span class="line"> <span class="string">location</span> <span class="string">/</span> {</span><br><span class="line"> <span class="comment"># Proxy</span></span><br><span class="line"> <span class="string">proxy_pass</span> <span class="string">http://127.0.0.1:9200;</span></span><br><span class="line"> <span class="string">proxy_redirect</span> <span class="string">off;</span></span><br><span class="line"> <span class="string">proxy_http_version</span> <span class="number">1.1</span><span class="string">;</span></span><br><span class="line"></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">Connection</span> <span class="string">""</span><span class="string">;</span></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">X-Real-IP</span> <span class="string">$remote_addr;</span></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">X-Forwarded-For</span> <span class="string">$proxy_add_x_forwarded_for;</span></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">Host</span> <span class="string">$http_host;</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment"># Kibana</span></span><br><span class="line"> <span class="string">server</span> {</span><br><span class="line"> <span class="string">listen</span> <span class="number">5602</span><span class="string">;</span></span><br><span class="line"> <span class="string">server_name</span> <span class="string">localhost;</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># Authorize</span></span><br><span class="line"> <span class="string">auth_basic</span> <span class="string">"Kibana Auth"</span><span class="string">;</span></span><br><span class="line"> <span class="string">auth_basic_user_file</span> <span class="string">/home/<user>/nginx/passwd/kibana.passwd;</span></span><br><span class="line"></span><br><span class="line"> <span class="string">location</span> <span class="string">/</span> {</span><br><span class="line"> <span class="comment"># Proxy</span></span><br><span class="line"> <span class="string">proxy_pass</span> <span class="string">http://127.0.0.1:5601;</span></span><br><span class="line"> <span class="string">proxy_redirect</span> <span class="string">off;</span></span><br><span class="line"></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">X-Real-IP</span> <span class="string">$remote_addr;</span></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">X-Forwarded-For</span> <span class="string">$proxy_add_x_forwarded_for;</span></span><br><span class="line"> <span class="string">proxy_set_header</span> <span class="string">Host</span> <span class="string">$http_host;</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>讓 Nginx 可以接收外部請求，轉發到內部指定的 port，並使用 HTTP Basic Auth 做基本的登入限制。</p>
<h3 id="建立密碼檔"><a href="#建立密碼檔" class="headerlink" title="建立密碼檔"></a>建立密碼檔</h3><p>再來我們來建立密碼檔:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ htpasswd -c ./passwd/elasticsearch.passwd <username></span><br><span class="line">$ htpasswd -c ./passwd/kibana.passwd <username></span><br></pre></td></tr></table></figure>
<p><code>-c</code> 代表 create, 如果要建立多組帳號密碼，把 <code>-c</code> 拿掉即可。</p>
<p>完成後重新啟動 Nginx ，就可以來測試有沒有設定成功了!</p>
<p>啟動 Nginx:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ./sbin/nginx</span><br></pre></td></tr></table></figure>
<p>連線到 Elasticsearch 可以使用此指令格式:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -u <username> <span class="string">'http://<nginx_host>:<nginx_es_port>'</span></span><br></pre></td></tr></table></figure>
<p>連線到 Kibana 的部分，用瀏覽器開啟 <code>http://<nginx_host>:<nginx_kibana_port</code> 就會看到詢問帳號密碼的畫面，輸入後才能繼續操作:</p>
<p><img src="https://i.imgur.com/fiIflfm.png" alt="Kibana Login"></p>
<p>這樣我們就完成設定 Elasticsearch 和 Kibana 的使用者驗證囉!</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://coder.tw/?p=7235">Elasticsearch、Kibana 使用者認證設定</a></li>
<li><a href="https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#sources">Install Nginx</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
<category>Elasticsearch</category>
</categories>
<tags>
<tag>Nginx</tag>
<tag>Elasticsearch</tag>
<tag>Kibana</tag>
</tags>
</entry>
<entry>
<title>[Elasticsearch] 使用 Elasticsearch + Kibana 實現中文全文檢索</title>
<url>/posts/e0cb3249/</url>
<content><![CDATA[<p><a href="https://blog.toright.com/posts/5319/fulltext-search-elasticsearch-kibana-bigdata.html"><img src="https://blog.toright.com/wp-content/uploads/2017/07/elasticsearch-logo.png" alt="Elasticsearch"></a></p>
<h2 id="簡介"><a href="#簡介" class="headerlink" title="簡介"></a>簡介</h2><p><a href="https://www.elastic.co/cn/products/elasticsearch" title="Elasticsearch">Elasticsearch</a> 是一個以 <a href="https://lucene.apache.org/core/">Apache Lucene</a> 為核心，分散式的 RESTful 風格的搜尋和數據分析引擎。它是以 JSON 的形式儲存資料，並提供即時的分析及搜尋。</p>
<p><a href="https://www.elastic.co/cn/products/kibana" title="Kibana">Kibana</a> 將 Elasticsearch 中的資料以視覺化的方式呈現，並提供操作 Elastic Stack 的 UI 介面。</p>
<p><a href="https://github.com/medcl/elasticsearch-analysis-ik" title="ik analyzer">ik analyzer</a> 是一個 Elasticsearch 的中文分詞 plugin，由於 Elasticsearch 預設對於中文的分詞是一個字一個字切割，沒有分詞的話，中文搜尋的效果會比較差，所以我們需要加上中文分詞的 plugin, 讓搜尋的結果更好。</p>
<span id="more"></span>
<h2 id="基本概念"><a href="#基本概念" class="headerlink" title="基本概念"></a>基本概念</h2><p>Elasticsearch 和一般 RDBMS 的架構不同，所以在名詞上也不一樣，以下表格是 MySQL 和 Elasticsearch 名詞的對應關係:</p>
<table>
<thead>
<tr>
<th>MySQL</th>
<th>Elasticsearch</th>
</tr>
</thead>
<tbody><tr>
<td>Server</td>
<td>Node</td>
</tr>
<tr>
<td>Database</td>
<td>Index</td>
</tr>
<tr>
<td>Table</td>
<td>Type</td>
</tr>
<tr>
<td>Row</td>
<td>Document</td>
</tr>
<tr>
<td>Column</td>
<td>Field</td>
</tr>
</tbody></table>
<p>在 Elasticsearch 中，Index 的名稱必須是小寫，而同一個 Index、同一個 Type 中的每筆 record 的資料欄位不需要相同 (NoSQL的概念)。另外，根據<a href="https://www.elastic.co/blog/index-type-parent-child-join-now-future-in-elasticsearch">此文章</a>，在 Elasticsearch 6.x 版只允許每個 Index 包含一個 Type，在 7.x 版將會<strong>完全移除</strong> Type.</p>
<h2 id="安裝及設定"><a href="#安裝及設定" class="headerlink" title="安裝及設定"></a>安裝及設定</h2><h3 id="安裝-amp-設定-Elasticsearch"><a href="#安裝-amp-設定-Elasticsearch" class="headerlink" title="安裝&設定 Elasticsearch"></a>安裝&設定 Elasticsearch</h3><p>在安裝 Elasticsearch 之前，需要先安裝 Java 環境:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get update</span><br><span class="line">$ sudo apt-get install default-jre</span><br></pre></td></tr></table></figure>
<p>安裝完 Java 後，就可以來安裝 Elasticsearch, 直接下載並解壓縮即可:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.7.2.tar.gz</span><br><span class="line">$ tar zxvf elasticsearch-6.7.2.tar.gz</span><br></pre></td></tr></table></figure>
<p>設定 Elasticsearch (<code><ES_DIR>/config/elasticsearch.yml</code>):</p>
<p><strong>cluster name</strong><br>如果有多台 Elasticsearch node 要加入 cluster, 則必須定義相同的 cluster name.</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">cluster.name:</span> <span class="string">cluster_name</span></span><br></pre></td></tr></table></figure>
<p><strong>node.name</strong><br>自訂義 node name.</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">node.name:</span> <span class="string">node1</span></span><br></pre></td></tr></table></figure>
<p><strong>bootstrap.memory_lock</strong><br>設定為 true 是為了防止 swap 到 ES 的 memory.</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">bootstrap.memory_lock:</span> <span class="literal">true</span></span><br></pre></td></tr></table></figure>
<p><strong>network</strong></p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="comment"># ------------------------ Network ------------------------</span></span><br><span class="line"><span class="comment"># Set the bind address to a specific IP (IPv4 or IPv6):</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="attr">network.host:</span> <span class="string">host_ip</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Set a custom port for HTTP:</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="attr">http.port:</span> <span class="string">port</span></span><br><span class="line"><span class="attr">http.cors.enabled:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr">http.cors.allow-origin:</span> <span class="string">'*'</span></span><br><span class="line"><span class="attr">http.cors.allow-methods :</span> <span class="string">OPTIONS,</span> <span class="string">HEAD,</span> <span class="string">GET,</span> <span class="string">POST,</span> <span class="string">PUT,</span> <span class="string">DELETE</span></span><br><span class="line"><span class="attr">http.cors.allow-headers :</span> <span class="string">X-Requested-With,X-Auth-Token,Content-Type,</span> <span class="string">Content-Length</span></span><br></pre></td></tr></table></figure>
<p><strong>thread_pool</strong></p>
<p>可以參考 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html">Elasticsearch官網</a></p>
<p><code>index</code><br> For index/delete operations. Thread pool type is <code>fixed</code> with a size of <code># of available processors</code>, queue_size of 200. The maximum size of this pool is <code>1 + # of available processors</code>.</p>
<p><code>write</code><br>For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of <code># of available processors</code>, queue_size of 200. The maximum size for this pool is <code>1 + # of available processors</code>.</p>
<p>The <code>size</code> parameter controls the number of threads, and defaults to the number of cores times 5.</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Thread pool</span></span><br><span class="line"><span class="attr">thread_pool:</span></span><br><span class="line"> <span class="attr">index:</span></span><br><span class="line"> <span class="attr">size:</span> <span class="number">13</span></span><br><span class="line"> <span class="attr">queue_size:</span> <span class="number">1000</span></span><br><span class="line"> <span class="attr">write:</span></span><br><span class="line"> <span class="attr">size:</span> <span class="number">13</span></span><br><span class="line"> <span class="attr">queue_size:</span> <span class="number">1000</span></span><br></pre></td></tr></table></figure>
<p>ps. 如果不知道機器的 available processors，可以使用 <code>nproc</code> 指令來查詢。</p>
<p>接下來修改 JVM 的設定，Elasticsearch 預設的 JVM 大小為 1GB, 如果需要調整 memory 大小，可以在 <code>./elasticsearch-6.7.2/config/jvm.options</code> 做修改:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Xms represents the initial size of total heap space</span></span><br><span class="line"><span class="comment"># Xmx represents the maximum size of total heap space</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Set the minimum heap size (Xms) and maximum heap size (Xmx) to be equal to each other</span></span><br><span class="line"><span class="string">-Xms4g</span></span><br><span class="line"><span class="string">-Xmx4g</span></span><br></pre></td></tr></table></figure>
<p>ps. Elasticsearch 最多只會使用系統的 50% memory, 且不建議設定超過 32GB.</p>
<p>設定好之後啟動 Elasticsearch:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ <span class="built_in">cd</span> elasticsearch-6.7.2/bin</span><br><span class="line">$ ./elasticsearch</span><br></pre></td></tr></table></figure>
<p>啟動時可能會遇到一些問題，這裡列了目前有遇到的錯誤訊息和解決方式，如果啟動時有出現這些錯誤訊息可以參考以下解法:</p>
<ul>
<li><p>vm.max_map_count 太小<br>Error message:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">max virtual memory areas vm.max_map_count [65530] is too low, increase to at least</span><br></pre></td></tr></table></figure>
<p>解決方式: 增加 <code>vm.max_map_count</code> 的大小限制，<code>vm.max_map_count</code> 是用來限制 process 在 Virtual Memory Areas 擁有的最大數量。</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo sysctl -w vm.max_map_count=262144</span><br></pre></td></tr></table></figure>
</li>
<li><p>無法 lock memory<br>Error message:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">Unable to lock JVM Memory: error=12, reason=Cannot allocate memory</span><br></pre></td></tr></table></figure>
<p>解決方式: 修改 <code>/etc/security/limits.conf</code> 設定</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line"># allow user 'userA' mlockall</span><br><span class="line">userA soft memlock unlimited</span><br><span class="line">userA hard memlock unlimited</span><br></pre></td></tr></table></figure>
<p>接著重新登入即可生效。</p>
</li>
<li><p>NullPointerException<br>Error message:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">[ERROR][o.e.b.Bootstrap ] Exception</span><br><span class="line">java.lang.NullPointerException: null</span><br></pre></td></tr></table></figure>
<p>解決方式: 設定 cgroup</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpu,cpuacct cgroup /sys/fs/cgroup/cpu,cpuacct </span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果設定 line 1 的指令後，一樣無法啟動，請再試著設定以下內容:</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,freezer cgroup /sys/fs/cgroup/freezer</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,blkio cgroup /sys/fs/cgroup/blkio</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,hugetlb cgroup /sys/fs/cgroup/hugetlb</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,devices cgroup /sys/fs/cgroup/devices</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,net_cls,net_prio cgroup /sys/fs/cgroup/net_cls,net_prio</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpuset cgroup /sys/fs/cgroup/cpuset</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,memory cgroup /sys/fs/cgroup/memory</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,pids cgroup /sys/fs/cgroup/pids</span></span><br><span class="line"><span class="comment"># sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,perf_event cgroup /sys/fs/cgroup/perf_event</span></span><br></pre></td></tr></table></figure></li>
</ul>
<h3 id="安裝-amp-設定-ik-analyzer"><a href="#安裝-amp-設定-ik-analyzer" class="headerlink" title="安裝&設定 ik analyzer"></a>安裝&設定 ik analyzer</h3><p>參考 <a href="https://github.com/medcl/elasticsearch-analysis-ik" title="ik analyzer">ik analyzer</a>, 使用 elasticsearch-plugin 安裝 ik analyzer:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.7.2/elasticsearch-analysis-ik-6.7.2.zip</span><br></pre></td></tr></table></figure>
<p>NOTE: 將 <code>6.7.2</code> 替換為所使用的 Elasticsearch 版本。</p>
<p>接著設定 Dictionary, 可以準備自定義的字典檔讓搜尋效果更好, 字典及設定檔可以放在 <code><ES_DIR>/config/analysis-ik/</code> 或是 <code><ES_DIR>/plugins/analysis-ik/config/</code> 底下，再來在 <code>IKAnalyzer.cfg.xml</code> 設定使用自定義的字典:</p>
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="meta"><?xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span>?></span></span><br><span class="line"><span class="meta"><!DOCTYPE <span class="keyword">properties</span> <span class="keyword">SYSTEM</span> <span class="string">"http://java.sun.com/dtd/properties.dtd"</span>></span></span><br><span class="line"><span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">comment</span>></span>IK Analyzer 扩展配置<span class="tag"></<span class="name">comment</span>></span></span><br><span class="line"> <span class="comment"><!--用户可以在这里配置自己的扩展字典 --></span></span><br><span class="line"> <span class="tag"><<span class="name">entry</span> <span class="attr">key</span>=<span class="string">"ext_dict"</span>></span></span><br><span class="line"> custom/mydict.dic;</span><br><span class="line"> extra_main.dic;</span><br><span class="line"> extra_single_word_low_freq.dic;</span><br><span class="line"> <span class="tag"></<span class="name">entry</span>></span></span><br><span class="line"> <span class="comment"><!--用户可以在这里配置自己的扩展停止词字典--></span></span><br><span class="line"> <span class="tag"><<span class="name">entry</span> <span class="attr">key</span>=<span class="string">"ext_stopwords"</span>></span></span><br><span class="line"> extra_stopword.dic</span><br><span class="line"> <span class="tag"></<span class="name">entry</span>></span></span><br><span class="line"> <span class="comment"><!--用户可以在这里配置远程扩展字典 --></span></span><br><span class="line"> <span class="comment"><!-- <entry key="remote_ext_dict">words_location</entry> --></span></span><br><span class="line"> <span class="comment"><!--用户可以在这里配置远程扩展停止词字典--></span></span><br><span class="line"> <span class="comment"><!-- <entry key="remote_ext_stopwords">words_location</entry> --></span></span><br><span class="line"><span class="tag"></<span class="name">properties</span>></span></span><br></pre></td></tr></table></figure>
<p>最後重新啟動 Elasticsearch 就可以使用 ik analyzer.</p>
<h3 id="安裝-amp-設定-Kibana"><a href="#安裝-amp-設定-Kibana" class="headerlink" title="安裝&設定 Kibana"></a>安裝&設定 Kibana</h3><p>直接從 <a href="https://www.elastic.co/cn/products/kibana" title="Kibana">Kibana</a> 網站中下載並解壓縮:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.7.2-linux-x86_64.tar.gz</span><br><span class="line">$ tar zxvf kibana-6.7.2-linux-x86_64.tar.gz</span><br><span class="line">$ <span class="built_in">cd</span> kibana-6.7.2-linux-x86_64</span><br></pre></td></tr></table></figure>
<p>設定 <code><KIBANA_DIR>/config/kibana.yml</code>:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Kibana host and port</span></span><br><span class="line"><span class="attr">server.port:</span> <span class="number">5601</span></span><br><span class="line"><span class="attr">server.host:</span> <span class="string">"localhost"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Elasticsearch url</span></span><br><span class="line"><span class="attr">elasticsearch.url:</span> <span class="string">"http://localhost:9200"</span></span><br><span class="line"></span><br><span class="line"><span class="attr">logging.dest:</span> <span class="string">/path/to/kibana/log</span></span><br></pre></td></tr></table></figure>
<p>執行 Kibana:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ./bin/kibana</span><br></pre></td></tr></table></figure>
<p>啟動 Kibana 之後，開啟瀏覽器進入: <code>http://localhost:5601</code>，如果有正常連線到 Elasticsearch 就會看到以下畫面:</p>
<p><img src="https://i.imgur.com/E6iiQTW.png" alt="Kibana"></p>
<h2 id="操作-Elasticsearch"><a href="#操作-Elasticsearch" class="headerlink" title="操作 Elasticsearch"></a>操作 Elasticsearch</h2><h3 id="設定-Mapping"><a href="#設定-Mapping" class="headerlink" title="設定 Mapping"></a>設定 Mapping</h3><p>在建立索引之前，要先設定 Index 的 mapping，設定資料欄位的 datatype, format, analyzer 等等，讓搜尋效果可以更好，詳細的 Mapping 設定方式可以參考<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html">官方文件</a>，這裡範例是指定資料欄位使用 ik analyzer:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XPOST <span class="string">'http://localhost:9200/index/type/_mapping'</span> -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "properties": {</span></span><br><span class="line"><span class="string"> "content": {</span></span><br><span class="line"><span class="string"> "type": "text",</span></span><br><span class="line"><span class="string"> "analyzer": "ik_max_word",</span></span><br><span class="line"><span class="string"> "search_analyzer": "ik_max_word"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}'</span></span><br></pre></td></tr></table></figure>
<h3 id="設定回傳的資料量限制"><a href="#設定回傳的資料量限制" class="headerlink" title="設定回傳的資料量限制"></a>設定回傳的資料量限制</h3><p>Elasticsearch 預設的 from + size 最多為 10000, 如果需要增加大小，需要修改以下設定:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XPUT <span class="string">'http://<host>:<port>/<index>/_settings'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index": {</span></span><br><span class="line"><span class="string"> "max_result_window" : "300000"</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}'</span></span><br></pre></td></tr></table></figure>
<p>但是有可能會占用更多記憶體，所以要小心設定。</p>
<h3 id="設定-Log-level"><a href="#設定-Log-level" class="headerlink" title="設定 Log level"></a>設定 Log level</h3><p>當資料量大的時候，在 Elasticsearch 做全文搜尋時，CPU 使用率常常會很高，甚至到達 100%，我們可以透過設定 Log level 來減少 IO，降低 CPU 使用率:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XPUT <span class="string">'http://<host>:<port>/<index>/_settings'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "index.search.slowlog.level": "info"</span></span><br><span class="line"><span class="string">}'</span></span><br></pre></td></tr></table></figure>
<h3 id="檢查狀態"><a href="#檢查狀態" class="headerlink" title="檢查狀態"></a>檢查狀態</h3><h4 id="Index-狀態"><a href="#Index-狀態" class="headerlink" title="Index 狀態"></a>Index 狀態</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XGET http://127.0.0.1:9200/_cat/indices?v</span><br></pre></td></tr></table></figure>
<p>回傳結果:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">health status index uuid pri rep docs.count docs.deleted store.size pri.store.size</span><br><span class="line">yellow open dcard SWUzC7XRSU6DfGGwa_rtHw 5 1 6 0 84.4kb 84.4kb</span><br><span class="line">yellow open ptt DzwrF9YMRQm_vzpp_Xz8eg 5 1 912 0 166.4kb 166.4kb</span><br><span class="line">yellow open mylog x74lnUFcTZO1IU4rznok0w 5 1 5 0 37.5kb 37.5kb</span><br><span class="line">yellow open my_index IFc49rh8So2AtXrW5gIXng 5 1 5 0 16.9kb 16.9kb</span><br><span class="line">yellow open fbpost xD9-BAR_R4ybkhAYnXzLYA 5 1 6 1 98.9kb 98.9kb</span><br><span class="line">yellow open news WLNPnFXpQ8yQEH8c_saClg 5 1 5 0 55.3kb 55.3kb</span><br><span class="line">green open .kibana isZ0WfytSzWQcXXH0PxmWA 1 0 5 2 33.1kb 33.1kb</span><br></pre></td></tr></table></figure>
<ul>
<li>health: index 的健康狀態<ul>
<li>red: 資料缺損，無法使用</li>
<li>yellow: 資料只有一份，沒有 shards, 如果單一節點損壞的話，無法回復</li>
<li>green: 資料有 shards 的備，如果單點損壞還是可以正常檢索</li>
</ul>
</li>
<li>status: 是否啟用</li>
<li>index: 索引名稱</li>
<li>uuid: unique key</li>
<li>pri: 主要 shards 數量</li>
<li>rep: 備份 shards 數量</li>
<li>docs.count: index 中 doc 筆數</li>
<li>docs.deleted: index 中刪除的 doc 筆數</li>
<li>store.size: 儲存主要和備份資料所占用的空間</li>
<li>pri.store.size: 儲存主要資料所佔用的空間</li>
</ul>
<h4 id="線程狀態"><a href="#線程狀態" class="headerlink" title="線程狀態"></a>線程狀態</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -s -XGET http://127.0.0.1:9200/_cat/thread_pool?v</span><br></pre></td></tr></table></figure>
<h4 id="Cluster-狀態"><a href="#Cluster-狀態" class="headerlink" title="Cluster 狀態"></a>Cluster 狀態</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XGET http://127.0.0.1:9200/_cat/health?v</span><br></pre></td></tr></table></figure>
<h4 id="Node-狀態"><a href="#Node-狀態" class="headerlink" title="Node 狀態"></a>Node 狀態</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XGET http://127.0.0.1:9200/_cat/nodes?v</span><br></pre></td></tr></table></figure>
<h3 id="新增資料"><a href="#新增資料" class="headerlink" title="新增資料"></a>新增資料</h3><p>基本的指令格式:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XPUT <span class="string">'http://<host>:<port>/<index>/<type>/<doc_id>'</span> -d <span class="string">'</span></span><br><span class="line"><span class="string">{"data": data}'</span></span><br></pre></td></tr></table></figure>
<p>其中 <code>doc_id</code> 不一定要有，如果沒有指定 <code>doc_id</code>, HTTP request method 需使用 <code>POST</code>. For example:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 指定 doc_id</span></span><br><span class="line">$ curl -XPUT http://localhost:9200/index/type/1 -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"蘋果好綠！宣布全球設施已 100％ 使用再生能源"}'</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 不指定 doc_id</span></span><br><span class="line">$ curl -XPOST http://localhost:9200/index/type -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"阿里山花季謝幕 紫藤接替營造紫色浪漫"}'</span></span><br><span class="line"></span><br><span class="line">$ curl -XPOST http://localhost:9200/index/type -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"日本環球影城全新夜間遊行 四大特點搶先看"}'</span></span><br><span class="line"></span><br><span class="line">$ curl -XPOST http://localhost:9200/index/type -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"阿里山花季今閉幕群花接力開"}'</span></span><br><span class="line"></span><br><span class="line">$ curl -XPOST http://localhost:9200/index/fulltext -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"不甩聯合國美國怒嗆：要讓「怪物」阿薩德付出代價"}'</span></span><br></pre></td></tr></table></figure>
<h3 id="查詢資料-By-doc-id"><a href="#查詢資料-By-doc-id" class="headerlink" title="查詢資料 (By doc_id)"></a>查詢資料 (By doc_id)</h3><p>基本指令格式:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl <span class="string">'http://<host>:<port>/<index>/<type>/<doc_id>?pretty=true</span></span><br></pre></td></tr></table></figure>
<p>其中 URL 參數 <code>pretty=true</code> 代表以方便讀取的格式回傳, for example:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl <span class="string">'http://localhost:9200/index/type/1?pretty=true'</span></span><br></pre></td></tr></table></figure>
<p>回傳結果:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"_index"</span> <span class="punctuation">:</span> <span class="string">"index"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_type"</span> <span class="punctuation">:</span> <span class="string">"type"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_id"</span> <span class="punctuation">:</span> <span class="string">"1"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_version"</span> <span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"found"</span> <span class="punctuation">:</span> <span class="literal"><span class="keyword">true</span></span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_source"</span> <span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"content"</span><span class="punctuation">:</span><span class="string">"蘋果好綠！宣布全球設施已 100％ 使用再生能源"</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<h3 id="刪除資料"><a href="#刪除資料" class="headerlink" title="刪除資料"></a>刪除資料</h3><p>使用 <code>DELETE</code> method, for example:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XDELETE <span class="string">'http://localhost:9200/index/type/1'</span></span><br></pre></td></tr></table></figure>
<p>也可以直接刪除 Index 中的所有資料:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XDELETE <span class="string">'http://localhost:9200/index'</span></span><br></pre></td></tr></table></figure>
<h3 id="更新資料"><a href="#更新資料" class="headerlink" title="更新資料"></a>更新資料</h3><p>使用 <code>PUT</code> method, 重新發送一次 request 即可:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl -XPUT http://localhost:9200/index/type/1 -H <span class="string">'Content-Type:application/json'</span> -d<span class="string">'</span></span><br><span class="line"><span class="string">{"content":"蘋果好綠！宣布全球設施已 100％ 使用再生能源. Update!!"}'</span></span><br></pre></td></tr></table></figure>
<p>回傳結果:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"_index"</span><span class="punctuation">:</span> <span class="string">"index"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_type"</span><span class="punctuation">:</span> <span class="string">"type"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_id"</span><span class="punctuation">:</span> <span class="string">"1"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_version"</span><span class="punctuation">:</span> <span class="number">2</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"result"</span><span class="punctuation">:</span> <span class="string">"updated"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_shards"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"total"</span><span class="punctuation">:</span> <span class="number">2</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"successful"</span><span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"failed"</span><span class="punctuation">:</span> <span class="number">0</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_seq_no"</span><span class="punctuation">:</span> <span class="number">3</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_primary_term"</span><span class="punctuation">:</span> <span class="number">1</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<p>我們可以看到回傳的結果中 <code>_version</code> 是 2, <code>result</code> 為 <code>updated</code>, 代表是更新資料而不是新增資料。</p>
<h3 id="搜尋"><a href="#搜尋" class="headerlink" title="搜尋"></a>搜尋</h3><p>使用 Elasticsearch 最重要的就是搜尋的功能，我們使用它所提供的 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search.html">Search API</a> 來搜尋:</p>
<h4 id="取得所有紀錄"><a href="#取得所有紀錄" class="headerlink" title="取得所有紀錄"></a>取得所有紀錄</h4><p>使用 <code>GET</code> method, 後面加上 <code>_search</code>:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl <span class="string">'http://localhost:9200/index/type/_search'</span></span><br></pre></td></tr></table></figure>
<p>回傳結果:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"took"</span><span class="punctuation">:</span> <span class="number">31</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"timed_out"</span><span class="punctuation">:</span> <span class="literal"><span class="keyword">false</span></span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_shards"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"total"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"successful"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"skipped"</span><span class="punctuation">:</span> <span class="number">0</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"failed"</span><span class="punctuation">:</span> <span class="number">0</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"hits"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"total"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"max_score"</span><span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"hits"</span><span class="punctuation">:</span> <span class="punctuation">[</span></span><br><span class="line"> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"_index"</span><span class="punctuation">:</span> <span class="string">"index"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_type"</span><span class="punctuation">:</span> <span class="string">"type"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_id"</span><span class="punctuation">:</span> <span class="string">"PXlMs2IBy_MbTvZJQ_1N"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_score"</span><span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_source"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"content"</span><span class="punctuation">:</span> <span class="string">"阿里山花季謝幕 紫藤接替營造紫色浪漫"</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"_index"</span><span class="punctuation">:</span> <span class="string">"index"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_type"</span><span class="punctuation">:</span> <span class="string">"type"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_id"</span><span class="punctuation">:</span> <span class="string">"1"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_score"</span><span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_source"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"content"</span><span class="punctuation">:</span> <span class="string">"蘋果好綠！宣布全球設施已 100％ 使用再生能源"</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> <span class="punctuation">]</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<h4 id="全文搜尋"><a href="#全文搜尋" class="headerlink" title="全文搜尋"></a>全文搜尋</h4><p>Elasticsearch 有自己的查詢語法，詳細可以參考 <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html">Query DSL</a>, 這裡的範例是搜尋 <code>美國</code> 並將比對到的地方 highlight:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ curl <span class="string">'localhost:9200/index/type/_search'</span> -d <span class="string">'</span></span><br><span class="line"><span class="string">{</span></span><br><span class="line"><span class="string"> "query" : {</span></span><br><span class="line"><span class="string"> "match": { "content": "美國" }</span></span><br><span class="line"><span class="string"> },</span></span><br><span class="line"><span class="string"> "highlight" : {</span></span><br><span class="line"><span class="string"> "fields" : {</span></span><br><span class="line"><span class="string"> "content" : {}</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string"> }</span></span><br><span class="line"><span class="string">}'</span></span><br></pre></td></tr></table></figure>
<p>搜尋結果:</p>
<figure class="highlight json"><table><tr><td class="code"><pre><span class="line"><span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"took"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"timed_out"</span><span class="punctuation">:</span> <span class="literal"><span class="keyword">false</span></span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_shards"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"total"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"successful"</span><span class="punctuation">:</span> <span class="number">5</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"skipped"</span><span class="punctuation">:</span> <span class="number">0</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"failed"</span><span class="punctuation">:</span> <span class="number">0</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"hits"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"total"</span><span class="punctuation">:</span> <span class="number">1</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"max_score"</span><span class="punctuation">:</span> <span class="number">0.92769736</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"hits"</span><span class="punctuation">:</span> <span class="punctuation">[</span></span><br><span class="line"> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"_index"</span><span class="punctuation">:</span> <span class="string">"index"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_type"</span><span class="punctuation">:</span> <span class="string">"type"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_id"</span><span class="punctuation">:</span> <span class="string">"QHlMs2IBy_MbTvZJXv3t"</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_score"</span><span class="punctuation">:</span> <span class="number">0.92769736</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"_source"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"content"</span><span class="punctuation">:</span> <span class="string">"不甩聯合國美國怒嗆：要讓「怪物」阿薩德付出代價"</span></span><br><span class="line"> <span class="punctuation">}</span><span class="punctuation">,</span></span><br><span class="line"> <span class="attr">"highlight"</span><span class="punctuation">:</span> <span class="punctuation">{</span></span><br><span class="line"> <span class="attr">"content"</span><span class="punctuation">:</span> <span class="punctuation">[</span></span><br><span class="line"> <span class="string">"不甩聯合國<em>美國</em>怒嗆：要讓「怪物」阿薩德付出代價"</span></span><br><span class="line"> <span class="punctuation">]</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"> <span class="punctuation">]</span></span><br><span class="line"> <span class="punctuation">}</span></span><br><span class="line"><span class="punctuation">}</span></span><br></pre></td></tr></table></figure>
<h3 id="備份"><a href="#備份" class="headerlink" title="備份"></a>備份</h3><p>Elasticsearch 的備份，可以參考由 taskrabbit 開發的 <a href="https://github.com/taskrabbit/elasticsearch-dump">elasticsearch-dump</a> 工具來做資料的備份及轉移。</p>
<h2 id="操作-Kibana"><a href="#操作-Kibana" class="headerlink" title="操作 Kibana"></a>操作 Kibana</h2><p>由於從 cmd 操作 Elasticsearch 比較難閱讀，Kibana 提供漂亮的介面，讓我們可以更方便地操作 Elasticsearch, 並將 Elasticsearch 中的資料以視覺化的方式呈現，這裡我們簡單介紹一下 Kibana 的使用方式。</p>
<p><img src="https://i.imgur.com/eqTx39V.png" alt="Kibana UI"></p>
<p>一進到 Kibana 頁面，我們可以看到左側主選單有以下幾個功能:</p>
<ul>
<li>Discover: 檢視每個索引下的紀錄筆數和內容。</li>
<li>Visualize: 將搜尋結果是以視覺化的圖表呈現，並可以將搜尋結果或圖表儲存。</li>
<li>Dashboard: 組合多個已儲存的圖表或搜尋結果，方便一次瀏覽所有資訊。</li>
<li>Dev Tools: 是一個方便的除錯測試工具，可以在 Console 輸入指令直接操作 Elasticsearch.</li>
<li>Management: 設定 Kibana 對應的 Elasticsearch index patterns, 管理已儲存的搜尋結果 object、視覺化圖表以及進階的設定。</li>
</ul>
<p>如果是第一次使用，我們需要先到 Management 建立 Index pattern, 這樣就可以在 Discover 中搜尋特定條件的資料，並且在 Visualize 中以視覺化圖表的方式呈現，最後透過 Dashboard 組合多個搜尋結果 object 和圖表，一次瀏覽所需要的資訊。</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://blog.toright.com/posts/5319/fulltext-search-elasticsearch-kibana-bigdata.html">安裝 ElasticSearch + Kibana 實現中文全文搜尋與數據分析</a></li>
<li><a href="http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html">全文搜索引擎 Elasticsearch 入门教程</a></li>
<li><a href="https://ithelp.ithome.com.tw/articles/10189791">[ELK] Elasticsearch 安裝 (5.4.1版本)</a></li>
<li><a href="https://www.jianshu.com/p/008f1c2a3653">elasticsearch6.3.2安装</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
<category>Elasticsearch</category>
</categories>
<tags>
<tag>Elasticsearch</tag>
<tag>Kibana</tag>
<tag>ik分詞</tag>
</tags>
</entry>
<entry>
<title>[Elasticsearch] 解決 FORBIDDEN/12/index read-only / allow delete (api) 的問題</title>
<url>/posts/c04104b4/</url>
<content><![CDATA[<p>在 Elasticsearch 運行的過程中，很有可能會遇到 <code>FORBIDDEN/12/index read-only / allow delete (api)</code> 的錯誤，發生的原因是 Disk 空間不足，這和 Elasticsearch 中的幾個設定有關:</p>
<span id="more"></span>
<p><code>cluster.routing.allocation.disk.watermark.low</code>: 控制 Disk usage 的 low watermark, 預設是 85%，代表 Elasticsearch 不會配置 shards 給 Disk 使用量高於 85% 的節點。其值可以是百分比，也可以是絕對的數值(例如: 20gb)。此設定可以避免 Elasticsearch 配置 shards 給 Disk 空間不足的節點。此設定對於新創建的 index 的 primary shard 是不影響的，也就是說如果 cluster 中所有節點的 Disk usage 都超過 85%，有新創建的 index 需要進行分配，此時新創建的 primary shard 會正常分配，非主要的 shard 會無法分配而導致 cluster status 為 yellow.</p>
<p><code>cluster.routing.allocation.disk.watermark.high</code>: 控制 Disk usage 的 high watermark, 預設是 90%，代表 Elasticsearch 會企圖嘗試將 Disk 使用量高於 90% 的節點重新配置 shards。其值和 <code>low watermark</code> 一樣可以是百分比，也可以是絕對的數值(例如: 10gb)。此設定會影響到所有 shards，不論是否已經配置過，都會影響到。</p>
<p><code>cluster.routing.allocation.disk.watermark.flood_stage</code>: 控制 flood stage watermark. 預設是 95%，代表 Elasticsearch 會強制將有一個或多個 shard 被配置到 Disk 使用量超過 95% 的節點上的所有 index 變成 read-only index block (<code>index.blocks.read_only_allow_delete</code>)，這是最後會採取的措施，用來避免節點的 Disk 空間被用盡。當有足夠 Disk 空間時，需要再手動釋放 index block.</p>
<p>所以當發生 <code>FORBIDDEN/12/index read-only / allow delete (api)</code> 錯誤時，代表已經達到 flood stage，解決方式就是整理 Disk 空間、刪除部分資料，當有足夠 Disk 空間後，執行以下指令來釋放 index block:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">PUT /<your_index>/_settings</span><br><span class="line">{</span><br><span class="line"> <span class="string">"index.blocks.read_only_allow_delete"</span>: <span class="literal">false</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>ps. 在設定時，不能混合百分比和絕對的數值，必須全部都是百分比，或是全部都是絕對數值。</p>
<p><strong>參考資料</strong> </p>
<ul>
<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html">Disk-based Shard Allocation</a></li>
<li><a href="https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282">FORBIDDEN/12/index read-only / allow delete (api)</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
<category>Elasticsearch</category>
</categories>
<tags>
<tag>Elasticsearch</tag>
</tags>
</entry>
<entry>
<title>[Elasticsearch] 解決 HTTP ERROR: 413 (Request Entity Too Large)</title>
<url>/posts/b7da9ddc/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>在匯入資料到 Elasticsearch 時，如果出現以下 Error:</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)</span><br><span class="line">elasticsearch.exceptions.TransportError: TransportError(413, '<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor="white">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n')</span><br></pre></td></tr></table></figure>
<p>我們可以透過設定 <code>http.max_content_length</code> 來調整大小限制。</p>
<span id="more"></span>
<h2 id="Solution"><a href="#Solution" class="headerlink" title="Solution"></a>Solution</h2><p>在 Elasticsearch config (<code><ES_DIR>/config/elasticsearch.yml</code>) 中加上以下設定，調整 <code>http.max_content_length</code>:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="comment"># Set a custom allowed content length:</span></span><br><span class="line"><span class="attr">http.max_content_length:</span> <span class="string">500mb</span></span><br></pre></td></tr></table></figure>
<p>接著重新啟動 Elasticsearch 就可以囉!</p>
<p>另外如果有搭配 Nginx 做 proxy, 也記得要檢查 Nginx 的設定，可以參考 <a href="/posts/f8ec6a24/">[Nginx] 解決 HTTP ERROR: 413 Request Entity Too Large</a>。</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://github.com/elastic/elasticsearch/issues/2902">Return HTTP 413 (Request Entity Too Large) when http.max_content_length exceeded</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
<category>Elasticsearch</category>
</categories>
<tags>
<tag>Elasticsearch</tag>
</tags>
</entry>
<entry>
<title>[Elasticsearch] 解決 max_clause_count is set to 1024 的錯誤</title>
<url>/posts/a7aaf577/</url>
<content><![CDATA[<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>Elasticsearch 預設的 Lucene BooleanQuery 的 clauses 長度限制為 1024, 不只有 <code>bool</code> query, 其他的 query 也幾乎會在內部被轉換為 Lucene 的 BooleanQuery. 此限制主要是要避免 query 過大，造成使用過多 CPU 和 memory, 一般來說預設的 1024 就很夠用，但是如果出現 <code>max_clause_count is set to 1024</code> 這個錯誤訊息，代表 query 長度超過限制，可以透過以下的設定來做調整，不過要注意將此限制調高的話，可能會降低 performance 或是產生記憶體相關問題。</p>
<span id="more"></span>
<h2 id="設定"><a href="#設定" class="headerlink" title="設定"></a>設定</h2><p>若要調整 <code>max_clause_count</code>，在 <code><ES_DIR>/config/elasticsearch.yml</code> 中加上以下設定:</p>
<figure class="highlight yml"><table><tr><td class="code"><pre><span class="line"><span class="attr">indices.query.bool.max_clause_count:</span> <span class="number">10240</span></span><br></pre></td></tr></table></figure>
<p>接著重啟 Elasticsearch 就可以了~</p>
<h2 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h2><ul>
<li><a href="https://stackoverflow.com/questions/40275514/elasticsearch-set-max-clause-count">Elasticsearch - set max_clause_count</a></li>
</ul>
]]></content>
<categories>
<category>ELK</category>
<category>Elasticsearch</category>
</categories>
<tags>
<tag>Elasticsearch</tag>
</tags>
</entry>
<entry>
<title>[GCP] Google Compute Engine SSH連線設定</title>
<url>/posts/3b5e3bdc/</url>
<content><![CDATA[<p><img src="https://bit.ly/2NF2Nvg" alt="Google Cloud Platform"></p>
<h1 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h1><p>最近正好在研究 Google Cloud Platform (GCP)，GCP 有提供 300 美元的免費試用額度，期限是一年，免費額度用完之後需要升級帳戶才會再繼續扣款，所以可以不用擔心額度用完後就會被收錢，另外 GCP 也有提供免費的方案，可以參考 <a href="https://cloud.google.com/free/">Google Cloud Platform 免費版</a>，在用量限制內可以免費使用特定的產品。</p>
<p>這裡我們主要使用的產品是 GCP 中的 Compute Engine，它提供了許多主機規格可以自行定義，並依照不同規格來收費，計價方式可以參考 <a href="https://cloud.google.com/pricing/list">Google Cloud 價目表</a>。</p>
<p>這篇文章主要是紀錄在 Google Compute Engine 架設好之後，如何使用 SSH 連線到 Compute Engine 上，而不用從 Google developer console 頁面中連線到機器上。</p>
<span id="more"></span>
<h1 id="Linux-和-OSX-連線至-Compute-Engine"><a href="#Linux-和-OSX-連線至-Compute-Engine" class="headerlink" title="Linux 和 OSX 連線至 Compute Engine"></a>Linux 和 OSX 連線至 Compute Engine</h1><p>首先，我們需要先產生 SSH key:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ssh-keygen -t rsa -f ~/.ssh/ssh-key-gcp -C [username]</span><br></pre></td></tr></table></figure>
<p>其中的 <code>username</code> 是執行個體中要套用的金鑰使用者，如果沒有指定，Compute Engine 會自動使用產生這個金鑰的使用者。</p>
<p>執行完上面的指令之後，會在 <code>~/.ssh/</code> 中看到以下檔案:</p>
<ul>
<li>ssh-key-gcp: private key, 權限必須是 <code>600</code>.</li>
<li>ssh-key-gcp.pub: public key, 權限為 <code>644</code>.</li>
</ul>
<p>接著輸入以下指令，印出 public key 內容:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ <span class="built_in">cat</span> ssh-key-gcp.pub</span><br></pre></td></tr></table></figure>
<p>將 public key 的內容複製後，前往專案的<a href="https://console.cloud.google.com/compute/metadata/sshKeys?hl=zh-tw&_ga=2.71562451.-1591071787.1529480427">中繼資料</a>頁面:</p>
<p><img src="https://i.imgur.com/r3MK14E.png" alt="中繼資料"></p>
<p>選擇 <strong>SSH 金鑰</strong> 後，點選 <strong>編輯</strong> 按鈕，修改專案的 SSH key，將 public key 的內容貼到 SSH key 清單中，<br>最後點選 <strong>儲存</strong>，即可在 Terminal 使用以下指令連線到 Compute Engine:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ ssh -i ~/.ssh/ssh-key-gcp [username]@[compute_engine_ip]</span><br></pre></td></tr></table></figure>
<h1 id="Windows-連線至-Compute-Engine"><a href="#Windows-連線至-Compute-Engine" class="headerlink" title="Windows 連線至 Compute Engine"></a>Windows 連線至 Compute Engine</h1><p>在 Windows 系統中，我們一樣要先產生 SSH key，需要透過 <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html">puttygen</a> 這個工具來產生。</p>
<p>下載並執行 puttygen，可以在這個工具中調整金鑰產生的設定，大部分情況下使用預設的選項即可，設定好之後點選 <strong>Generate</strong> 即可產生金鑰。</p>
<p>接著在 <strong>key comment</strong> 的地方輸入 <code>username</code> (Google 使用者名稱)，並點選 <strong>Save private key</strong> 將 private key 儲存為 <code>ssh-key-gcp.ppk</code>， 以及點選 <strong>Save public key</strong> 儲存 public key 以便之後使用。</p>
<p><img src="https://i.imgur.com/JWpp7NU.png" alt="puttygen"></p>
<p>再來將 puttygen 中顯示的 public key 內容複製後，前往專案的<a href="https://console.cloud.google.com/compute/metadata/sshKeys?hl=zh-tw&_ga=2.71562451.-1591071787.1529480427">中繼資料</a>頁面:</p>
<p><img src="https://i.imgur.com/r3MK14E.png" alt="中繼資料"></p>
<p>選擇 <strong>SSH 金鑰</strong> 後，點選 <strong>編輯</strong> 按鈕，修改專案的 SSH key，將 public key 的內容貼到 SSH key 清單中，最後點選 <strong>儲存</strong>，再來開啟 putty 後，輸入使用者名稱和 Compute Engine IP:</p>
<p><img src="https://i.imgur.com/6zLm8Gx.png" alt="putty"></p>
<p>接著點選 <strong>Connection</strong> > <strong>SSH</strong> > <strong>Auth</strong>，選擇上面步驟所產生的 Private key file (<code>ssh-key-gcp.ppk</code>):</p>
<p><img src="https://i.imgur.com/Z1ME2FQ.png" alt="putty"></p>
<p>這樣就可以在 Windows 中使用 putty 連線到 Compute Engine 囉!</p>
<h1 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h1><ul>
<li><a href="https://cloud.google.com/compute/docs/instances/connecting-to-instance?hl=zh-tw">連線至 Linux 執行個體</a></li>
</ul>
]]></content>
<categories>
<category>GCP</category>
</categories>
<tags>
<tag>GCP</tag>
</tags>
</entry>
<entry>
<title>[GCP] Google Compute Engine 防火牆規則與設定</title>
<url>/posts/7801bf01/</url>
<content><![CDATA[<p>每次新開一個 Google Compute Engine 就會忘記要去哪裡設定防火牆 XD，所以就寫了這一篇文章來紀錄一下設定的步驟~</p>
<span id="more"></span>
<p>首先要先到 <strong>VPC 網路</strong> > <strong>防火牆規則</strong> 的頁面:</p>
<p><img src="https://i.imgur.com/tRUN5OR.png" alt="防火牆規則"></p>
<p>點選上方的 <strong>建立防火牆規則</strong> 按鈕，輸入規則的內容:</p>
<p><img src="https://i.imgur.com/gViYcYQ.png" alt="防火牆規則"></p>
<p><img src="https://i.imgur.com/rsg8dmB.png" alt="防火牆規則"></p>
<p>輸入完之後，按儲存就可以建立防火牆規則，這邊要記一下所輸入的目標代碼，接下來的步驟會需要用到~</p>
<p>再來到 <strong>Compute Engine</strong> > <strong>VM 執行個體</strong> 的頁面，點選要設定防火牆的 VM 並按下編輯按鈕，接著找到 <strong>網路標記</strong> 的欄位:</p>
<p><img src="https://i.imgur.com/kdOrXNo.png" alt="網路標記"></p>
<p>輸入剛才填寫的目標代碼，最後按下儲存就可以套用防火牆規則囉!</p>
]]></content>
<categories>
<category>GCP</category>
</categories>
<tags>
<tag>GCP</tag>
</tags>
</entry>
<entry>
<title>[Git] Git Flow 開發流程</title>
<url>/posts/96228b34/</url>
<content><![CDATA[<p><a href="https://nvie.com/posts/a-successful-git-branching-model/"><img src="http://bit.ly/2EjGXut" alt="Git flow"></a></p>
<h1 id="Git-Flow"><a href="#Git-Flow" class="headerlink" title="Git Flow"></a>Git Flow</h1><p>當越來越多人一起開發一個專案時，如果沒有好的開發流程，每個人的開發習慣都不同，到後期會讓整個專案很混亂，所以有人提出了一套開發流程: <a href="http://nvie.com/posts/a-successful-git-branching-model/">A successful Git branching model</a>，而後來也陸續有一些不錯的開發流程: <a href="https://guides.github.com/introduction/flow/">GitHub flow</a>, <a href="https://about.gitlab.com/2014/09/29/gitlab-flow/">GitLab flow</a>，這一篇主要是紀錄 Git flow 開發流程。</p>
<span id="more"></span>
<h1 id="分支"><a href="#分支" class="headerlink" title="分支"></a>分支</h1><p>分支分為 <code>master</code>, <code>develop</code>, <code>feature</code>, <code>release</code> 和 <code>hotfix</code> 這五種分支，各分支分別負責不同的功能。</p>
<h2 id="主要分支"><a href="#主要分支" class="headerlink" title="主要分支"></a>主要分支</h2><p>主要的分支是 <code>master</code> 和 <code>develop</code>，其他的分支大部分都會在完成後刪除。</p>
<h3 id="Master"><a href="#Master" class="headerlink" title="Master"></a>Master</h3><p>最新、穩定、隨時可以上線的版本，只能從其他分支合併過來，不可直接 commit 到這個分支。通常會在這個分支的 commit 加上版本號的 tag.</p>
<h3 id="Develop"><a href="#Develop" class="headerlink" title="Develop"></a>Develop</h3><p>主要用來開發的分支，當要開發新功能時，會從這個分支切出 <code>feature</code> 分支，當 <code>feature</code> 分支完成後，再合併回 <code>develop</code> 分支。</p>
<h2 id="次要分支"><a href="#次要分支" class="headerlink" title="次要分支"></a>次要分支</h2><p>次要的分支則是 <code>feature</code>, <code>release</code> 和 <code>hotfix</code>.</p>
<h3 id="Feature"><a href="#Feature" class="headerlink" title="Feature"></a>Feature</h3><p>主要是在新功能開發的時候，從 <code>develop</code> 分出來的，當完成後，會再合併回 <code>develop</code> 分支。</p>
<h3 id="Release"><a href="#Release" class="headerlink" title="Release"></a>Release</h3><p>為了發布而建立的分支，主要是做為上線前的最後測試。測試完後，<code>release</code> 會同時合併到 <code>master</code> 和 <code>develop</code> 分支，確保所有程式都是最新的狀態。</p>
<h3 id="Hotfix"><a href="#Hotfix" class="headerlink" title="Hotfix"></a>Hotfix</h3><p>當已經發布的產品發生緊急的問題時，從 <code>master</code> 切出來的分支，當修復完成後，會合併回 <code>master</code>，同時也會合併回 <code>develop</code> 分支，以確保所有程式都是最新狀態。</p>
<h1 id="開始使用-Git-Flow-開發流程"><a href="#開始使用-Git-Flow-開發流程" class="headerlink" title="開始使用 Git Flow 開發流程"></a>開始使用 Git Flow 開發流程</h1><p><a href="https://github.com/nvie/gitflow">Git flow GitHub 專案</a> 中有 <code>git flow</code> 工具，或是也可以直接使用 <code>git</code> 來實現 Git Flow 開發流程，這兩個方式我們都會稍微紀錄一下~</p>
<h2 id="安裝-Git-Flow"><a href="#安裝-Git-Flow" class="headerlink" title="安裝 Git Flow"></a>安裝 Git Flow</h2><p>如果想要使用 <code>git flow</code> 工具，需要先依照以下步驟來安裝。</p>
<p>我們的開發環境是在 Windows 10 上使用 VS code，所以接下來的安裝及使用方式都是以此環境為範例，其他環境的安裝方式可以參考 <a href="https://github.com/nvie/gitflow">Git flow GitHub 專案</a> 中的說明。</p>
<p>建議安裝 <a href="https://cygwin.com/install.html">Cygwin</a> , 在安裝時需選擇安裝 <code>git</code>, <code>util-linux</code> and <code>wget</code> tools.</p>
<p>完成安裝後，在 Cygwin shell 執行以下指令即可安裝 git flow:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ wget -q -O - --no-check-certificate https://github.com/nvie/gitflow/raw/develop/contrib/gitflow-installer.sh | bash</span><br></pre></td></tr></table></figure>
<h3 id="初始化"><a href="#初始化" class="headerlink" title="初始化"></a>初始化</h3><p>先初始化 git repo</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git init</span><br></pre></td></tr></table></figure>
<p>Initialize repository for git flow</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git flow init -d <span class="comment"># -d: 使用所有預設值</span></span><br></pre></td></tr></table></figure>
<p>依照提示設定即可完成初始設定。</p>
<h2 id="建立分支"><a href="#建立分支" class="headerlink" title="建立分支"></a>建立分支</h2><h3 id="建立-Feature-分支"><a href="#建立-Feature-分支" class="headerlink" title="建立 Feature 分支"></a>建立 Feature 分支</h3><p>如果一個功能有很多人一起進行，可以開多個 <code>feature/myFeature</code> 分支，例如: <code>feature/myFeature/user1</code>, <code>feature/myFeature/user2</code>, 最後 merge 回 <code>feature/myFeature</code>, 整個功能完全完成後再 merge 回 <code>develop</code>.</p>
<h4 id="使用-git-flow"><a href="#使用-git-flow" class="headerlink" title="使用 git flow"></a>使用 git flow</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git flow feature <span class="comment"># 顯示 feature 分支</span></span><br><span class="line">$ git flow feature start <name> [<base>] <span class="comment"># 建立新的 feature 分支</span></span><br><span class="line"></span><br><span class="line">$ git flow feature publish <name> <span class="comment"># push feature branch to remote repo.</span></span><br><span class="line">$ git flow feature pull <remote> <name> <span class="comment"># pull feature branch to remote repo.</span></span><br></pre></td></tr></table></figure>
<p>For feature branches, the <code><base></code> arg must be a commit on <code>develop</code>.</p>
<h4 id="使用-git"><a href="#使用-git" class="headerlink" title="使用 git"></a>使用 git</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git checkout -b feature/myFeature develop</span><br></pre></td></tr></table></figure>
<p><code>feature</code> 分支的名稱通常會加上 prefix: <code>feature/</code>, <code>base</code> 則是 <code>develop</code>.</p>
<h3 id="建立-Release-分支"><a href="#建立-Release-分支" class="headerlink" title="建立 Release 分支"></a>建立 Release 分支</h3><h4 id="使用-git-flow-1"><a href="#使用-git-flow-1" class="headerlink" title="使用 git flow"></a>使用 git flow</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git flow release <span class="comment"># 顯示 release 分支</span></span><br><span class="line">$ git flow release start <name> [<base>] <span class="comment"># 建立新的 release 分支</span></span><br></pre></td></tr></table></figure>
<p>For release branches, the <code><base></code> arg must be a commit on <code>develop</code>.</p>
<h4 id="使用-git-1"><a href="#使用-git-1" class="headerlink" title="使用 git"></a>使用 git</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git checkout -b release/1.0.0 develop</span><br></pre></td></tr></table></figure>
<p><code>release</code> 分支的名稱通常會加上 prefix: <code>release/</code>, 後面通常是加上版本號, <code>base</code> 則是 <code>develop</code>.</p>
<h3 id="建立-Hotfix-分支"><a href="#建立-Hotfix-分支" class="headerlink" title="建立 Hotfix 分支"></a>建立 Hotfix 分支</h3><h4 id="使用-git-flow-2"><a href="#使用-git-flow-2" class="headerlink" title="使用 git flow"></a>使用 git flow</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git flow hotfix <span class="comment"># 顯示 hotfix 分支</span></span><br><span class="line">$ git flow hotfix start <name> [<base>] <span class="comment"># 建立新的 hotfix 分支</span></span><br></pre></td></tr></table></figure>
<p>For hotfix branches, the <code><base></code> arg must be a commit on <code>master</code>.</p>
<h4 id="使用-git-2"><a href="#使用-git-2" class="headerlink" title="使用 git"></a>使用 git</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git checkout -b hotfix/1.0.1</span><br></pre></td></tr></table></figure>
<p><code>hotfix</code> 分支的名稱通常會加上 prefix: <code>hotfix/</code>, 後面通常是加上修改的版本號.</p>
<h2 id="完成分支任務"><a href="#完成分支任務" class="headerlink" title="完成分支任務"></a>完成分支任務</h2><p>最後，執行以下指令來完成分支任務:</p>
<h4 id="使用-git-flow-3"><a href="#使用-git-flow-3" class="headerlink" title="使用 git flow"></a>使用 git flow</h4><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git flow feature finish <name></span><br><span class="line">$ git flow release finish <name></span><br><span class="line">$ git flow hotfix finish <name></span><br></pre></td></tr></table></figure>
<h4 id="使用-git-3"><a href="#使用-git-3" class="headerlink" title="使用 git"></a>使用 git</h4><p>合併分支時，<code>release</code> 和 <code>hotfix</code> 分支記得要合併到 <code>master</code> 和 <code>develop</code>，合併可以使用 <code>merge</code> 或 <code>rebase</code> 指令:</p>
<p><strong>merge</strong><br> 修改內容的歷史記錄會維持原狀，但是合併後的歷史紀錄會變得更複雜。</p>
<p> 切換到要合併的<strong>主要分支</strong></p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git checkout <main_branch> <span class="comment"># <main_branch>: 要合併到的主要分支 (develop or master)</span></span><br></pre></td></tr></table></figure>
<p> Merge:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git merge --no-ff <name> <span class="comment"># <name>: 要合併的分支名稱, ex. feature/myFeature</span></span><br></pre></td></tr></table></figure>
<p> <code>--no-ff</code> 不要做 fast-forward 的動作。 Fast-forward 是指當被 merge 對象 (ex. feature/myFeature) 是目前分支 (ex. develop) 的 child 時， merge 後會完全看不出來有 merge 過。</p>
<p><strong>rebase</strong><br> 修改內容的歷史記錄會接在要合併的分支後面，合併後的歷史記錄會比較清楚簡單，但是比使用 merge 更容易發生衝突。<br> <strong>注意: 只能用在還沒有 push 的情況下。</strong></p>
<p> 切換到要 rebase 的<strong>次要分支</strong><br> <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git checkout feature/myFeature</span><br></pre></td></tr></table></figure></p>
<p> Rebase:<br> <figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git pull --rebase origin <main_branch> <span class="comment"># <main_branch>: 要 rebase 到的主要分支 (develop or master)</span></span><br></pre></td></tr></table></figure></p>
<p> 如果有先把 <code>feature/myFeature</code> 分支 pull 到本地, 可以直接執行:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git rebase <main_branch></span><br></pre></td></tr></table></figure>
<p> 當發生衝突的時候，需要去修改衝突的地方，修改好之後，如果要繼續 rebase 就加上 –continue，如果要取消 rebase 的話，則加上–abort.</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git add file.txt <span class="comment"># file.txt 為發生衝突的檔案, 修改好之後再加入</span></span><br><span class="line">$ git rebase --<span class="built_in">continue</span> <span class="comment"># 繼續 rebase</span></span><br><span class="line">$ git rebase --abort <span class="comment"># 取消 rebase</span></span><br></pre></td></tr></table></figure>
<p>最後記得要刪除次要分支:</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ git branch -d <name> <span class="comment"># <name>: 已合併到主要分支的次要分支名稱, ex. feature/myFeature</span></span><br></pre></td></tr></table></figure>
<h1 id="參考資料"><a href="#參考資料" class="headerlink" title="參考資料"></a>參考資料</h1><ul>
<li><a href="https://github.com/nvie/gitflow">Git flow Github專案</a></li>
<li><a href="http://blog.hellojcc.tw/2015/04/27/understanding-git-flow/">了解 Git Flow</a></li>
<li><a href="https://gitbook.tw/chapters/gitflow/why-need-git-flow.html">Git Flow 是什麼？為什麼需要這種東西？</a></li>
</ul>
]]></content>
<categories>
<category>Git</category>
</categories>
<tags>
<tag>Git</tag>
<tag>Git Flow</tag>
</tags>
</entry>
<entry>
<title>[Git] Git 安裝及使用</title>
<url>/posts/223ce999/</url>
<content><![CDATA[<p><a href="https://git-scm.com/"><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Git-logo.svg/400px-Git-logo.svg.png" alt="Git"></a></p>
<h1 id="簡介"><a href="#簡介" class="headerlink" title="簡介"></a>簡介</h1><p>在開發專案時，版本控制是很重要的，透過版本控制系統可以紀錄檔案修改的歷史紀錄、追蹤檔案修改前後的差異、回復到特定版本的檔案內容。若修改後發生問題，也能夠追蹤修改的部分，方便地找出是哪個部分導致了問題的發生。</p>
<p>Git 是一個分散式的版本控制系統，由 Linus Torvalds 開發，一開始是為了管理 Linux Kernel 原始碼，後來設計出 Git 版本控制系統，因為它的分散式、效能好、支援本地存取以及無痛分支的特性，近年來受到許多人的喜愛。</p>
<span id="more"></span>
<h1 id="安裝"><a href="#安裝" class="headerlink" title="安裝"></a>安裝</h1><h2 id="On-Linux"><a href="#On-Linux" class="headerlink" title="On Linux"></a>On Linux</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">$ sudo apt-get install git</span><br></pre></td></tr></table></figure>