Elasticsearch, Logstash, and Kibana 4

==============

Java 1.8+Elasticsearch 2.0+Kibana 4.2.0(失敗)
後降版至Kibana 3 成功
之後再補上筆記
==============




參考網址：
http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html

https://www.digitalocean.com/community/tutorial_series/centralized-logging-with-logstash-and-kibana-on-centos-7

http://zettadata.blogspot.tw/2014/09/elkelastic-searchlogstashkibanalog.html

http://icodding.blogspot.tw/2015/09/centos-elkelastic-searchlogstashkibana.html?showComment=1446104855572


看了許多文章和測試後目前自己設計的實驗架構如下：

Java 1.8

sudo  yum install java-1.8.0-openjdk httpd unzip


Elasticsearch 2.0

安裝
sudo vi /etc/yum.repos.d/elasticsearch.repo

Java 1.8

yum install java-1.8.0-openjdk httpd unzip

Elastic Search
Logstash

Kibana

https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz



安裝Elastic Search
下載公鑰
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

在 /etc/yum.repos.d/ 目錄下新增檔案，例如 elasticsearch.repo，內容如下
vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-2.0]
name=Elasticsearch repository for 2.0.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.3/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1




安裝Elastersearch
yum install elasticsearch

設定Elasticsearch
編輯 /etc/elasticsearch/elasticsearch.yml
sudo vi /etc/elasticsearch/elasticsearch.yml
cluster.name: "LogCluster"
node.name: "LogMaster"
node.master: true
node.data: true

path.conf: /etc/elasticsearch
path.data: /datapool/data1
path.work: /datapool/work
path.logs: /datapool/log

mkdir /datapool
mkdir /datapool/data1
mkdir /datapool/work
mkdir /datapool/log

將服務加入系統
sudo chkconfig --add elasticsearch

啟動服務
sudo /etc/init.d/elasticsearch start

測試啟動是否成功
curl localhost:9200/_nodes/process?pretty



防火牆開啟
sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5403/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5601/tcp
sudo firewall-cmd --permanent --zone=public --add-port=9200/tcp
sudo firewall-cmd --permanent --zone=public --add-port=9300/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-all


安裝Logstash
安裝金鑰
rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

在 /etc/yum.repos.d/ 目錄下新增檔案，例如 logstash.repo，內容如下
sudo vi /etc/yum.repos.d/logstash.repo
[logstash-2.0]
name=logstash repository for 2.0.x packages
baseurl=http://packages.elasticsearch.org/logstash/2.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

安裝Logstash
sudo yum install logstash

配置logstash，如下是Logstash的配置文件
==
sudo vi /etc/logstash/conf.d/logstasg_access.conf
==
sudo vi /etc/logstash/conf.d/01_apache_logstasg_input.conf
input {
file {
path => "/var/log/httpd/access_log"
type => "apache" # a type to identify those logs (will need this later)
}
}

filter {
if [type] == "apache" { # this is where we use the type from the input section
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
date {
# Try to pull the timestamp from the 'timestamp' field (parsed above with
# grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
if [user-agent] != "-" and [user-agent] != "" {
useragent {
add_tag => [ "UA" ]
source => "user-agent"
}
}
if "UA" in [tags] {
if [device] == "Other" { mutate { remove_field => "device" } }
if [name] == "Other" { mutate { remove_field => "name" } }
if [os] == "Other" { mutate { remove_field => "os" } }
}
}
}


sudo vi /etc/logstash/conf.d/02_apache_logstasg_output.conf

output {
elasticsearch {
host => "lsap.adtc.com.tw"
cluster => "elasticsearch"
node_name => "Elektra Natchios"
}
}

啟動 logstash 服務
cd /opt/logstash/bin/
./logstash

安裝Kibana 4.2.0(結構改變 需再研究)

sudo wget   https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz &&  gunzip kibana-4.2.0-linux-x64.tar.gz && tar -xvf kibana-4.2.0-linux-x64.tar && mv kibana-4.2.0-linux-x64  kibana && sudo mv kibana  /var/www/html/


安裝Kibana 3.1.2
wget  https://download.elasticsearch.org/kibana/kibana/kibana-3.1.2.zip &&  unzip kibana-3.1.2.zip &&  mv kibana-3.1.2  kibana && mv kibana  /var/www/html/


設定apache
vi /etc/httpd/conf/httpd.conf
<VirtualHost *:80>
ServerAdmin edwin.chen@adtc.com.tw
DocumentRoot /var/www/html/kibana
ServerName lsap.adtc.com.tw
ErrorLog logs/kibana.opsnotes.net-error_log
CustomLog logs/kibana.opsnotes.net-access_log common
</VirtualHost>

設定防火牆
#
# 開啟HTTP 80 port
#
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
#
# 設定ElasticSearch使用的port 9200-9300
#
iptables -A INPUT -p tcp --dport 9200:9300 -j ACCEPT

重啟 apache 服務
service httpd restart




sudo vim /etc/init.d/logstash

LS_USER=root
LS_GROUP=root

※為讀取LOG需改變權限為root
(應該可以透過SU更改)

==================

20161216 5.Xtest



#===JAVA=====
yum -y install net-tools
yum -y install java-1.8.0-openjdk.x86_64
yum -y install wget vim
echo $JAVA_HOME
/usr/sbin/alternatives --config java
java -version
vim /etc/profile

find /usr/lib/jvm/java-1.x.x-openjdk

export JAVA_HOME="path that you found"
export PATH=$JAVA_HOME/bin:$PATH

vim /root/.bash_profile
echo $JAVA_HOME
#===JAVA=====
#===elasticsearch

vim /etc/hosts
nmtui

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.4.3/elasticsearch-2.4.3.rpm
rpm -ivh elasticsearch-2.1.1.rpm

vim /etc/elasticsearch/elasticsearch.yml
vim  /etc/sysconfig/elasticsearch
vim  /usr/lib/systemd/system/elasticsearch.service
 vim /etc/sysconfig/elasticsearch

firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
firewall-cmd --reload
sudo systemctl start elasticsearch.service
sudo systemctl status elasticsearch.service



cd /usr/share/elasticsearch/bin
./plugin install lmenezes/elasticsearch-kopf

hostname -F /etc/hostname

#kibana-5.x

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

echo "[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md" | tee /etc/yum.repos.d/kibana.repo

yum install -y kibana

firewall-cmd --permanent --add-port=5601/tcp
firewall-cmd --reload
systemctl daemon-reload
systemctl enable kibana.service
systemctl start kibana.service


vim config/kibana.yml
elasticsearch.url