ELK分布式部署

scadm 2021年4月10日19:37:131 8956 5953字
ELK分布式部署

ELK工作流程

客户端安装Logstash日志收集工具,通过logstash收集客户端APP的日志数据,将所有的日志过滤出来,存入Elasticsearch 搜索引擎里,然后通过Kibana GUI在WEB前端展示给用户,用户需要可以进行查看指定的日志内容。同时也可以加入redis通信队列:

ELK分布式部署

ELK分布式部署

  • 加入Redis队列后工作流程;

Logstash包含Index和Agent(shipper) ,Agent负责客户端监控和过滤日志,而Index负责收集日志并将日志交给ElasticSearch,ElasticSearch将日志存储本地,建立索引、提供搜索,kibana可以从ES集群中获取想要的日志信息。

输入密码查看隐藏内容:

部署环境

elasticsearch:192.168.204.140

kibana:192.168.204.133

logstash:192.168.204.143

Elasticsearch配置

部署配置elasticsearch,需要配置JDK 环境,JDK(Java Development Kit) 是 Java 语言的软件开发工具包sdk

配置JAVA

[root@node1 ~]# tar -xf jdk-11.0.10_linux-x64_bin.tar.gz
[root@node1 ~]# mv jdk-11.0.10 /usr/local/jdk-11.0.10
[root@node1 ~]# echo  "export JAVA_HOME=/usr/local/jdk-11.0.10"   >> /etc/profile
[root@node1 ~]# echo  "export PATH=$JAVA_HOME/bin:$PATH "  >> /etc/profile
[root@node1 ~]# source /etc/profile

安装elasticsearch

[root@node1 ~]# tar -xf elasticsearch-7.11.1-linux-x86_64.tar.gz 
[root@node1 ~]# mv elasticsearch-7.11.1 /usr/local/elasticsearch

配置es文件

[root@node1 ~]# cd /usr/local/elasticsearch/config/
[root@node1 config]# vim elasticsearch.yml
#修改文件内容
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]

创建ES用户,并授权访问,并启动

[root@node1 config]# useradd elk 
[root@node1 config]# chown -R elk. /usr/local/elasticsearch/
[root@node1 config]# su - elk 
[elk@node1 ~]$ /usr/local/elasticsearch/bin/elasticsearch -d

##查看ES启动日志
[elk@node1 ~]$ tail -fn 10 /usr/local/elasticsearch/logs/elasticsearch.log

ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/local/elasticsearch/logs/elasticsearch.log
[2021-04-10T21:03:56,552][ERROR][o.e.b.Bootstrap ] [node-1] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-04-10T21:03:56,563][INFO ][o.e.n.Node ] [node-1] stopping ...
[2021-04-10T21:03:56,589][INFO ][o.e.n.Node ] [node-1] stopped
[2021-04-10T21:03:56,590][INFO ][o.e.n.Node ] [node-1] closing ...
[2021-04-10T21:03:56,612][INFO ][o.e.n.Node ] [node-1] closed
[2021-04-10T21:03:56,614][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started

处理启动报错

使用root用户修改
vim 	/etc/security/limits.conf 
* soft nofile 65535
* hard nofile 65535

vim /etc/sysctl.conf
vm.max_map_count=262144

启动ES服务

[root@node1 config]# su - elk
上一次登录:六 4月 10 21:11:16 CST 2021pts/0 上
[elk@node1 ~]$ !/usr
/usr/local/elasticsearch/bin/elasticsearch -d

查看ES端口

[root@node1 config]# netstat -nltp |grep -aiwE "9300|9200"
tcp6       0      0 :::9200                 :::*                    LISTEN      43179/java          
tcp6       0      0 :::9300                 :::*                    LISTEN      43179/java

 

Kibana WEB配置

配置java

[root@node1 ~]# tar -xf jdk-11.0.10_linux-x64_bin.tar.gz
[root@node1 ~]# mv jdk-11.0.10 /usr/local/jdk-11.0.10
[root@node1 ~]# echo  "export JAVA_HOME=/usr/local/jdk-11.0.10"   >> /etc/profile
[root@node1 ~]# echo  "export PATH=$JAVA_HOME/bin:$PATH "  >> /etc/profile
[root@node1 ~]# source /etc/profile

安装kibana

[root@node2 ~]# tar -xf kibana-7.11.1-linux-x86_64.tar.gz 
[root@node2 ~]# mv kibana-7.11.1-linux-x86_64 /usr/local/kibana
[root@node2 ~]# cd /usr/local/kibana/config/
[root@node2 config]# ls
kibana.yml  node.options

配置Kibana

##修改配置文件 
[root@node2 config]# vim kibana.yml 
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.204.140:9200"]  #地址为ES地址
#配置中文(文件尾部添加)
i18n.locale: "zh-CN"

启动Kibana

[root@node2 config]# nohup sh ../bin/kibana --allow-root &

#查看5601端口
[root@node2 config]# netstat -nltp |grep -aiwE 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 42367/../bin/../nod 

[root@node2 config]# ps -ef |grep node
root 42367 41939 5 21:31 pts/0 00:00:20 ../bin/../node/bin/node ../bin/../src/cli/dist --allow-root
root 42434 41939 0 21:37 pts/0 00:00:00 grep --color=auto node

访问Kibana webELK分布式部署

logstash配置

安装java

[root@node1 ~]# tar -xf jdk-11.0.10_linux-x64_bin.tar.gz
[root@node1 ~]# mv jdk-11.0.10 /usr/local/jdk-11.0.10
[root@node1 ~]# echo  "export JAVA_HOME=/usr/local/jdk-11.0.10"   >> /etc/profile
[root@node1 ~]# echo  "export PATH=$JAVA_HOME/bin:$PATH "  >> /etc/profile
[root@node1 ~]# source /etc/profile

安装logstash

[root@node3 ~]# tar -xf logstash-7.11.1-linux-x86_64.tar.gz 
[root@node3 ~]# mv logstash-7.11.1 /usr/local/logstash

收集ngixn日志

[root@node3 ~]# cd /usr/local/logstash/config/
[root@node3 config]# ls
jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options
[root@node3 config]# vim nginx.conf

input {
   file {
       type => "nginx-access"
       path => "/usr/local/nginx/logs/access.log"
   }
}
output {
 elasticsearch {
 hosts => "192.168.204.140:9200" }

启动logstash

[root@node3 config]# ../bin/logstash -f nginx.conf

创建索引

ELK分布式部署

ELK分布式部署

ELK分布式部署

ELK分布式部署

ELK分布式部署

 

访问NGINX WEB再去看页面有没有访问记录

ELK分布式部署

继续阅读
运维最后更新:2021-4-25
scadm
  • 本文由 发表于 2021年4月10日19:37:13
  • 转载请务必保留本文链接:https://www.wscyun.com/207
sameersbn/gitlab9.3.5升级gitlab-ce14.3.4 运维

sameersbn/gitlab9.3.5升级gitlab-ce14.3.4

因为之前没有接触过gitlab,现在有一台版本比较低的sameersbn/gitlab9.3.5需要升级,由于网上可查的资料有限,所以我打算把sameersbn/gitlab迁移为gitlab-ce1...