ELK简易Nginx日志系统搭建:ElasticSearch+Kibana+Filebeat

该教程以 ubuntu 16.04 为例, 其他系统类似

官网: https://www.elastic.co

使用ElasticSearch 和 Kibana, 日志输入采用Filebeat, 没有使用Logstash

参考: https://github.com/shildondu/elkstack_filebeat

准备阶段

1. 一台比较干净的可联外网的 ubuntu 16.04 系统的服务器, 并具备root权限
2. 安装 JDK 1.8 和 supervisor和nginx

# 安装JDK和supervisor和nginx
sudo apt-get update && sudo apt-get install openjdk-8-jdk supervisor nginx
# 启动nginx
nginx -t && nginx
# 测试nginx
curl localhost

创建用户与下载安装包

# 创建用户elk
sudo useradd -d /home/elk/ -m elk
su elk
cd /home/elk/
# 下载安装包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-amd64.deb
# 解压安装包
tar -xzvf elasticsearch-6.2.3.tar.gz
tar -xzvf kibana-6.2.3-linux-x86_64.tar.gz
# 安装filebeat
dpkg -i filebeat-6.2.3-amd64.deb

安装 elasticsearch

搜索引擎

su elk
cd /home/elk/elasticsearch-6.2.3/
bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent

# 配置全局访问, 不限制访问者ip
echo "network.host: 0.0.0.0" >> config/elasticsearch.yml
# 启动elasticsearch
bin/elasticsearch -d -p pid.txt
# 测试elasticsearch启动
curl localhost:9200

kibana

用于搜索以及展示数据

su elk
cd /home/elk/kibana-6.2.3-linux-x86_64/

# 修改配置文件,让kibana支持外网访问
echo "server.host: 0.0.0.0" >> config/kibana.yml

bin/kibana

filebeat

监控日志文件, 并输出的ElasticSearch

详见 https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html

# 启用filebeat的nginx模块(提前安装好nginx)
filebeat modules enable nginx

# nginx的模块配置
# vim /etc/filebeat/modules.d/nginx.yml

# filebeat基本配置, 配置ElasticSearch和Kibana
# vim /etc/filebeat/filebeat.yml
output.elasticsearch:
   hosts: ["xxx.xxx.xxx.xxx:9200"]

# 设置filebeat
filebeat setup -e
# 启动filebeat
service filebeat start

配置 supervisor

让服务可以后台运行

[program:elasticsearch]
directory=/home/elk/elasticsearch-6.2.3/
command=/home/elk/elasticsearch-6.2.3/bin/elasticsearch
autorestart=true
user=elk
stdout_logfile=NONE


[program:kibana]
directory=/home/elk/kibana-6.2.3-linux-x86_64/
command=/home/elk/kibana-6.2.3-linux-x86_64/bin/kibana
autorestart=true
user=elk
stdout_logfile=NONE

结果展示

拓展其他的日志

配置Java工程

1. 添加依赖compile('net.logstash.logback:logstash-logback-encoder:4.11')
2. 修改logback-spring.xml配置

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

<!-- 日志路径 -->
<property name="logDir" value="./logs"/>

<!-- 输出器:只打印WARN等级及以上的日志到文件 -->
<appender name="JSON_ERROR_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <File>${logDir}/json_error.log</File>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <FileNamePattern>${logDir}/json_error-%d{yyyy-MM-dd}-%i.log.gz</FileNamePattern>
        <MaxHistory>2</MaxHistory>
        <TimeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
            <MaxFileSize>100MB</MaxFileSize>
        </TimeBasedFileNamingAndTriggeringPolicy>
    </rollingPolicy>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>WARN</level>
    </filter>
</appender>

<root level="INFO">
     <appender-ref ref="JSON_ERROR_LOG"/>
</root>

</configuration>

根据条件输出到不同的index

/etc/filebeat/filebeat.yml

filebeat.prospectors:

- type: log
  paths:
      - /my_app/server/logs/json_error.log
  json.keys_under_root: true
  json.add_error_key: true
  fields.log_type: "server-error"

- type: log
  paths:
      - /my_app/shangyuer/server/logs/json.log
  json.keys_under_root: true
  json.add_error_key: true
  fields.log_type: "server-log"

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  hosts: ["xxx.xxx.xxx.xxx:9200"]
  indices:
    - index: "nginx-access-%{+yyyy.MM.dd}"
      when.equals:
        fileset.module: "nginx"
        fileset.name: "access"
    - index: "nginx-error-%{+yyyy.MM.dd}"
      when.equals:
        fileset.module: "nginx"
        fileset.name: "error"
    - index: "server-error-%{+yyyy.MM.dd}"
      when.equals:
        fields.log_type: "server-error"
    - index: "server-log-%{+yyyy.MM.dd}"
      when.equals:
        fields.log_type: "server-log"

setup.kibana:
  host: "xxx.xxx.xxx.xxx:5601"

#setup.template.name: "nginx-access"
#setup.template.pattern: "nginx-access-*"
#setup.dashboards.index: "nginx-access-*"
#setup.template.overwrite: true
setup.dashboards.enabled: false
setup.template.enabled: false

赏

谢谢你请我吃糖果

支付宝

微信