logstash获取nginx日志的配置

nginx部分配置直接用json，省去很多麻烦

   log_format json '{"@timestamp":"$time_iso8601",'
                     '"server_addr":"$server_addr",'
                     '"remote_addr":"$remote_addr",'
                     '"http_x_forwarded_for":"$http_x_forwarded_for",'
                     '"body_bytes_sent":$body_bytes_sent,'
                     '"request_uri":"$request_uri",'
                     '"request_method":"$request_method",'
                     '"server_protocol":"$server_protocol",'
                     '"scheme":"$scheme",'
                     '"request_time":$request_time,'
                     '"upstream_response_time":"$upstream_response_time",'
                     '"upstream_addr":"$upstream_addr",'
                     '"host":"$host",'
                     '"uri":"$uri",'
                     '"http_referer":"$http_referer",'
                     '"http_user_agent":"$http_user_agent",'
                     '"status":$status}';

filebeat前台启动命令 filebeat -e -c filebeat.yml -d "publish"

filebeat配置部分：

filebeat.inputs:
- type: log
  enabled: true

  paths:
    - /data/wwwlogs/www.myzabbix.com_access.log
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: ["192.168.80.11:5041"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logstash前台启动命令 /usr/share/logstash/bin/logstash -f 文件名

logstash配置部分：

input {
    beats {
        port => 5041  #配置文件输入的端口号。
        #codec => json
    } 
}
filter {
    #if [type] == "log" {
        mutate {
            gsub => ["message", "\\x", "\\\x"]
        }
        
        json {
            source => "message"
        }
        
        mutate {
            remove_field => [ "message" ]
        }
        mutate {
            remove_field => [ "ecs" ]
        }
        mutate {
            remove_field => [ "agent" ]
        }
        mutate {
            remove_field => [ "@version" ]
        }
    
        if "HEAD" in [request_method] {
            drop {}
        }

        useragent {
            source => "http_user_agent"
            target => "ua"
        }


        if "-" in [upstream_response_time] {
            mutate {
                replace => {
                    "upstream_response_time" => "0"
                }
            }
        }

        mutate {
            convert => ["upstream_response_time","float"]
        }
        mutate {
            convert => ["status", "integer"]
        }

        geoip {
            source => "remote_addr"
            database => "/etc/logstash/GeoLite2-City.mmdb"
            target => "geoip"
        }
    #}
}

output {
    #if [status] > 300 {
    #    exec {
    #        command => "/usr/bin/echo '网页url是%{request_uri}'"
    #    }
    #}else{
    #    exec {
    #        command => "/usr/bin/echo '网页状态码是%{status}'"
    #    }
    #}
    #stdout {
    #    codec => rubydebug
    #}

   elasticsearch{

              hosts => ["http://192.168.80.11:9200"]

              index => "zabbixlog-%{+YYYY.MM.dd}"

              #document_type => "sparkfileType"

   }

}

注释部分可以打开调试，codec => rubydebug代表输出到界面，还可以输出到file，if else注释部分可以判断页面 url状态码，如果有问题调用外部命令发送报警通知。也可以一段时间内达到N次错误发送报警通知，具体根据业务来调试。