logstash插件推荐

1、kafka
参考:https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
2、hdfs
参考:https://www.elastic.co/guide/en/logstash/5.4/plugins-outputs-webhdfs.html
3、zabbix
参考:https://www.elastic.co/guide/en/logstash/5.4/plugins-outputs-zabbix.html

input插件
https://www.elastic.co/guide/en/logstash/5.4/input-plugins.html
output插件
https://www.elastic.co/guide/en/logstash/5.4/output-plugins.html

logstash配置codec插件-多行模式

用途
应用日志多行打印
配置logstash
input {
    file {
        path => ["/data/test/test/test.log"]
        type => "demo-codec-multiline-log"
        start_position => "beginning"
        codec => multiline {
            pattern => "^["
            negate => true
            what => "previous"
        }
    }
}
output {
    stdout{
        codec=>rubydebug
    }
}
备注:
what 只能是previous或者next,previous指定行匹配pattern选项的内容是上一行的一部分,next指定行匹配pattern选项的内容是下一行的一部分
启动
bin/logstash -f /etc/logstash/conf.d/demo-codec-multiline.conf
结果
{
          "path" => "/data/test/test/test.log",
    "@timestamp" => 2017-06-13T07:09:16.452Z,
      "@version" => "1",
          "host" => "192-168-56-201",
       "message" => "[info] test 4\ntest 5\ntest 6",
          "type" => "demo-codec-multiline-log",
          "tags" => [
        [0] "multiline"
    ]
}
{
          "path" => "/data/test/test/test.log",
    "@timestamp" => 2017-06-13T07:09:40.516Z,
      "@version" => "1",
          "host" => "192-168-56-201",
       "message" => "[error]test 6\ntest 7",
          "type" => "demo-codec-multiline-log",
          "tags" => [
        [0] "multiline"
    ]
}

logstash配置codec插件-JSON模式

配置nginx日志
log_format json '{"remote_addr":"$remote_addr" ,"host":"$host" ,"server_addr":"$server_addr" ,"timestamp":"$time_iso8601" ,"request_time":$request_time, "remote_user":"$remote_user",  "request":"$request" ,"status":$status, "body_sent":$body_bytes_sent ,"http_referer":"$http_referer" ,"http_user_agent":"$http_user_agent" ,"http_x_forwarded_for":"$http_x_forwarded_for"}';
配置logstash
input {
	file {
		path => ["/data/logs/nginx/collectd.dev-access.log"]
		type => "demo-codec-json-log"
		start_position => "beginning"
        codec => "json"
	}
}
output {
	stdout{
		codec=>rubydebug
	}
}
启动
bin/logstash -f /etc/logstash/conf.d/demo-codec-json.conf
结果
{
             "remote_addr" => "192.168.56.1",
                 "request" => "GET /graph.php?p=load&t=load&h=192.168.56.201&s=86400 HTTP/1.1",
                    "type" => "demo-codec-json-log",
             "server_addr" => "192.168.56.201",
         "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36",
             "remote_user" => "-",
                    "path" => "/data/logs/nginx/collectd.dev-access.log",
            "request_time" => 0.026,
              "@timestamp" => 2017-06-13T06:31:12.761Z,
            "http_referer" => "http://collectd.dev/host.php?h=192.168.56.201&p=load",
                    "host" => "collectd.dev",
    "http_x_forwarded_for" => "-",
                "@version" => "1",
               "body_sent" => 13863,
               "timestamp" => "2017-06-13T06:31:12+00:00",
                  "status" => 200
}
备注
nginx日志当中部分字段可能会是数字或者-,可以将日志全部转换为字符串,然后通过filter来处理

logstash之input配置redis类型详解

用途
监控redis数据
配置示例
input {
    redis {
        data_type => "list"
        key => "logstash-demo"
        host => "127.0.0.1"
        port => 6379
        threads => 5
    }
}
output {
 stdout {
 codec => rubydebug
 }
}

启动
bin/logstash -f /etc/logstash/conf.d/demo-input-redis.conf

测试
redis-cli -h 127.0.0.1
rpush logstash-demo test
结果
{
    "@timestamp" => 2017-06-12T13:55:11.689Z,
      "@version" => "1",
       "message" => "test",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}
date_type	只能是list(使用BLPOP获取消息)、channel(使用SUBSCRIBE获取消息)、pattern_channel(使用PSUBSCRIBE获取消息)

logstash之input配置syslog类型详解

用途
监控syslog,监控系统运行情况
配置示例
input {
    syslog {
        port => 5000
        type => "demo-syslog"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}


启动
bin/logstash -f /etc/logstash/conf.d/demo-input-syslog.conf

测试
telnet localhost 5000
结果
{
          "severity" => 0,
        "@timestamp" => 2017-06-12T09:41:46.655Z,
          "@version" => "1",
              "host" => "127.0.0.1",
           "message" => "heloooooooo\r\n",
              "type" => "demo-syslog",
          "priority" => 0,
          "facility" => 0,
    "severity_label" => "Emergency",
              "tags" => [
        [0] "_grokparsefailure_sysloginput"
    ],
    "facility_label" => "kernel"
}

Logstash启动测试

一、logstash启动测试
在logstash目录执行
bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
然后输入1
返回
{
    "@timestamp" => 2017-06-11T14:35:00.816Z,
      "@version" => "1",
          "host" => "ubuntu-101",
       "message" => "message"
}
二、logstash的input、filter、output详解
提醒:logstash配置文件里至少需要有input和output两个部分构成

Logstash配置详解

node.name: logstash-102 		#节点名称,一般为机器的hostname
path.data: /var/lib/logstash 	#logstash存储插件等数据目录
pipeline.workers: 2 		 	#进程数量
pipeline.output.workers: 1		#每个输出插件进程数量
pipeline.batch.size: 125 		#单个工作线程将从输入中收集到的最大事件数量
pipeline.batch.delay: 5			#分发延迟时间
pipeline.unsafe_shutdown: false #当设置为true时,即使在内存中仍然存在一些信息事件,在关闭期间也会强制退出,启用此选项可能导致关闭期间的数据丢失
path.config: /etc/logstash/conf.d #配置目录
config.string:					#用于主管道的管道配置的字符串
config.test_and_exit: false		#当设置为true时,检查配置是否有效,然后退出,不检查grok模式正确性
config.reload.automatic: false	#当设置为true时,定期检查配置是否更改,并在更改时重新加载配置
config.reload.interval: 3		#定期检查时间间隔,单位s
config.debug: false				#设置为true显示调试信息
queue.type: memory				#队列类型,可设置为内存memory或者持久化persisted
####持久化队列才生效的配置-开始
path.queue:						#队列存储路径;如果队列类型为persisted,则生效
queue.page_capacity: 250mb		#队列为持久化,单个队列大小
queue.max_events: 0				#当启用持久化队列时,队列中未读事件的最大数量,0为不限制
queue.max_bytes: 1024mb			#队列最大容量
queue.checkpoint.acks: 1024		#在启用持久队列时强制执行检查点的最大数量,0为不限制
queue.checkpoint.writes: 1024	#在启用持久队列时强制执行检查点之前的最大数量的写入事件,0为不限制
queue.checkpoint.interval: 1000	#当启用持久队列时,在头页面上强制一个检查点的时间间隔
####持久化队列才生效的配置-结束
http.host: "127.0.0.1"			#监听IP
http.port: 9600					#监听端口
log.level: info					#日志级别,默认info;fatal,error,warn,info (default),debug,trace
path.logs: /var/log/logstash	#日志路径
#path.plugins: []				#插件路径