Category Archives: Tools

工具

领域专用语言(DSL)和通用编程语言(GPL)

  By | |
领域专用语言(DSL)和通用编程语言(GPL)已关闭评论

DSL解释
领域专用语言（Domain Specific Language/DSL）
DSL 通过在表达能力上做的妥协换取在某一领域内的高效(世界级软件开发大师 Martin Fowler 对于DSL的解释)

DSL相对应的GPL
通用编程语言（General Purpose Language/GPL）

通用编程语言指被设计为各种应用领域服务的编程语言。通常通用编程语言不含有为特定应用领域设计的结构。就是我们非常熟悉的 Objective-C、Java、Python 以及 C 语言等等

Continue reading →

logstash插件推荐

  By | |
logstash插件推荐已关闭评论

1、kafka
参考：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
2、hdfs
参考：https://www.elastic.co/guide/en/logstash/5.4/plugins-outputs-webhdfs.html
3、zabbix
参考：https://www.elastic.co/guide/en/logstash/5.4/plugins-outputs-zabbix.html

input插件
https://www.elastic.co/guide/en/logstash/5.4/input-plugins.html
output插件
https://www.elastic.co/guide/en/logstash/5.4/output-plugins.html

logstash配置output到Elasticsearch

  By | | , ,
logstash配置output到Elasticsearch已关闭评论 
input {
	file {
		path => ["/data/logs/nginx/201/*.log"]
		type => "test-log"
		start_position => "beginning"
        codec => "json"
	}
}
output {
	elasticsearch {
        hosts => "192.168.56.201:9200"
        index => "nginx-201"
	}
}

更多参考：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html

logstash配置codec插件-多行模式

  By | |
logstash配置codec插件-多行模式已关闭评论 
用途
应用日志多行打印
配置logstash
input {
    file {
        path => ["/data/test/test/test.log"]
        type => "demo-codec-multiline-log"
        start_position => "beginning"
        codec => multiline {
            pattern => "^["
            negate => true
            what => "previous"
        }
    }
}
output {
    stdout{
        codec=>rubydebug
    }
}
备注：
what 只能是previous或者next，previous指定行匹配pattern选项的内容是上一行的一部分，next指定行匹配pattern选项的内容是下一行的一部分
启动
bin/logstash -f /etc/logstash/conf.d/demo-codec-multiline.conf
结果
{
          "path" => "/data/test/test/test.log",
    "@timestamp" => 2017-06-13T07:09:16.452Z,
      "@version" => "1",
          "host" => "192-168-56-201",
       "message" => "[info] test 4\ntest 5\ntest 6",
          "type" => "demo-codec-multiline-log",
          "tags" => [
        [0] "multiline"
    ]
}
{
          "path" => "/data/test/test/test.log",
    "@timestamp" => 2017-06-13T07:09:40.516Z,
      "@version" => "1",
          "host" => "192-168-56-201",
       "message" => "[error]test 6\ntest 7",
          "type" => "demo-codec-multiline-log",
          "tags" => [
        [0] "multiline"
    ]
}

logstash配置codec插件-JSON模式

  By | |
logstash配置codec插件-JSON模式已关闭评论 
配置nginx日志
log_format json '{"remote_addr":"$remote_addr" ,"host":"$host" ,"server_addr":"$server_addr" ,"timestamp":"$time_iso8601" ,"request_time":$request_time, "remote_user":"$remote_user",  "request":"$request" ,"status":$status, "body_sent":$body_bytes_sent ,"http_referer":"$http_referer" ,"http_user_agent":"$http_user_agent" ,"http_x_forwarded_for":"$http_x_forwarded_for"}';
配置logstash
input {
	file {
		path => ["/data/logs/nginx/collectd.dev-access.log"]
		type => "demo-codec-json-log"
		start_position => "beginning"
        codec => "json"
	}
}
output {
	stdout{
		codec=>rubydebug
	}
}
启动
bin/logstash -f /etc/logstash/conf.d/demo-codec-json.conf
结果
{
             "remote_addr" => "192.168.56.1",
                 "request" => "GET /graph.php?p=load&t=load&h=192.168.56.201&s=86400 HTTP/1.1",
                    "type" => "demo-codec-json-log",
             "server_addr" => "192.168.56.201",
         "http_user_agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.110 Safari/537.36",
             "remote_user" => "-",
                    "path" => "/data/logs/nginx/collectd.dev-access.log",
            "request_time" => 0.026,
              "@timestamp" => 2017-06-13T06:31:12.761Z,
            "http_referer" => "http://collectd.dev/host.php?h=192.168.56.201&p=load",
                    "host" => "collectd.dev",
    "http_x_forwarded_for" => "-",
                "@version" => "1",
               "body_sent" => 13863,
               "timestamp" => "2017-06-13T06:31:12+00:00",
                  "status" => 200
}
备注
nginx日志当中部分字段可能会是数字或者-，可以将日志全部转换为字符串，然后通过filter来处理

logstash之input配置collectd类型详解

  By | |
logstash之input配置collectd类型详解已关闭评论 
配置
input {
    udp {
        port => 12000
        codec => collectd {}
        type => "collectd-demo"
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

启动
bin/logstash -f /etc/logstash/conf.d/demo-input-collectd.conf
结果
{
         "@timestamp" => 2017-06-13T03:20:19.620Z,
    "plugin_instance" => "root",
      "type_instance" => "free",
             "plugin" => "df",
               "host" => "192.168.56.201",
           "@version" => "1",
      "collectd_type" => "df_complex",
               "type" => "collectd-demo",
              "value" => 5521645568.0
}
{
       "@timestamp" => 2017-06-13T03:20:19.620Z,
           "plugin" => "entropy",
             "host" => "192.168.56.201",
         "@version" => "1",
    "collectd_type" => "entropy",
             "type" => "collectd-demo",
            "value" => 844.0
}