logstash grok 多项匹配

本文由 简悦 SimpRead 转码， 原文地址 blog.csdn.net

业务场景：新版本日志需要添加字段，需要兼容新旧日志匹配

版本：logstash-2.3

filter {
    grok {
         match => [
            "message" , "%{DATA:hostname}\|%{DATA:tag}\|%{DATA:types}\|%{DATA:uid}\|%{GREEDYDATA:msg}",
            "message" , "%{DATA:hostname}\|%{DATA:tag}\|%{GREEDYDATA:msg}"
         ]
        remove_field => ['type','_id','input_type','tags','message','beat','offset']
    }
}

filter {
    grok {
    match => {
        "message"=>[
        "%{DATA:hostname}\|%{DATA:tag}\|%{DATA:types}\|%{DATA:uid}\|%{GREEDYDATA:msg}",
        "%{DATA:hostname}\|%{DATA:tag}\|%{GREEDYDATA:msg}"]
        }
    }
}

太多使用 DATA 和 GREEDYDAYA 会导致性能 cpu 负载严重。建议多使用正则匹配，或者 ruby 代码块

filter {
     grok {
        match => [
               "message", "(?<hostname>[a-zA-Z0-9._-]+)\|%{DATA:tag}\|%{NUMBER:types}\|(?<uid>[0-9]+)\|%{GREEDYDATA:msg}",
               "message", "(?<hostname>[a-zA-Z0-9._-]+)\|%{DATA:tag}\|%{GREEDYDATA:msg}",
        ]
       remove_field => ['type','_id','input_type','tags','message','beat','offset']
    }
}

filter {
    ruby {
        code =>'
        arr = event["message"].split("|")
        if arr.length == 5
            event["hostname"] = arr[0]
            event["tag"] = arr[1]
            event["types"] = arr[2]
            event["uid"] = arr[3]
            event["msg"] = arr[4]
        elsif arr.length == 3
            event["hostname"] = arr[0]
            event["tag"] = arr[1]
            event["msg"] = arr[2]
        end'
       remove_field => ['type','_id','input_type','tags','message','beat','offset']
    }
}

参考文档：https://github.com/chenryn/logstash-best-practice-cn/blob/master/filter/grok.md