> 文章列表 > filebeat收集不规则多行日志

filebeat收集不规则多行日志

filebeat收集不规则多行日志

现环境有多行日志输出内容和格式不确定,合并后使用grok默认正则无法收集,需要自己编写正则
日志内容如下:

ERROR|2023-04-06 14:27:52|helper|test|http|/api/ad/listBanner|1d60fff861bqwe4b0397be554141eb 127.0.0.1|1b4429-5adb-44d4-acfe-0tee65eec19d|gw|Error: Class "HttpEncodingException" not found in /var/www/html/helper/59_20230406135132/app/Api/Controller/AdController.php:22
Stack trace:
#0 /var/www/html/helper/59_20230406135132/vendor/workerman/webman-framework/src/App.php(387): App\\Api\\Controller\\AdController->listBanner()
#1 /var/www/html/helper/59_20230406135132/vendor/workerman/webman-framework/src/App.php(348): Webman\\App::Webman\\{closure}()
#2 /var/www/html/helper/59_20230406135132/common/Library/LogAccess/LogAccessMiddleware.php(40): Webman\\App::Webman\\{closure}()
#3 /var/www/html/helper/59_20230406135132/vendor/workerman/webman-framework/src/App.php(340): Common\\Library\\LogAccess\\LogAccessMiddleware->process()
#4 /var/www/html/helper/59_20230406135132/vendor/workerman/webman-framework/src/App.php(167): Webman\\App::Webman\\{closure}()
#5 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Connection/TcpConnection.php(646): Webman\\App->onMessage()
#6 [internal function]: Workerman\\Connection\\TcpConnection->baseRead()
#7 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Events/Event.php(193): EventBase->loop()
#8 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Worker.php(1629): Workerman\\Events\\Event->loop()
#9 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Worker.php(1423): Workerman\\Worker::forkOneWorkerForLinux()
#10 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Worker.php(1397): Workerman\\Worker::forkWorkersForLinux()
#11 /var/www/html/helper/59_20230406135132/vendor/workerman/workerman/Worker.php(560): Workerman\\Worker::forkWorkers()
#12 /var/www/html/helper/59_20230406135132/vendor/workerman/webman-framework/src/support/App.php(131): Workerman\\Worker::runAll()
#13 /var/www/html/helper/59_20230406135132/start.php(4): support\\App::run()
#14 {main}|[]

原先pipeline中grok的写法如下,并且已经在filebeat.yml将日志合并为单个事件,也无法在message中使用官方提供的GREEDYDATA匹配到多行日志,个人认为是日志中有大量的反斜杠'\\'造成,反斜杠可能会被转义导致报错丢掉该条日志

        "grok" : {"field" : "message","patterns" : ["""%{DATA:level}\\|%{DATA:logdate}\\|%{DATA:app}\\|%{DATA:env}\\|%{DATA:type}\\|%{DATA:site}\\|%{DATA:device_id}\\|%{DATA:request_id}\\|%{DATA:from}\\|%{GREEDYDATA:message}\\|%{GREEDYDATA:context}"""]}

最后将message字段修改成如下内容可以进行正确匹配

%{DATA:level}\\|%{DATA:logdate}\\|%{DATA:app}\\|%{DATA:env}\\|%{DATA:type}\\|%{DATA:site}\\|%{DATA:device_id}\\|%{DATA:request_id}\\|%{DATA:from}\\|(?<message>[^\\|]+)\\|%{GREEDYDATA:context}