ELK日志收集告警

elastic stack

• elastic search 日志持久化
• filebeats 日志收集
• kibana 日志展示
• elaticalert 日志告警 elastalert官网
• Elastic Observability APM 指标监控 java-agent

1. 基于logback根据level进行日志的切分聚合
2. 宿主机上安装filebeats
   1. 配置inputs插件
   2. 配置采集路径
   3. 配置多行匹配规则
   4. 配置tags
   5. 配置Output插件
      1. 选择output到Logstash或者直接到es
      2. 配置es索引模板规则
      3. 配置索引分片 副本规则
   6. 可选配置processor 时间戳timestamp格式化
3. kibana配置Index Pattern 进行索引匹配 可视化展示
4. 配置elastalert
   1. 安装python 3.11以上版本 或者使用anaconda
   2. 修改config.yml 指定es host username/password
   3. 修改config 中run_every 采集时间 buffer_time 缓冲时间 rules告警规则目录等
   4. 编写rule.yaml告警规则
      1. 选择一个合适的alert type
      2. 配置filter 参考es DSL
      3. error可以通过filebeat中配置tags进行匹配
      4. 配置采集恢复时间
   5. 选择一个合适的告警通道 原生支持webhook dingtalk jira等如需拓展参考官方文档实现python代码
   6. 启动elast alert
5. APM监控 可选
   1. es stack 自带的APM监控通过java agent的形式 在中央仓库下载指定的jar包 启动参考官方文档
   2. 自定义指标监控 早期基于spring acturator

参考配置

filebeats.yml

filebeat.inputs:
- type: logenabled: truepaths:- C:\\Users\\JimWu\\Desktop\\test_log\\info/*.logmultiline.pattern: '^20'                    #多行匹配规则multiline.negate: true                    #将不匹配的规则的行合并在一起multiline.match: after                #合并到匹配规则的上一行末尾tags: ["demo","info"]- type: logenabled: truepaths:- C:\\Users\\JimWu\\Desktop\\test_log\\*-error-*.logmultiline.pattern: '^20'                    #多行匹配规则multiline.negate: true                    #将不匹配的规则的行合并在一起multiline.match: after                #合并到匹配规则的上一行末尾tags: ["demo","error"]output.elasticsearch:hosts: ["localhost:9200"]username: "elastic"password: "elastic"indices:- index: "demo-%{+yyyy.MM.dd}"when.contains:tags: "demo"setup.ilm.enable: false
setup.template.name: "demo-log"
setup.template.pattern: "demo-dev-*"
setup.template.overwrite: false
setup.template.settings:index.number_of_shards: 1index.number_of_replicas: 1processors:- script:lang: javascriptid: my_filtertag: enablesource: >function process(event) {var str= event.Get("message");var reg = /\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}/;var time = str.match(reg)[0];event.Put("log_time",time);}- timestamp:field: log_timetimezone: Asia/Shanghailayouts:- '2006-01-02 15:04:05'- '2006-01-02 15:04:05.999'test:- '2019-06-22 16:33:51'

APM

java -javaagent:/path/to/elastic-apm-agent-<version>.jar \\
-Delastic.apm.service_name=my-application \\
-Delastic.apm.server_urls=http://localhost:8200 \\
-Delastic.apm.secret_token= \\
-Delastic.apm.environment=production \\
-Delastic.apm.application_packages=org.example \\
-jar my-application.jar

elastalert rule参考

# Alert when the rate of events exceeds a threshold# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com# (Optional)
# Elasticsearch port
# es_port: 14900# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword# (Required)
# Rule name, must be unique
name: Demo frequency rule# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency# (Required)
# Index to search, wildcard supported
index: demo-*# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:minutes: 10# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:tags: "error"# (Required)
# The alert is use when a match is found
alert:
- "post"http_post_url: "http://localhost:3000/alert"