容器特性给日志采集带来的困难

• K8s弹性伸缩性：导致不能预先确定采集的目标
• 容器隔离性：容器的文件系统与宿主机是隔离，导致日志采集器读取日志文件受阻。

日志按体现方式分类

应用程序日志记录体现方式分为两类：
• 标准输出：输出到控制台，使用kubectl logs可以看到。
例如 nginx日志是将访问日志输出到标准输出，可以用kubectl log 查看

kubectl  logs ==>> apiserver ==>> kubecet ==>> docker api ==>><container-id>-json.log  

• 日志文件：写到容器的文件系统的文件。

Kubernetes应用日志收集

针对标准输出：以DaemonSet方式在每个Node上部署一个日志收集程序，采集
/var/lib/docker/containers/目录下所有容器日志。

针对容器中日志文件：在Pod中增加一个容器运行日志采集器，使用emptyDir共享日志目录让日志采集器读取到日志文件。

ELK 日志系统

ELK 是三个开源软件的缩写，提供一套完整的企业级日志平台解决方案。
分别是：
• Elasticsearch：搜索、分析和存储数据
• Logstash ：采集日志、格式化、过滤，最后将数据
推送到Elasticsearch存储
• Kibana：数据可视化
• Beats ：集合了多种单一用途数据采集器，用于实现从边缘机器向 Logstash 和 Elasticsearch 发送数
据。里面应用最多的是Filebeat，是一个轻量级日志采集器。

轻量级日志：graylog、grafana loki

搭建日志系统：
• elasticsearch.yaml # ES数据库
• kibana.yaml # 可视化展示
日志收集：
• filebeat-kubernetes.yaml # 采集所有容器标准输出
• app-log-stdout.yaml # 标准输出测试应用
• app-log-logfile.yaml # 日志文件测试应用

针对标准输出：在每个节以deamsent方式部署filebeat，采集节点上所有的日志文件，目录为/var/lib/docker/containers//容器id/xxxx-json.log

kubectl apply -f elasticsearch.yaml 
kubectl apply -f kibana.yaml 
kubectl apply -f filebeat-kubernetes.ya

 #elasticsearch.yamlapiVersion: apps/v1
kind: Deployment
metadata:name: elasticsearchnamespace: opslabels:k8s-app: elasticsearch
spec:replicas: 1selector:matchLabels:k8s-app: elasticsearchtemplate:metadata:labels:k8s-app: elasticsearchspec:containers:- image: elasticsearch:7.9.2name: elasticsearchresources:limits:cpu: 2memory: 3Girequests:cpu: 0.5memory: 500Mienv:- name: "discovery.type"value: "single-node"- name: ES_JAVA_OPTSvalue: "-Xms512m -Xmx2g"ports:- containerPort: 9200name: dbprotocol: TCPvolumeMounts:- name: elasticsearch-datamountPath: /usr/share/elasticsearch/datavolumes:- name: elasticsearch-datapersistentVolumeClaim:claimName: es-pvc---apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: es-pvcnamespace: ops
spec:storageClassName: "managed-nfs-storage"accessModes:- ReadWriteManyresources:requests:storage: 10Gi---apiVersion: v1
kind: Service
metadata:name: elasticsearchnamespace: ops
spec:ports:- port: 9200protocol: TCPtargetPort: 9200selector:k8s-app: elasticsearch#kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: kibananamespace: opslabels:k8s-app: kibana
spec:replicas: 1selector:matchLabels:k8s-app: kibanatemplate:metadata:labels:k8s-app: kibanaspec:containers:- name: kibanaimage: kibana:7.9.2resources:limits:cpu: 2memory: 2Girequests:cpu: 0.5memory: 500Mienv:- name: ELASTICSEARCH_HOSTSvalue: http://10.244.169.139:9200- name: I18N_LOCALEvalue: zh-CNports:- containerPort: 5601name: uiprotocol: TCP---
apiVersion: v1
kind: Service
metadata:name: kibananamespace: ops
spec:type: NodePortports:- port: 5601protocol: TCPtargetPort: uinodePort: 30601selector:k8s-app: kibana#filebeat-kubernetes.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-confignamespace: opslabels:k8s-app: filebeat
data:filebeat.yml: |-filebeat.config:inputs:# Mounted `filebeat-inputs` configmap:path: ${path.config}/inputs.d/*.yml# Reload inputs configs as they change:reload.enabled: falsemodules:path: ${path.config}/modules.d/*.yml# Reload module configs as they change:reload.enabled: falseoutput.elasticsearch:hosts: ['elasticsearch.ops:9200']
---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-inputsnamespace: opslabels:k8s-app: filebeat
data:kubernetes.yml: |-- type: dockercontainers.ids:- "*"processors:- add_kubernetes_metadata:in_cluster: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: filebeatnamespace: opslabels:k8s-app: filebeat
spec:selector:matchLabels:k8s-app: filebeattemplate:metadata:labels:k8s-app: filebeatspec:serviceAccountName: filebeatterminationGracePeriodSeconds: 30containers:- name: filebeatimage: elastic/filebeat:7.9.2args: ["-c", "/etc/filebeat.yml","-e",]securityContext:runAsUser: 0# If using Red Hat OpenShift uncomment this:#privileged: trueresources:limits:memory: 200Mirequests:cpu: 100mmemory: 100MivolumeMounts:- name: configmountPath: /etc/filebeat.ymlreadOnly: truesubPath: filebeat.yml- name: inputsmountPath: /usr/share/filebeat/inputs.dreadOnly: true- name: datamountPath: /usr/share/filebeat/data- name: varlibdockercontainersmountPath: /var/lib/docker/containersreadOnly: truevolumes:- name: configconfigMap:defaultMode: 0600name: filebeat-config- name: varlibdockercontainershostPath:path: /var/lib/docker/containers- name: inputsconfigMap:defaultMode: 0600name: filebeat-inputs# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart- name: datahostPath:path: /var/lib/filebeat-datatype: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: filebeat
subjects:
- kind: ServiceAccountname: filebeatnamespace: ops
roleRef:kind: ClusterRolename: filebeatapiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: filebeatlabels:k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API groupresources:- namespaces- podsverbs:- get- watch- list
---
apiVersion: v1
kind: ServiceAccount
metadata:name: filebeatnamespace: opslabels:k8s-app: filebeat

可视化展示日志：

1. 查看索引（日志记录集合）：Management -> Stack Management -> 索引管理
2. 将索引关联到Kibana：索引模式 -> 创建 -> 匹配模式 -> 选择时间戳
3. 在Discover选择索引模式查看日志

将索引关联到Kibana
Discover选择索引模式查看日志

按条件查询

二、针对容器中日志文件

#app-log-logfile.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: app-log-logfile
spec:replicas: 3selector:matchLabels:project: microserviceapp: nginx-logfiletemplate:metadata:labels:project: microserviceapp: nginx-logfilespec:containers:# 应用容器- name: nginximage: lizhenliang/nginx-php# 将数据卷挂载到日志目录volumeMounts:- name: nginx-logsmountPath: /usr/local/nginx/logs# 日志采集器容器- name: filebeatimage: elastic/filebeat:7.9.2args: ["-c", "/etc/filebeat.yml","-e",]resources:requests:cpu: 100mmemory: 100Milimits:memory: 500MisecurityContext:runAsUser: 0volumeMounts:# 挂载filebeat配置文件- name: filebeat-configmountPath: /etc/filebeat.ymlsubPath: filebeat.yml# 将数据卷挂载到日志目录- name: nginx-logsmountPath: /usr/local/nginx/logs# 数据卷共享日志目录volumes:- name: nginx-logsemptyDir: {}- name: filebeat-configconfigMap:name: filebeat-nginx-config
---
apiVersion: v1
kind: Service
metadata:name: app-log-logfile
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:project: microserviceapp: nginx-logfile
---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-nginx-configdata:# 配置文件保存在ConfigMapfilebeat.yml: |-filebeat.inputs:- type: logpaths:- /usr/local/nginx/logs/access.log# tags: ["access"]fields_under_root: truefields:project: microservice #项目名称app: nginx            #pod名称setup.ilm.enabled: falsesetup.template.name: "nginx-access"setup.template.pattern: "nginx-access-*"output.elasticsearch:hosts: ['elasticsearch.ops:9200']index: "nginx-access-%{+yyyy.MM.dd}" #索引名称

kubectl apply -f app-log-logfile.yaml 

kibana建立索引，查看日志