k8s日志收集方案

三种收集方案的优缺点：

下面我们就实践第二种日志收集方案：

一、安装ELK

下面直接采用yum的方式安装ELK（源码包安装参考：https://www.cnblogs.com/Dev0ps/p/9309103.html）
1.安装jdk

  1. yum install -y java

2.添加ELK源

  1. cat << EOF >>/etc/yum.repos.d/logstash.repo
  2. [logstash-7.x]
  3. name=Elastic repository for 7.x packages
  4. baseurl=https://artifacts.elastic.co/packages/7.x/yum
  5. gpgcheck=1
  6. gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
  7. enabled=1
  8. autorefresh=1
  9. type=rpm-md
  10. EOF

3.安装ELK组件

  1. yum install elasticsearch logstash kibana -y

4.修改kibana的配置文件

  1. [root@master ~]# grep -Ev '^$|^#' /etc/kibana/kibana.yml
  2. server.port: 5601
  3. server.host: "0.0.0.0"
  4. elasticsearch.hosts: ["http://localhost:9200"]

5.启动服务

  1. systemctl start elasticsearch
  2. systemctl start kibana

二、收集k8s的日志

1.添加logstash配置文件

  1. input {
  2.   beats {
  3.      port => 5044
  4.   }
  5. }
  6.  
  7. filter {
  8. }
  9.  
  10. output {
  11.        if [app] == "k8s" {
  12.         if [type] == "module" {
  13.            elasticsearch {
  14.               hosts => ["http://127.0.0.1:9200"]
  15.               index => "k8s-log-%{+YYYY.MM.dd}"
  16.            }
  17.         }
  18.     }
  19. }

2.启动logstash

  1. /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-es.conf &

3.创建filebeat的yaml文件

  1. [root@master logs]# cat k8s-logs.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: k8s-logs-filebeat-config
  6.   namespace: kube-system
  7.  
  8. data:
  9.   filebeat.yml: |-
  10.     filebeat.prospectors:
  11.       - type: log
  12.         paths:
  13.           - /messages
  14.         fields:
  15.           app: k8s
  16.           type: module
  17.         fields_under_root: true
  18.  
  19.     output.logstash:
  20.       hosts: ['172.31.182.140:5044']
  21.  
  22. ---
  23.  
  24. apiVersion: apps/v1
  25. kind: DaemonSet
  26. metadata:
  27.   name: k8s-logs
  28.   namespace: kube-system
  29. spec:
  30.   selector:
  31.     matchLabels:
  32.       project: k8s
  33.       app: filebeat
  34.   template:
  35.     metadata:
  36.       labels:
  37.         project: k8s
  38.         app: filebeat
  39.     spec:
  40.       containers:
  41.       - name: filebeat
  42.         image: docker.elastic.co/beats/filebeat:6.4.2
  43.         args: [
  44.           "-c", "/etc/filebeat.yml",
  45.           "-e",
  46.         ]
  47.         resources:
  48.           requests:
  49.             cpu: 100m
  50.             memory: 100Mi
  51.           limits:
  52.             cpu: 500m
  53.             memory: 500Mi
  54.         securityContext:
  55.           runAsUser: 0
  56.         volumeMounts:
  57.         - name: filebeat-config
  58.           mountPath: /etc/filebeat.yml
  59.           subPath: filebeat.yml
  60.         - name: k8s-logs
  61.           mountPath: /messages
  62.       volumes:
  63.       - name: k8s-logs
  64.         hostPath:
  65.           path: /var/log/messages
  66.           type: File
  67.       - name: filebeat-config
  68.         configMap:
  69.           name: k8s-logs-filebeat-config

4.创建pod

  1. kubectl apply -f k8s-logs.yaml

5.kibana添加索引

三、收集nginx日志

1.创建nginx-deployment
默认的日志路径为：/var/log/nginx/

  1. [root@master logs]# cat nginx-deployment.yaml
  2. apiVersion: apps/v1beta1
  3. kind: Deployment
  4. metadata:
  5.   name: nginx-demo
  6. #  namespace: test
  7. spec:
  8.   replicas: 3
  9.   selector:
  10.     matchLabels:
  11.       project: www
  12.       app: www
  13.   template:
  14.     metadata:
  15.       labels:
  16.         project: www
  17.         app: www
  18.     spec:
  19.       imagePullSecrets:
  20.       - name: registry-pull-secret
  21.       containers:
  22.       - name: nginx
  23.         image: nginx:latest
  24.         imagePullPolicy: Always
  25.         ports:
  26.         - containerPort: 80
  27.           name: web
  28.           protocol: TCP
  29.         resources:
  30.           requests:
  31.             cpu: 0.5
  32.             memory: 256Mi
  33.           limits:
  34.             cpu: 1
  35.             memory: 1Gi
  36.         resources:
  37.           requests:
  38.             cpu: 0.5
  39.             memory: 256Mi
  40.           limits:
  41.             cpu: 1
  42.             memory: 1Gi
  43.         livenessProbe:
  44.           httpGet:
  45.             path: /index.html
  46.             port: 80
  47.           initialDelaySeconds: 6
  48.           timeoutSeconds: 20
  49.         volumeMounts:
  50.         - name: nginx-logs
  51.           mountPath: /var/log/nginx/
  52.  
  53.       - name: filebeat
  54.         image: docker.elastic.co/beats/filebeat:6.4.2
  55.         args: [
  56.           "-c", "/etc/filebeat.yml",
  57.           "-e",
  58.         ]
  59.         resources:
  60.           limits:
  61.             memory: 500Mi
  62.           requests:
  63.             cpu: 100m
  64.             memory: 100Mi
  65.         securityContext:
  66.           runAsUser: 0
  67.         volumeMounts:
  68.         - name: filebeat-config
  69.           mountPath: /etc/filebeat.yml
  70.           subPath: filebeat.yml
  71.         - name: nginx-logs
  72.           mountPath: /var/log/nginx/
  73.  
  74.       volumes:
  75.       - name: nginx-logs
  76.         emptyDir: {}
  77.       - name: filebeat-config
  78.         configMap:
  79.           name: filebeat-nginx-config

2.创建filebest收集日志

  1. [root@master logs]# cat filebeat-nginx-configmap.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: filebeat-nginx-config
  6. #  namespace: test
  7.  
  8. data:
  9.   filebeat.yml: |-
  10.     filebeat.prospectors:
  11.       - type: log
  12.         paths:
  13.           - /var/log/nginx/access.log
  14.         # tags: ["access"]
  15.         fields:
  16.           app: www
  17.           type: nginx-access
  18.         fields_under_root: true
  19.  
  20.       - type: log
  21.         paths:
  22.           - /var/log/nginx/error.log
  23.         # tags: ["error"]
  24.         fields:
  25.           app: www
  26.           type: nginx-error
  27.         fields_under_root: true
  28.  
  29.     output.logstash:
  30.       hosts: ['172.31.182.140:5044']

3.添加logstas配置

  1. [root@master logs]# cat /etc/logstash/conf.d/logstash-to-es.conf
  2. input {
  3.   beats {
  4.      port => 5044
  5.   }
  6. }
  7.  
  8. filter {
  9. }
  10.  
  11. output {
  12.     if [app] == "www" {
  13.         if [type] == "nginx-access" {
  14.            elasticsearch {
  15.               hosts => ["http://127.0.0.1:9200"]
  16.               index => "nginx-access-%{+YYYY.MM.dd}"
  17.            }
  18.         }
  19.         else if [type] == "nginx-error" {
  20.            elasticsearch {
  21.               hosts => ["http://127.0.0.1:9200"]
  22.               index => "nginx-error-%{+YYYY.MM.dd}"
  23.            }
  24.         }
  25.     } else if [app] == "k8s" {
  26.         if [type] == "module" {
  27.            elasticsearch {
  28.               hosts => ["http://127.0.0.1:9200"]
  29.               index => "k8s-log-%{+YYYY.MM.dd}"
  30.            }
  31.         }
  32.     }
  33. #  stdout { codec=> rubydebug }
  34. }

4.kibana添加索引

四、收集tomcat日志

1.1.创建tomcat-deployment
默认的日志路径为：/usr/local/tomcat/logs

  1. [root@master logs]# cat tomcat-deployment.yaml
  2. apiVersion: apps/v1beta1
  3. kind: Deployment
  4. metadata:
  5.   name: tomcat-java-demo
  6. #  namespace: test
  7. spec:
  8.   replicas: 3
  9.   selector:
  10.     matchLabels:
  11.       project: www
  12.       app: www
  13.   template:
  14.     metadata:
  15.       labels:
  16.         project: www
  17.         app: www
  18.     spec:
  19.       imagePullSecrets:
  20.       - name: registry-pull-secret
  21.       containers:
  22.       - name: tomcat
  23.         image: tomcat:latest
  24.         imagePullPolicy: Always
  25.         ports:
  26.         - containerPort: 8080
  27.           name: web
  28.           protocol: TCP
  29.         resources:
  30.           requests:
  31.             cpu: 0.5
  32.             memory: 1Gi
  33.           limits:
  34.             cpu: 1
  35.             memory: 2Gi
  36.         livenessProbe:
  37.           httpGet:
  38.             path: /
  39.             port: 8080
  40.           initialDelaySeconds: 60
  41.           timeoutSeconds: 20
  42.         readinessProbe:
  43.           httpGet:
  44.             path: /
  45.             port: 8080
  46.           initialDelaySeconds: 60
  47.           timeoutSeconds: 20
  48.         volumeMounts:
  49.         - name: tomcat-logs
  50.           mountPath: /usr/local/tomcat/logs
  51.  
  52.       - name: filebeat
  53.         image: docker.elastic.co/beats/filebeat:6.4.2
  54.         args: [
  55.           "-c", "/etc/filebeat.yml",
  56.           "-e",
  57.         ]
  58.         resources:
  59.           limits:
  60.             memory: 500Mi
  61.           requests:
  62.             cpu: 100m
  63.             memory: 100Mi
  64.         securityContext:
  65.           runAsUser: 0
  66.         volumeMounts:
  67.         - name: filebeat-config
  68.           mountPath: /etc/filebeat.yml
  69.           subPath: filebeat.yml
  70.         - name: tomcat-logs
  71.           mountPath: /usr/local/tomcat/logs
  72.       volumes:
  73.       - name: tomcat-logs
  74.         emptyDir: {}
  75.       - name: filebeat-config
  76.         configMap:
  77.           name: filebeat-config

2.创建filebest收集tomcat日志

  1. [root@master logs]# cat filebeat-tomcat-configmap.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: filebeat-config
  6. #  namespace: test
  7.  
  8. data:
  9.   filebeat.yml: |-
  10.     filebeat.prospectors:
  11.     - type: log
  12.       paths:
  13.         - /usr/local/tomcat/logs/catalina.*
  14.       # tags: ["tomcat"]
  15.       fields:
  16.         app: www
  17.         type: tomcat-catalina
  18.       fields_under_root: true
  19.       multiline:
  20.         pattern: '^\['
  21.         negate: true
  22.         match: after
  23.     output.logstash:
  24.       hosts: ['172.31.182.140:5044']

3.添加logstas配置后重启

  1. [root@master logs]# cat /etc/logstash/conf.d/logstash-to-es.conf
  2. input {
  3.   beats {
  4.      port => 5044
  5.   }
  6. }
  7.  
  8. filter {
  9. }
  10.  
  11. output {
  12.     if [app] == "www" {
  13.         if [type] == "nginx-access" {
  14.            elasticsearch {
  15.               hosts => ["http://127.0.0.1:9200"]
  16.               index => "nginx-access-%{+YYYY.MM.dd}"
  17.            }
  18.         }
  19.         else if [type] == "nginx-error" {
  20.            elasticsearch {
  21.               hosts => ["http://127.0.0.1:9200"]
  22.               index => "nginx-error-%{+YYYY.MM.dd}"
  23.            }
  24.         }
  25.         else if [type] == "tomcat-catalina" {
  26.            elasticsearch {
  27.               hosts => ["http://127.0.0.1:9200"]
  28.               index => "tomcat-catalina-%{+YYYY.MM.dd}"
  29.            }
  30.         }
  31.     } else if [app] == "k8s" {
  32.         if [type] == "module" {
  33.            elasticsearch {
  34.               hosts => ["http://127.0.0.1:9200"]
  35.               index => "k8s-log-%{+YYYY.MM.dd}"
  36.            }
  37.         }
  38.     }
  39. #  stdout { codec=> rubydebug }
  40. }

4.kibana添加索引

k8s日志收集方案的更多相关文章

  1. k8s 日志收集之 EFK

    如今越来越多的应用部署在容器之中,如何收集日志也是一个很重要的问题.服务出问题了,排查问题需要给开发看日志.服务一般会在多个不同的 pod 中,一个一个的登进去看也的确不方便.业务数据统计也需要日志. ...

  2. K8S学习笔记之k8s日志收集实战

    0x00 简介 本文主要介绍在k8s中收集应用的日志方案,应用运行中日志,一般情况下都需要收集存储到一个集中的日志管理系统中,可以方便对日志进行分析统计,监控,甚至用于机器学习,智能分析应用系统问题, ...

  3. Kubernetes 常用日志收集方案

    Kubernetes 常用日志收集方案 学习了 Kubernetes 集群中监控系统的搭建,除了对集群的监控报警之外,还有一项运维工作是非常重要的,那就是日志的收集. 介绍 应用程序和系统日志可以帮助 ...

  4. docker容器日志收集方案汇总评价总结

    docker日志收集方案有太多,下面截图罗列docker官方给的日志收集方案(详细请转docker官方文档).很多方案都不适合我们下面的系列文章没有说. 经过以下5篇博客的叙述简单说下docker容器 ...

  5. docker容器日志收集方案（方案N，其他中间件传输方案）

    由于docker虚拟化的特殊性导致日志收集方案的多样性和复杂性下面接收几个可能的方案 ​ 这个方案各大公司都在用只不过传输方式大同小异 中间件使用kafka是肯定的,kafka的积压与吞吐能力相当强悍 ...

  6. k8s日志收集及存档

    k8s日志收集架构图 利用阿里开源的工具log-pilot,往kafka内写日志,然后吐一份至es,另外一份用flume消费kafka数据落盘

  7. k8s日志收集配置

    容器日志样例 172.101.32.1 - - [03/Jun/2019:17:14:10 +0800] "POST /ajaxVideoQueues!queryAllUser.action ...

  8. 轻量级日志收集方案Loki

    先看看结果有多轻量吧 官方文档:https://grafana.com/docs/loki/latest/ 简介 Grafana Loki 是一个日志聚合工具,它是功能齐全的日志堆栈的核心. Loki ...

  9. golang日志收集方案之ELK

    每个系统都有日志,当系统出现问题时,需要通过日志解决问题 当系统机器比较少时,登陆到服务器上查看即可满足 当系统机器规模巨大,登陆到机器上查看几乎不现实 当然即使是机器规模不大,一个系统通常也会涉及到 ...

随机推荐

  1. html5 标签在 IE 下使用

    (function(){if(!/*@cc_on!@*/0)return;var e = "abbr,article,aside,audio,bb,canvas,datagrid,datal ...

  2. flex 分页打印表格功能

    private function printHandler():void{ var printJob:FlexPrintJob = new FlexPrintJob(); printJob.print ...

  3. 万网主机使用wordpress发送邮件的方法

    今天弄了一下午总算明白了,这里写一下具体过程. 首先是邮箱,万网主机是不支持mail()函数的,所以默认的不可用,如果你想发送邮件的话,只能使用fsockopen()函数.首先进入万网主机管理平台,启 ...

  4. Java核心卷笔记(一)

    第三章Java基程序设计结构 1.注释 三种注释方式: // 注释单行 /* 内容 */ 注释单行 /** * 内容 */ 2. java 数据类型 Java数据类型可分为两种:基本数据类型和引用数据 ...

  5. servlet3.0 新特性——异步处理

    Servlet 3.0 之前,一个普通 Servlet 的主要工作流程大致如下: 首先,Servlet 接收到请求之后,可能需要对请求携带的数据进行一些预处理: 接着,调用业务接口的某些方法,以完成业 ...

  6. 16个必须熟悉的linux服务器监控命令

    本原创文章属于<Linux大棚>博客. 博客地址为http://roclinux.cn. 文章作者为roc. == 原文:16 Linux Server Monitoring Comman ...

  7. Xor Sum 01字典树 hdu4825

    Xor Sum Time Limit: 2000/1000 MS (Java/Others)    Memory Limit: 132768/132768 K (Java/Others)Total S ...

  8. shell 中各种括号的作用()、(())、[]、[[]]、{}

    一.小括号,圆括号 () 1.单小括号 () 命令组.括号中的命令将会新开一个子shell顺序执行,所以括号中的变量不能够被脚本余下的部分使用.括号中多个命令之间用分号隔开,最后一个命令可以没有分号, ...

  9. 了解mysqlpump工具

    Ⅰ.功能分析 1.1 多线程介绍 mysqlpump是MySQL5.7的官方工具,用于取代mysqldump,其参数与mysqldump基本一样 mysqlpump是多线程备份,但只能到表级别,单表备 ...

  10. What is the best way to handle Invalid CSRF token found in the request when session times out in Spring security

    18.5.1 Timeouts One issue is that the expected CSRF token is stored in the HttpSession, so as soon a ...