K8S监控实战-ELK收集K8S内应用日志

K8S监控实战-ELK收集K8S内应用日志

1 收集K8S日志方案

K8s系统里的业务应用是高度“动态化”的，随着容器编排的进行，业务容器在不断的被创建、被摧毁、被漂移、被扩缩容…

我们需要这样一套日志收集、分析的系统：

收集 – 能够采集多种来源的日志数据（流式日志收集器）
传输 – 能够稳定的把日志数据传输到中央系统（消息队列）
存储 – 可以将日志以结构化数据的形式存储起来（搜索引擎）
分析 – 支持方便的分析、检索方法，最好有GUI管理系统（web）
警告 – 能够提供错误报告，监控机制（监控系统）

1.1 传统ELk模型缺点：

Logstash使用Jruby语言开发，吃资源，大量部署消耗极高
业务程序与logstash耦合过松，不利于业务迁移
日志收集与ES耦合又过紧，（Logstash）易打爆（ES）、丢数据
在容器云环境下，传统ELk模型难以完成工作

1.2 K8s容器日志收集模型

2 制作tomcat底包

2.1 准备tomcat底包

2.1.1 下载tomcat8

cd /opt/src/

wget http://mirror.bit.edu.cn/apache/tomcat/tomcat-8/v8.5.50/bin/apache-tomcat-8.5.50.tar.gz

mkdir /data/dockerfile/tomcat

tar xf apache-tomcat-8.5.50.tar.gz -C /data/dockerfile/tomcat

cd /data/dockerfile/tomcat

2.1.2 简单配置tomcat

删除自带网页

rm -rf apache-tomcat-8.5.50/webapps/*

关闭AJP端口

tomcat]# vim apache-tomcat-8.5.50/conf/server.xml

  <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

修改日志类型

删除3manager，4host-manager的handlers

tomcat]# vim apache-tomcat-8.5.50/conf/logging.properties

handlers = [1catalina.org.apache.juli.AsyncFileHandler](http://1catalina.org.apache.juli.asyncfilehandler/), [2localhost.org.apache.juli.AsyncFileHandler](http://2localhost.org.apache.juli.asyncfilehandler/), java.util.logging.ConsoleHandler

日志级别改为INFO

1catalina.org.apache.juli.AsyncFileHandler.level = INFO

2localhost.org.apache.juli.AsyncFileHandler.level = INFO

java.util.logging.ConsoleHandler.level = INFO

注释所有关于3manager，4host-manager日志的配置

#3manager.org.apache.juli.AsyncFileHandler.level = FINE

#3manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs

#3manager.org.apache.juli.AsyncFileHandler.prefix = manager.

#3manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

#4host-manager.org.apache.juli.AsyncFileHandler.level = FINE

#4host-manager.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs

#4host-manager.org.apache.juli.AsyncFileHandler.prefix = host-manager.

#4host-manager.org.apache.juli.AsyncFileHandler.encoding = UTF-8

2.2 准备docker镜像

2.2.1 创建dockerfile

cat >Dockerfile <<'EOF'

From harbor.od.com/public/jre:8u112

RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\

  echo 'Asia/Shanghai' >/etc/timezone

ENV CATALINA_HOME /opt/tomcat

ENV LANG zh_CN.UTF-8

ADD apache-tomcat-8.5.50/ /opt/tomcat

ADD config.yml /opt/prom/config.yml

ADD jmx_javaagent-0.3.1.jar /opt/prom/jmx_javaagent-0.3.1.jar

WORKDIR /opt/tomcat

ADD entrypoint.sh /entrypoint.sh

CMD ["/bin/bash","/entrypoint.sh"]

EOF

2.2.2 准备dockerfile所需文件

JVM监控所需jar包

wget  -O jmx_javaagent-0.3.1.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar

jmx_agent读取的配置文件

cat >config.yml <<'EOF'

---

rules:

 - pattern: '.*'

EOF

容器启动脚本

cat  >entrypoint.sh <<'EOF'

#!/bin/bash

M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):${M_PORT:-"12346"}:/opt/prom/config.yml" # Pod ip:port 监控规则传给jvm监控客户端

C_OPTS=${C_OPTS}             # 启动追加参数

MIN_HEAP=${MIN_HEAP:-"128m"} # java虚拟机初始化时的最小内存

MAX_HEAP=${MAX_HEAP:-"128m"} # java虚拟机初始化时的最大内存

JAVA_OPTS=${JAVA_OPTS:-"-Xmn384m -Xss256k -Duser.timezone=GMT+08  -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram  -Dfile.encoding=UTF8 -Dsun.jnu.encoding=UTF8"}     # 年轻代，gc回收

CATALINA_OPTS="${CATALINA_OPTS}"

JAVA_OPTS="${M_OPTS} ${C_OPTS} -Xms${MIN_HEAP} -Xmx${MAX_HEAP} ${JAVA_OPTS}"

sed -i -e "1a\JAVA_OPTS=\"$JAVA_OPTS\"" -e "1a\CATALINA_OPTS=\"$CATALINA_OPTS\"" /opt/tomcat/bin/catalina.sh

cd /opt/tomcat && /opt/tomcat/bin/catalina.sh run 2>&1 >> /opt/tomcat/logs/stdout.log # 日志文件

EOF

2.2.3 构建docker

docker build . -t harbor.zq.com/base/tomcat:v8.5.50

docker push       harbor.zq.com/base/tomcat:v8.5.50

3 部署ElasticSearch

官网

 官方github地址

 下载地址

部署HDSS7-12.host.com上：

3.1 安装ElasticSearch

3.1.1 下载二进制包

cd /opt/src

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.6.tar.gz

tar xf elasticsearch-6.8.6.tar.gz -C /opt/

ln -s /opt/elasticsearch-6.8.6/ /opt/elasticsearch

cd /opt/elasticsearch

3.1.2 配置elasticsearch.yml

mkdir -p /data/elasticsearch/{data,logs}

cat >config/elasticsearch.yml <<'EOF'

cluster.name: es.zq.com

node.name: hdss7-12.host.com

path.data: /data/elasticsearch/data

path.logs: /data/elasticsearch/logs

bootstrap.memory_lock: true

network.host: 10.4.7.12

http.port: 9200

EOF

3.2 优化其他设置

3.2.1 设置jvm参数

elasticsearch]# vi config/jvm.options

# 根据环境设置，-Xms和-Xmx设置为相同的值，推荐设置为机器内存的一半左右

-Xms512m

-Xmx512m

3.2.2 创建普通用户

useradd -s /bin/bash -M es

chown -R es.es /opt/elasticsearch-6.8.6

chown -R es.es /data/elasticsearch/

3.2.3 调整文件描述符

vim /etc/security/limits.d/es.conf

es hard nofile 65536

es soft fsize unlimited

es hard memlock unlimited

es soft memlock unlimited

3.2.4 调整内核参数

sysctl -w vm.max_map_count=262144

echo "vm.max_map_count=262144" > /etc/sysctl.conf

sysctl -p

3.3 启动ES

3.3.1 启动es服务

]# su -c "/opt/elasticsearch/bin/elasticsearch -d" es

]# netstat -luntp|grep 9200

tcp6    0   0 10.4.7.12:9200     :::*          LISTEN   16784/java

3.3.1 调整ES日志模板

curl -XPUT http://10.4.7.12:9200/_template/k8s -d '{

 "template" : "k8s*",

 "index_patterns": ["k8s*"],

 "settings": {

  "number_of_shards": 5,

  "number_of_replicas": 0    # 生产为3份副本集，本es为单节点，不能配置副本集

 }

}'

4 部署kafka和kafka-manager

官网

 官方github地址

 下载地址

HDSS7-11.host.com上：

4.1 但节点安装kafka

4.1.1 下载包

cd /opt/src

wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz

tar xf kafka_2.12-2.2.0.tgz -C /opt/

ln -s /opt/kafka_2.12-2.2.0/ /opt/kafka

cd /opt/kafka

4.1.2 修改配置

mkdir /data/kafka/logs -p

cat >config/server.properties <<'EOF'

log.dirs=/data/kafka/logs

zookeeper.connect=localhost:2181    # zk消息队列地址

log.flush.interval.messages=10000

log.flush.interval.ms=1000

delete.topic.enable=true

host.name=hdss7-11.host.com

EOF

4.1.3 启动kafka

bin/kafka-server-start.sh -daemon config/server.properties

]# netstat -luntp|grep 9092

tcp6    0   0 10.4.7.11:9092     :::*          LISTEN   34240/java

4.2 获取kafka-manager的docker镜像

官方github地址

 源码下载地址

运维主机HDSS7-200.host.com上：

kafka-manager是kafka的一个web管理页面,非必须

4.2.1 方法一通过dockerfile获取

1 准备Dockerfile

cat >/data/dockerfile/kafka-manager/Dockerfile <<'EOF'

FROM hseeberger/scala-sbt

ENV ZK_HOSTS=10.4.7.11:2181 \

     KM_VERSION=2.0.0.2

RUN mkdir -p /tmp && \

    cd /tmp && \

    wget https://github.com/yahoo/kafka-manager/archive/${KM_VERSION}.tar.gz && \

    tar xxf ${KM_VERSION}.tar.gz && \

    cd /tmp/kafka-manager-${KM_VERSION} && \

    sbt clean dist && \

    unzip  -d / ./target/universal/kafka-manager-${KM_VERSION}.zip && \

    rm -fr /tmp/${KM_VERSION} /tmp/kafka-manager-${KM_VERSION}

WORKDIR /kafka-manager-${KM_VERSION}

EXPOSE 9000

ENTRYPOINT ["./bin/kafka-manager","-Dconfig.file=conf/application.conf"]

EOF

2 制作docker镜像

cd /data/dockerfile/kafka-manager

docker build . -t harbor.od.com/infra/kafka-manager:v2.0.0.2

(漫长的过程)

docker push harbor.zq.com/infra/kafka-manager:latest

构建过程极其漫长,大概率会失败,因此可以通过第二种方式下载构建好的镜像

但构建好的镜像写死了zk地址,要注意传入变量修改zk地址

4.2.2 直接下载docker镜像

镜像下载地址

docker pull sheepkiller/kafka-manager:latest

docker images|grep kafka-manager

docker tag  4e4a8c5dabab harbor.zq.com/infra/kafka-manager:latest

docker push harbor.zq.com/infra/kafka-manager:latest

4.3 部署kafka-manager

mkdir /data/k8s-yaml/kafka-manager

cd /data/k8s-yaml/kafka-manager

4.3.1 准备dp清单

cat >deployment.yaml <<'EOF'

kind: Deployment

apiVersion: extensions/v1beta1

metadata:

  name: kafka-manager

  namespace: infra

  labels:

    name: kafka-manager

spec:

  replicas: 1

  selector:

    matchLabels:

      name: kafka-manager

  template:

    metadata:

      labels:

        app: kafka-manager

        name: kafka-manager

    spec:

      containers:

      - name: kafka-manager

        image: harbor.zq.com/infra/kafka-manager:latest

        ports:

        - containerPort: 9000

          protocol: TCP

        env:

        - name: ZK_HOSTS

          value: zk1.od.com:2181

        - name: APPLICATION_SECRET

          value: letmein

        imagePullPolicy: IfNotPresent

      imagePullSecrets:

      - name: harbor

      restartPolicy: Always

      terminationGracePeriodSeconds: 30

      securityContext:

        runAsUser: 0

      schedulerName: default-scheduler

  strategy:

    type: RollingUpdate

    rollingUpdate:

      maxUnavailable: 1

      maxSurge: 1

  revisionHistoryLimit: 7

  progressDeadlineSeconds: 600

EOF

4.3.2 准备svc资源清单

cat >service.yaml <<'EOF'

kind: Service

apiVersion: v1

metadata:

  name: kafka-manager

  namespace: infra

spec:

  ports:

  - protocol: TCP

    port: 9000

    targetPort: 9000

  selector:

    app: kafka-manager

EOF

4.3.3 准备ingress资源清单

cat >ingress.yaml <<'EOF'

kind: Ingress

apiVersion: extensions/v1beta1

metadata:

  name: kafka-manager

  namespace: infra

spec:

  rules:

  - host: km.zq.com

    http:

      paths:

      - path: /

        backend:

          serviceName: kafka-manager

          servicePort: 9000

EOF

4.3.4 应用资源配置清单

任意一台运算节点上：

kubectl apply -f http://k8s-yaml.od.com/kafka-manager/deployment.yaml

kubectl apply -f http://k8s-yaml.od.com/kafka-manager/service.yaml

kubectl apply -f http://k8s-yaml.od.com/kafka-manager/ingress.yaml

4.3.5 解析域名

HDSS7-11.host.com上

~]# vim /var/named/zq.com.zone

km    A   10.4.7.10

~]# systemctl restart named

~]# dig -t A km.od.com @10.4.7.11 +short

10.4.7.10

4.3.6 浏览器访问

http://km.zq.com

添加集群

查看集群信息

5 部署filebeat

官方下载地址

运维主机HDSS7-200.host.com上：

5.1 制作docker镜像

mkdir /data/dockerfile/filebeat

cd /data/dockerfile/filebeat

5.1.1 准备Dockerfile

cat >Dockerfile <<'EOF'

FROM debian:jessie

# 如果更换版本,需在官网下载同版本LINUX64-BIT的sha替换FILEBEAT_SHA1

ENV FILEBEAT_VERSION=7.5.1 \ FILEBEAT_SHA1=daf1a5e905c415daf68a8192a069f913a1d48e2c79e270da118385ba12a93aaa91bda4953c3402a6f0abf1c177f7bcc916a70bcac41977f69a6566565a8fae9c

RUN set -x && \

 apt-get update && \

 apt-get install -y wget && \

 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz -O /opt/filebeat.tar.gz && \

 cd /opt && \

 echo "${FILEBEAT_SHA1} filebeat.tar.gz" | sha512sum -c - && \

 tar xzvf filebeat.tar.gz && \

 cd filebeat-* && \

 cp filebeat /bin && \

 cd /opt && \

 rm -rf filebeat* && \

 apt-get purge -y wget && \

 apt-get autoremove -y && \

 apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

COPY filebeat.yaml /etc/

COPY docker-entrypoint.sh /

ENTRYPOINT ["/bin/bash","/docker-entrypoint.sh"]

EOF

5.1.2 准备filebeat配置文件

cat >/etc/filebeat.yaml << EOF

filebeat.inputs:

- type: log

  fields_under_root: true

  fields:

    topic: logm-PROJ_NAME

  paths:

    - /logm/*.log

    - /logm/*/*.log

    - /logm/*/*/*.log

    - /logm/*/*/*/*.log

    - /logm/*/*/*/*/*.log

  scan_frequency: 120s

  max_bytes: 10485760

  multiline.pattern: 'MULTILINE'

  multiline.negate: true

  multiline.match: after

  multiline.max_lines: 100

- type: log

  fields_under_root: true

  fields:

    topic: logu-PROJ_NAME

  paths:

    - /logu/*.log

    - /logu/*/*.log

    - /logu/*/*/*.log

    - /logu/*/*/*/*.log

    - /logu/*/*/*/*/*.log

    - /logu/*/*/*/*/*/*.log

output.kafka:

  hosts: ["10.4.7.11:9092"]

  topic: k8s-fb-ENV-%{[topic]}

  version: 2.0.0      # kafka版本超过2.0，默认写2.0.0

  required_acks: 0

  max_message_bytes: 10485760

EOF

5.1.3 准备启动脚本

cat >docker-entrypoint.sh <<'EOF'

#!/bin/bash

ENV=${ENV:-"test"}                    # 定义日志收集的环境

PROJ_NAME=${PROJ_NAME:-"no-define”}   # 定义项目名称

MULTILINE=${MULTILINE:-"^\d{2}"}      # 多行匹配，以2个数据开头的为一行，反之

# 替换配置文件中的内容

sed -i 's#PROJ_NAME#${PROJ_NAME}#g' /etc/filebeat.yaml

sed -i 's#MULTILINE#${MULTILINE}#g' /etc/filebeat.yaml

sed -i 's#ENV#${ENV}#g'             /etc/filebeat.yaml

if [[ "$1" == "" ]]; then

     exec filebeat  -c /etc/filebeat.yaml

else

    exec "$@"

fi

EOF

5.1.4 构建镜像

docker build . -t harbor.od.com/infra/filebeat:v7.5.1

docker push       harbor.od.com/infra/filebeat:v7.5.1

5.2 以边车模式运行POD

5.2.1 准备资源配置清单

使用dubbo-demo-consumer的镜像,以边车模式运行filebeat

]# vim /data/k8s-yaml/test/dubbo-demo-consumer/deployment.yaml

kind: Deployment

apiVersion: extensions/v1beta1

metadata:

  name: dubbo-demo-consumer

  namespace: test

  labels:

    name: dubbo-demo-consumer

spec:

  replicas: 1

  selector:

    matchLabels:

      name: dubbo-demo-consumer

  template:

    metadata:

      labels:

        app: dubbo-demo-consumer

        name: dubbo-demo-consumer

      annotations:

        blackbox_path: "/hello?name=health"

        blackbox_port: "8080"

        blackbox_scheme: "http"

        prometheus_io_scrape: "true"

        prometheus_io_port: "12346"

        prometheus_io_path: "/"

    spec:

      containers:

      - name: dubbo-demo-consumer

        image: harbor.zq.com/app/dubbo-tomcat-web:apollo_200513_1808

        ports:

        - containerPort: 8080

          protocol: TCP

        - containerPort: 20880

          protocol: TCP

        env:

        - name: JAR_BALL

          value: dubbo-client.jar

        - name: C_OPTS

          value: -Denv=fat -Dapollo.meta=http://config-test.zq.com

        imagePullPolicy: IfNotPresent

#--------新增内容--------

        volumeMounts:

        - mountPath: /opt/tomcat/logs

          name: logm

      - name: filebeat

        image: harbor.zq.com/infra/filebeat:v7.5.1

        imagePullPolicy: IfNotPresent

        env:

        - name: ENV

          value: test             # 测试环境

        - name: PROJ_NAME

          value: dubbo-demo-web   # 项目名

        volumeMounts:

        - mountPath: /logm

          name: logm

      volumes:

      - emptyDir: {} #随机在宿主机找目录创建,容器删除时一起删除

        name: logm

#--------新增结束--------

      imagePullSecrets:

      - name: harbor

      restartPolicy: Always

      terminationGracePeriodSeconds: 30

      securityContext:

        runAsUser: 0

      schedulerName: default-scheduler

  strategy:

    type: RollingUpdate

    rollingUpdate:

      maxUnavailable: 1

      maxSurge: 1

  revisionHistoryLimit: 7

  progressDeadlineSeconds: 600

5.2.2 应用资源清单

任意node节点

kubectl apply -f http://k8s-yaml.od.com/test/dubbo-demo-consumer/deployment.yaml

5.2.3 验证

浏览器访问http://km.zq.com,看到kafaka-manager里，topic打进来，即为成功

进入dubbo-demo-consumer的容器中,查看logm目录下是否有日志

kubectl -n test exec -it dobbo...... -c filebeat /bin/bash

ls /logm

# -c参数指定pod中的filebeat容器

# /logm是filebeat容器挂载的目录

6 部署logstash

运维主机HDSS7-200.host.com上：

6.1 准备docker镜像

6.1.1 下载官方镜像

docker pull logstash:6.8.6

docker tag  d0a2dac51fcb harbor.od.com/infra/logstash:v6.8.6

docker push harbor.zq.com/infra/logstash:v6.8.6

6.1.2 准备配置文件

准备目录

mkdir /etc/logstash/

创建test.conf

cat >/etc/logstash/logstash-test.conf <<'EOF'

input {

  kafka {

    bootstrap_servers => "10.4.7.11:9092"

    client_id => "10.4.7.200"

    consumer_threads => 4

    group_id => "k8s_test"               # 为test组

    topics_pattern => "k8s-fb-test-.*"   # 只收集k8s-fb-test开头的topics

  }

}

filter {

  json {

    source => "message"

  }

}

output {

  elasticsearch {

    hosts => ["10.4.7.12:9200"]

    index => "k8s-test-%{+YYYY.MM.DD}"

  }

}

EOF

创建prod.conf

cat >/etc/logstash/logstash-prod.conf <<'EOF'

input {

  kafka {

    bootstrap_servers => "10.4.7.11:9092"

    client_id => "10.4.7.200"

    consumer_threads => 4

    group_id => "k8s_prod"

    topics_pattern => "k8s-fb-prod-.*"

  }

}

filter {

  json {

    source => "message"

  }

}

output {

  elasticsearch {

    hosts => ["10.4.7.12:9200"]

    index => “k8s-prod-%{+YYYY.MM.DD}"

  }

}

EOF

6.2 启动logstash

6.2.1 启动测试环境的logstash

docker run -d \

    --restart=always \

    --name logstash-test \

    -v /etc/logstash:/etc/logstash  \

    -f /etc/logstash/logstash-test.conf  \

    harbor.od.com/infra/logstash:v6.8.6 

~]# docker ps -a|grep logstash

6.2.2 查看es是否接收数据

~]# curl http://10.4.7.12:9200/_cat/indices?v

health status index        uuid          pri rep docs.count docs.deleted store.size pri.store.size

green open  k8s-test-2020.01.07 mFEQUyKVTTal8c97VsmZHw  5  0     12      0   78.4kb     78.4kb

6.2.3 启动正式环境的logstash

docker run -d \

    --restart=always \

    --name logstash-prod \

    -v /etc/logstash:/etc/logstash  \

    -f /etc/logstash/logstash-prod.conf  \

    harbor.od.com/infra/logstash:v6.8.6

7 部署Kibana