使用ConfigMap管理应用配置

拆分环境

主机名 角色 ip
HDSS7-11.host.com zk1.od.com(Test环境) 10.4.7.11
HDSS7-12.host.com zk2.od.com(Prod环境) 10.4.7.12

重配zookeeper

HDSS7-11.host.com上：

复制/opt/zookeeper/conf/zoo.cfg

1
2
3
4
5
6
 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

HDSS7-12.host.com上：

复制/opt/zookeeper/conf/zoo.cfg

1
2
3
4
5
6
 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181

重启zk(删除数据文件)

复制

1
2
3
 
[root@hdss7-11 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-12 ~]# /opt/zookeeper/bin/zkServer.sh restart && /opt/zookeeper/bin/zkServer.sh status
[root@hdss7-21 ~]# /opt/zookeeper/bin/zkServer.sh stop

准备资源配置清单(dubbo-monitor)

在运维主机HDSS7-200.host.com上：

vi /data/k8s-yaml/dubbo-monitor/configmap.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
apiVersion: v1
kind: ConfigMap
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties: |
    dubbo.container=log4j,spring,registry,jetty
    dubbo.application.name=simple-monitor
    dubbo.application.owner=
    dubbo.registry.address=zookeeper://zk1.od.com:2181
    dubbo.protocol.port=20880
    dubbo.jetty.port=8080
    dubbo.jetty.directory=/dubbo-monitor-simple/monitor
    dubbo.charts.directory=/dubbo-monitor-simple/charts
    dubbo.statistics.directory=/dubbo-monitor-simple/statistics
    dubbo.log4j.file=/dubbo-monitor-simple/logs/dubbo-monitor.log
    dubbo.log4j.level=WARN

vi /data/k8s-yaml/dubbo-monitor/deployment.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels: 
    name: dubbo-monitor
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-monitor
  template:
    metadata:
      labels: 
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: configmap-volume
            mountPath: /dubbo-monitor-simple/conf
      volumes:
        - name: configmap-volume
          configMap:
            name: dubbo-monitor-cm
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

应用资源配置清单

在任意一台k8s运算节点执行：

复制

1
2
3
4
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/configmap.yaml
configmap/dubbo-monitor-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-monitor/deployment.yaml
deployment.extensions/dubbo-monitor configured

重新发版，修改dubbo项目的配置文件

修改项目源代码

  • duboo-demo-service

    复制dubbo-server/src/main/java/config.properties

    1
    2
    		 
    dubbo.registry=zookeeper://zk1.od.com:2181
    dubbo.port=28080

  • dubbo-demo-web

    复制dubbo-client/src/main/java/config.properties

    1
    		 
    dubbo.registry=zookeeper://zk1.od.com:2181

使用Jenkins进行CI

略

修改/应用资源配置清单

k8s的dashboard上，修改deployment使用的容器版本，提交应用

验证configmap的配置

在K8S的dashboard上，修改dubbo-monitor的configmap配置为不同的zk，重启POD，浏览器打开http://dubbo-monitor.od.com 观察效果

交付Apollo至Kubernetes集群

Apollo简介

Apollo（阿波罗）是携程框架部门研发的分布式配置中心，能够集中化管理应用不同环境、不同集群的配置，配置修改后能够实时推送到应用端，并且具备规范的权限、流程治理等特性，适用于微服务配置管理场景。

官方GitHub地址

Apollo官方地址
官方release包

基础架构

简化模型

交付apollo-configservice

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

复制/opt/src

1
2
3
4
5
6
7
8
9
10
11
12
 
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 23:29 apollo-configservice-1.3.0-github.zip
[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-configservice && unzip -o apollo-configservice-1.3.0-github.zip -d /data/dockerfile/apollo-configservice
Archive:  apollo-configservice-1.3.0-github.zip
   creating: /data/dockerfile/apollo-configservice/scripts/
  inflating: /data/dockerfile/apollo-configservice/config/application-github.properties  
  inflating: /data/dockerfile/apollo-configservice/scripts/shutdown.sh  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0-sources.jar  
  inflating: /data/dockerfile/apollo-configservice/scripts/startup.sh  
  inflating: /data/dockerfile/apollo-configservice/config/app.properties  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice-1.3.0.jar  
  inflating: /data/dockerfile/apollo-configservice/apollo-configservice.conf

执行数据库脚本

在数据库主机HDSS7-11.host.com上：
注意：MySQL版本应为5.6或以上！

  • 更新yum源
复制/etc/yum.repos.d/MariaDB.repo

1
2
3
4
5
 
[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64/
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1

  • 导入GPG-KEY

    复制

    1
    		 
    [root@hdss7-11 ~]# rpm --import https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB

  • 更新数据库版本

    复制

    1
    		 
    [root@hdss7-11 ~]# yum update MariaDB-server -y

数据库脚本地址

复制

1
2
3
 
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloConfigDB;
mysql> source ./apolloconfig.sql

数据库用户授权

复制

1
 
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

修改初始数据

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 
mysql> update ApolloConfigDB.ServerConfig set ServerConfig.Value="http://config.od.com/eureka" where ServerConfig.Key="eureka.service.url";
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from ServerConfig\G
*************************** 1. row ***************************
                       Id: 1
                      Key: eureka.service.url
                  Cluster: default
                    Value: http://config.od.com/eureka
                  Comment: Eureka服务Url，多个service以英文逗号分隔
                IsDeleted:  
     DataChange_CreatedBy: default
   DataChange_CreatedTime: 2019-04-10 15:07:34
DataChange_LastModifiedBy: 
      DataChange_LastTime: 2019-04-11 16:28:57

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串

    复制/data/dockerfile/apollo-configservice

    1
    		 
    [root@hdss7-200 apollo-configservice]# cat config/application-github.properties

  • 更新startup.sh

    复制/data/dockerfile/apollo-configservice/scripts/startup.sh

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    		 
    #!/bin/bash
    SERVICE_NAME=apollo-configservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-config-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_CONFIG_SERVICE_NAME=$(hostname -i)
    SERVER_URL="http://${APOLLO_CONFIG_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms6144m -Xmx6144m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=4096m -XX:MaxNewSize=4096m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi

if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi

tail -f /dev/null

  • 写Dockerfile

    复制/data/dockerfile/apollo-configservice/Dockerfile

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    		 
    FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone

ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
    ADD config/ /apollo-configservice/config
    ADD scripts/ /apollo-configservice/scripts

CMD ["/apollo-configservice/scripts/startup.sh"]

  • 制作镜像并推送

    复制

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    		 
    [root@hdss7-200 apollo-configservice]# docker build . -t harbor.od.com/infra/apollo-configservice:v1.3.0
    Sending build context to Docker daemon 61.91 MB
    Step 1 : FROM stanleyws/jre8:8u112
     ---> fa3a085d6ef1
    Step 2 : ENV VERSION 1.3.0
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in 685d51b5adb4
     ---> feb4c0289f04
    Removing intermediate container 685d51b5adb4
    Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in eaa05073feeb
     ---> a3e3fd61ae35
    Removing intermediate container eaa05073feeb
    Step 4 : ADD apollo-configservice-${VERSION}.jar /apollo-configservice/apollo-configservice.jar
     ---> be09a59b83a2
    Removing intermediate container ac6b8af3979b
    Step 5 : ADD config/ /apollo-configservice/config
     ---> fb64fc0f3194
    Removing intermediate container b73c5315ad20
    Step 6 : ADD scripts/ /apollo-configservice/scripts
     ---> 96ff3d9b9456
    Removing intermediate container 67ba203b3101
    Step 7 : CMD /apollo-configservice/scripts/startup.sh
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in 80bd3f53fefc
     ---> 551ea2ba8de3
    Removing intermediate container 80bd3f53fefc
    Successfully built 551ea2ba8de3

[root@hdss7-200 apollo-configservice]# docker push harbor.od.com/infra/apollo-configservice:v1.3.0
    The push refers to a repository [harbor.od.com/infra/apollo-configservice]
    25efb9a44683: Pushed 
    b3572bb46247: Pushed 
    e7994b936025: Pushed 
    0ff1d078cbc4: Pushed 
    ebfb473df5c2: Pushed 
    aae5c057d1b6: Pushed 
    dee6aef5c2b6: Pushed 
    a464c54f93a9: Pushed 
    v1.3.0: digest: sha256:6a8e4fdda58de0dfba9985ebbf91c4d6f46f5274983d2efa8853b03f4e45fa06 size: 1992

解析域名

DNS主机HDSS7-11.host.com上：

复制/var/named/od.com.zone

1
2
 
mysql   60 IN A 10.4.7.11
config	60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com上

复制/data/k8s-yaml

1
 
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-configservice && cd /data/k8s-yaml/apollo-configservice

vi deployment.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-configservice
  namespace: infra
  labels: 
    name: apollo-configservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-configservice
  template:
    metadata:
      labels: 
        app: apollo-configservice 
        name: apollo-configservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-configservice-cm
      containers:
      - name: apollo-configservice
        image: harbor.od.com/infra/apollo-configservice:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-configservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi svc.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
kind: Service
apiVersion: v1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-configservice
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi ingress.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-configservice
  namespace: infra
spec:
  rules:
  - host: config.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-configservice
          servicePort: 8080

vi configmap.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-configservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003171

应用资源配置清单

在任意一台k8s运算节点执行：

复制

1
2
3
4
5
6
7
8
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/configmap.yaml
configmap/apollo-configservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/deployment.yaml
deployment.extensions/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/svc.yaml
service/apollo-configservice created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-configservice/ingress.yaml
ingress.extensions/apollo-configservice created

浏览器访问

http://config.od.com

交付apollo-adminservice

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

复制

1
2
3
4
5
 
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:47 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-adminservice && unzip -o apollo-adminservice-1.3.0-github.zip -d /data/dockerfile/apollo-adminservice

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串

    复制/data/dockerfile/apollo-adminservice

    1
    		 
    [root@hdss7-200 apollo-adminservice]# cat config/application-github.properties

  • 更新starup.sh

    复制/data/dockerfile/apollo-adminservice/scripts/startup.sh

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    		 
    #!/bin/bash
    SERVICE_NAME=apollo-adminservice
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-adminservice
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_ADMIN_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:${SERVER_PORT}"
    SERVER_URL="http://${APOLLO_ADMIN_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi

if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi

tail -f /dev/null

  • 写Dockerfile

    复制/data/dockerfile/apollo-adminservice/Dockerfile

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    		 
    FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
        echo "Asia/Shanghai" > /etc/timezone

ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
    ADD config/ /apollo-adminservice/config
    ADD scripts/ /apollo-adminservice/scripts

CMD ["/apollo-adminservice/scripts/startup.sh"]

  • 制作镜像并推送

    复制

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    		 
    [root@hdss7-200 apollo-adminservice]# docker build . -t harbor.od.com/infra/apollo-adminservice:v1.3.0
    Sending build context to Docker daemon 58.31 MB
    Step 1 : FROM stanleyws/jre8:8u112
     ---> fa3a085d6ef1
    Step 2 : ENV VERSION 1.3.0
     ---> Using cache
     ---> feb4c0289f04
    Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
     ---> Using cache
     ---> a3e3fd61ae35
    Step 4 : ADD apollo-adminservice-${VERSION}.jar /apollo-adminservice/apollo-adminservice.jar
     ---> 6a1eb9565777
    Removing intermediate container 7196df9af6af
    Step 5 : ADD config/ /apollo-adminservice/config
     ---> 9f364b732d46
    Removing intermediate container 9b24669c6c78
    Step 6 : ADD scripts/ /apollo-adminservice/scripts
     ---> b7bc5517b0fc
    Removing intermediate container f3e34e759148
    Step 7 : CMD /apollo-adminservice/scripts/startup.sh
     ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
     ---> Running in 18c6597914b4
     ---> 82145db3ee88
    Removing intermediate container 18c6597914b4
    Successfully built 82145db3ee88

[root@hdss7-200 apollo-adminservice]# docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    docker push harbor.od.com/infra/apollo-adminservice:v1.3.0
    The push refers to a repository [harbor.od.com/infra/apollo-adminservice]
    19b1ca6c066d: Pushed 
    8fa6cde49908: Pushed 
    0b2c9b9226cc: Pushed 
    ebfb473df5c2: Mounted from infra/apollo-configservice 
    aae5c057d1b6: Mounted from infra/apollo-configservice 
    dee6aef5c2b6: Mounted from infra/apollo-configservice 
    a464c54f93a9: Mounted from infra/apollo-configservice 
    v1.3.0: digest: sha256:75367caab9bad3d0d281eb3324451a0734e84b6aa3ee860e38ad758d7166a7d1 size: 1785

准备资源配置清单

在运维主机HDSS7-200.host.com上

复制/data/k8s-yaml

1
 
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-adminservice && cd /data/k8s-yaml/apollo-adminservice

vi deployment.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-adminservice
  namespace: infra
  labels: 
    name: apollo-adminservice
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-adminservice
  template:
    metadata:
      labels: 
        app: apollo-adminservice 
        name: apollo-adminservice
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-adminservice-cm
      containers:
      - name: apollo-adminservice
        image: harbor.od.com/infra/apollo-adminservice:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-adminservice/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi configmap.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-adminservice-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config.od.com/eureka
  app.properties: |
    appId=100003172

应用资源配置清单

在任意一台k8s运算节点执行：

复制

1
2
3
4
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/configmap.yaml
configmap/apollo-adminservice-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-adminservice/deployment.yaml
deployment.extensions/apollo-adminservice created

浏览器访问

http://config.od.com

交付apollo-portal

准备软件包

在运维主机HDSS7-200.host.com上：
下载官方release包

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 
[root@hdss7-200 src]# ls -l|grep apollo
-rw-r--r-- 1 root root 52713404 Feb 16 08:37 apollo-configservice-1.3.0-github.zip
-rw-r--r-- 1 root root 49418246 Feb 16 09:54 apollo-adminservice-1.3.0-github.zip
-rw-r--r-- 1 root root 36459359 Feb 16 10:00 apollo-portal-1.3.0-github.zip

[root@hdss7-200 src]# mkdir /data/dockerfile/apollo-portal && unzip -o apollo-portal-1.3.0-github.zip -d /data/dockerfile/apollo-portal
Archive:  apollo-portal-1.3.0-github.zip
  inflating: /data/dockerfile/apollo-portal/scripts/shutdown.sh  
  inflating: /data/dockerfile/apollo-portal/apollo-portal.conf  
  inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0-sources.jar  
   creating: /data/dockerfile/apollo-portal/config/
  inflating: /data/dockerfile/apollo-portal/config/application-github.properties  
  inflating: /data/dockerfile/apollo-portal/scripts/startup.sh  
  inflating: /data/dockerfile/apollo-portal/config/apollo-env.properties  
  inflating: /data/dockerfile/apollo-portal/config/app.properties  
  inflating: /data/dockerfile/apollo-portal/apollo-portal-1.3.0.jar

执行数据库脚本

在数据库主机HDSS7-11.host.com上：
数据库脚本地址

复制

1
2
3
 
[root@hdss7-11 ~]# mysql -uroot -p
mysql> create database ApolloPortalDB;
mysql> source ./apolloportal.sql

数据库用户授权

复制

1
 
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloPortalDB.* to "apolloportal"@"172.7.%" identified by "123456";

制作Docker镜像

在运维主机HDSS7-200.host.com上：

  • 配置数据库连接串

    复制/data/dockerfile/apollo-portal

    1
    		 
    [root@hdss7-200 apollo-portal]# cat config/application-github.properties

  • 配置Portal的meta service

    复制/data/dockerfile/apollo-portal/config/apollo-env.properties

    1
    		 
    dev.meta=http://config.od.com

  • 更新starup.sh

    复制/data/dockerfile/apollo-portal/scripts/startup.sh

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    		 
    #!/bin/bash
    SERVICE_NAME=apollo-portal
    ## Adjust log dir if necessary
    LOG_DIR=/opt/logs/apollo-portal-server
    ## Adjust server port if necessary
    SERVER_PORT=8080
    APOLLO_PORTAL_SERVICE_NAME=$(hostname -i)
    # SERVER_URL="http://localhost:$SERVER_PORT"
    SERVER_URL="http://${APOLLO_PORTAL_SERVICE_NAME}:${SERVER_PORT}"

## Adjust memory settings if necessary
    #export JAVA_OPTS="-Xms2560m -Xmx2560m -Xss256k -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=384m -XX:NewSize=1536m -XX:MaxNewSize=1536m -XX:SurvivorRatio=8"

## Only uncomment the following when you are using server jvm
    #export JAVA_OPTS="$JAVA_OPTS -server -XX:-ReduceInitialCardMarks"

########### The following is the same for configservice, adminservice, portal ###########
    export JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=4 -XX:MaxTenuringThreshold=9 -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -Duser.timezone=Asia/Shanghai -Dclient.encoding.override=UTF-8 -Dfile.encoding=UTF-8 -Djava.security.egd=file:/dev/./urandom"
    export JAVA_OPTS="$JAVA_OPTS -Dserver.port=$SERVER_PORT -Dlogging.file=$LOG_DIR/$SERVICE_NAME.log -XX:HeapDumpPath=$LOG_DIR/HeapDumpOnOutOfMemoryError/"

# Find Java
    if [[ -n "$JAVA_HOME" ]] && [[ -x "$JAVA_HOME/bin/java" ]]; then
        javaexe="$JAVA_HOME/bin/java"
    elif type -p java > /dev/null 2>&1; then
        javaexe=$(type -p java)
    elif [[ -x "/usr/bin/java" ]];  then
        javaexe="/usr/bin/java"
    else
        echo "Unable to find Java"
        exit 1
    fi

if [[ "$javaexe" ]]; then
        version=$("$javaexe" -version 2>&1 | awk -F '"' '/version/ {print $2}')
        version=$(echo "$version" | awk -F. '{printf("%03d%03d",$1,$2);}')
        # now version is of format 009003 (9.3.x)
        if [ $version -ge 011000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 010000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        elif [ $version -ge 009000 ]; then
            JAVA_OPTS="$JAVA_OPTS -Xlog:gc*:$LOG_DIR/gc.log:time,level,tags -Xlog:safepoint -Xlog:gc+heap=trace"
        else
            JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
            JAVA_OPTS="$JAVA_OPTS -Xloggc:$LOG_DIR/gc.log -XX:+PrintGCDetails"
            JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSFullGCsBeforeCompaction=9 -XX:+CMSClassUnloadingEnabled  -XX:+PrintGCDateStamps -XX:+PrintGCApplicationConcurrentTime -XX:+PrintHeapAtGC -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=5M"
        fi
    fi

printf "$(date) ==== Starting ==== \n"

cd `dirname $0`/..
    chmod 755 $SERVICE_NAME".jar"
    ./$SERVICE_NAME".jar" start

rc=$?;

if [[ $rc != 0 ]];
    then
        echo "$(date) Failed to start $SERVICE_NAME.jar, return code: $rc"
        exit $rc;
    fi

tail -f /dev/null

  • 写Dockerfile

复制/data/dockerfile/apollo-portal/Dockerfile

1
2
3
4
5
6
7
8
9
10
11
12
 
FROM stanleyws/jre8:8u112

ENV VERSION 1.3.0

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\
    echo "Asia/Shanghai" > /etc/timezone

ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
ADD config/ /apollo-portal/config
ADD scripts/ /apollo-portal/scripts

CMD ["/apollo-portal/scripts/startup.sh"]
  • 制作镜像并推送
复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
[root@hdss7-200 apollo-portal]# docker build . -t harbor.od.com/infra/apollo-portal:v1.3.0
Sending build context to Docker daemon 43.35 MB
Step 1 : FROM stanleyws/jre8:8u112
 ---> fa3a085d6ef1
Step 2 : ENV VERSION 1.3.0
 ---> Using cache
 ---> feb4c0289f04
Step 3 : RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&    echo "Asia/Shanghai" > /etc/timezone
 ---> Using cache
 ---> a3e3fd61ae35
Step 4 : ADD apollo-portal-${VERSION}.jar /apollo-portal/apollo-portal.jar
 ---> cfcf63e8eedc
Removing intermediate container 860b55bd3fc5
Step 5 : ADD config/ /apollo-portal/config
 ---> 3ee780369431
Removing intermediate container 6b67ee4224b5
Step 6 : ADD scripts/ /apollo-portal/scripts
 ---> 42c9aea2e9e3
Removing intermediate container 2dcf8d1bf4cf
Step 7 : CMD /apollo-portal/scripts/startup.sh
 ---> [Warning] IPv4 forwarding is disabled. Networking will not work.
 ---> Running in 9162dab8b63a
 ---> 0c020b79c36f
Removing intermediate container 9162dab8b63a
Successfully built 0c020b79c36f
[root@hdss7-200 apollo-portal]# docker push harbor.od.com/infra/apollo-portal:v1.3.0
docker push harbor.od.com/infra/apollo-portal:v1.3.0
The push refers to a repository [harbor.od.com/infra/apollo-portal]
e7c0e96ded4e: Pushed 
0076c5344476: Pushed 
3851a45d7440: Pushed 
ebfb473df5c2: Mounted from infra/apollo-adminservice 
aae5c057d1b6: Mounted from infra/apollo-adminservice 
dee6aef5c2b6: Mounted from infra/apollo-adminservice 
a464c54f93a9: Mounted from infra/apollo-adminservice 
v1.3.0: digest: sha256:1aa30aac8642cceb97c053b7d74632240af08f64c49b65d8729021fef65628a4 size: 1785

解析域名

DNS主机HDSS7-11.host.com上：

复制/var/named/od.com.zone

1
 
portal	60 IN A 10.4.7.10

准备资源配置清单

在运维主机HDSS7-200.host.com上

复制/data/k8s-yaml

1
 
[root@hdss7-200 k8s-yaml]# mkdir /data/k8s-yaml/apollo-portal && cd /data/k8s-yaml/apollo-portal

vi deployment.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: apollo-portal
  namespace: infra
  labels: 
    name: apollo-portal
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: apollo-portal
  template:
    metadata:
      labels: 
        app: apollo-portal 
        name: apollo-portal
    spec:
      volumes:
      - name: configmap-volume
        configMap:
          name: apollo-portal-cm
      containers:
      - name: apollo-portal
        image: harbor.od.com/infra/apollo-portal:v1.3.0
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: configmap-volume
          mountPath: /apollo-portal/config
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

vi svc.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
kind: Service
apiVersion: v1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector: 
    app: apollo-portal
  clusterIP: None
  type: ClusterIP
  sessionAffinity: None

vi ingress.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 
kind: Ingress
apiVersion: extensions/v1beta1
metadata: 
  name: apollo-portal
  namespace: infra
spec:
  rules:
  - host: portal.od.com
    http:
      paths:
      - path: /
        backend: 
          serviceName: apollo-portal
          servicePort: 8080

vi configmap.yaml

复制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
apiVersion: v1
kind: ConfigMap
metadata:
  name: apollo-portal-cm
  namespace: infra
data:
  application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloPortalDB?characterEncoding=utf8
    spring.datasource.username = apolloportal
    spring.datasource.password = 123456
  app.properties: |
    appId=100003173
  apollo-env.properties: |
    dev.meta=http://config.od.com

应用资源配置清单

在任意一台k8s运算节点执行：

复制

1
2
3
4
5
6
7
8
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/configmap.yaml
configmap/apollo-portal-cm created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/deployment.yaml
deployment.extensions/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/svc.yaml
service/apollo-portal created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/apollo-portal/ingress.yaml
ingress.extensions/apollo-portal created

浏览器访问

http://portal.od.com

  • 用户名：apollo
  • 密码： admin

实战dubbo微服务接入Apollo配置中心

改造dubbo-demo-service项目

使用IDE拉取项目（这里使用git bash作为范例）

复制

1
 
$ git clone git@gitee.com/stanleywang/dubbo-demo-service.git

切到apollo分支

复制

1
 
$ git checkout -b apollo

修改pom.xml

  • 加入apollo客户端jar包的依赖
复制dubbo-server/pom.xml

1
2
3
4
5
 
<dependency>
  <groupId>com.ctrip.framework.apollo</groupId>
  <artifactId>apollo-client</artifactId>
  <version>1.1.0</version>
</dependency>
  • 修改resource段
复制dubbo-server/pom.xml

1
2
3
4
5
6
7
 
<resource>
  <directory>src/main/resources</directory>
  <includes>
  <include>**/*</include>
  </includes>
  <filtering>false</filtering>
</resource>

增加resources目录

复制/d/workspace/dubbo-demo-service/dubbo-server/src/main

1
2
3
 
$ mkdir -pv resources/META-INF
mkdir: created directory 'resources'
mkdir: created directory 'resources/META-INF'

修改config.properties文件

复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/config.properties

1
2
 
dubbo.registry=${dubbo.registry}
dubbo.port=${dubbo.port}

修改srping-config.xml文件

  • beans段新增属性
复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml

1
 
xmlns:apollo="http://www.ctrip.com/schema/apollo"
  • xsi:schemaLocation段内新增属性
复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml

1
 
http://www.ctrip.com/schema/apollo http://www.ctrip.com/schema/apollo.xsd
  • 新增配置项
复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml

1
 
<apollo:config/>
  • 删除配置项（注释）
复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/spring-config.xml

1
 
<!-- <context:property-placeholder location="classpath:config.properties"/> -->

增加app.properties文件

复制/d/workspace/dubbo-demo-service/dubbo-server/src/main/resources/META-INF/app.properties

1
 
app.id=dubbo-demo-service

提交git中心仓库（gitee）

复制

1
 
$ git push origin apollo

配置apollo-portal

创建项目

  • 部门

    样例部门1（TEST1）

  • 应用id

    dubbo-demo-service

  • 应用名称

    dubbo服务提供者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

新增配置项2

  • Key

    dubbo.port

  • Value

    20880

  • 选择集群

    DEV

提交

发布配置

点击发布，配置生效

使用jenkins进行CI

略（注意记录镜像的tag）

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上：

复制/data/k8s-yaml/dubbo-demo-service/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels: 
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-service
  template:
    metadata:
      labels: 
        app: dubbo-demo-service
        name: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: harbor.od.com/app/dubbo-demo-service:apollo_190119_1815
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

注意：增加了env段配置
注意：docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行：

复制

1
2
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-service/deployment.yaml
deployment.extensions/dubbo-demo-service configured

观察项目运行情况

http://dubbo-monitor.od.com

改造dubbo-demo-web

略

配置apollo-portal

创建项目

  • 部门

    样例部门1（TEST1）

  • 应用id

    dubbo-demo-web

  • 应用名称

    dubbo服务消费者

  • 应用负责人

    apollo|apollo

  • 项目管理员

    apollo|apollo

提交

进入配置页面

新增配置项1

  • Key

    dubbo.registry

  • Value

    zookeeper://zk1.od.com:2181

  • 选择集群

    DEV

提交

发布配置

点击发布，配置生效

使用jenkins进行CI

略（注意记录镜像的tag）

上线新构建的项目

准备资源配置清单

运维主机HDSS7-200.host.com上：

复制/data/k8s-yaml/dubbo-demo-consumer/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels: 
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels: 
      name: dubbo-demo-consumer
  template:
    metadata:
      labels: 
        app: dubbo-demo-consumer
        name: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: harbor.od.com/app/dubbo-demo-consumer:apllo_190120_1815
        ports:
        - containerPort: 20880
          protocol: TCP
        - containerPort: 8080
          protocol: TCP
        env:
        - name: C_OPTS
          value: -Denv=dev -Dapollo.meta=http://config.od.com
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext: 
        runAsUser: 0
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate: 
      maxUnavailable: 1
      maxSurge: 1
  revisionHistoryLimit: 7
  progressDeadlineSeconds: 600

注意：增加了env段配置
注意：docker镜像新版的tag

应用资源配置清单

在任意一台k8s运算节点上执行：

复制

1
2
 
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/dubbo-demo-web/deployment.yaml
deployment.extensions/dubbo-demo-consumer configured

通过Apollo配置中心动态维护项目的配置

以dubbo-demo-service项目为例，不用修改代码

实战维护多套dubbo微服务环境

生产实践

  1. 迭代新需求/修复BUG（编码->提GIT）
  2. 测试环境发版，测试（应用通过编译打包发布至TEST命名空间）
  3. 测试通过，上线（应用镜像直接发布至PROD命名空间）

系统架构

  • 物理架构
主机名 角色 ip
HDSS7-11.host.com zk-test(测试环境Test) 10.4.7.11
HDSS7-12.host.com zk-prod(生产环境Prod) 10.4.7.12
HDSS7-21.host.com kubernetes运算节点 10.4.7.21
HDSS7-22.host.com kubernetes运算节点 10.4.7.22
HDSS7-200.host.com 运维主机，harbor仓库 10.4.7.200
  • K8S内系统架构
环境 命名空间 应用
测试环境（TEST） test apollo-config，apollo-admin
测试环境（TEST） test dubbo-demo-service，dubbo-demo-web
生产环境（PROD） prod apollo-config，apollo-admin
生产环境（PROD） prod dubbo-demo-service，dubbo-demo-web
ops环境（infra） infra jenkins，dubbo-monitor，apollo-portal

修改/添加域名解析

DNS主机HDSS7-11.host.com上：

复制/var/named/od.com.zone

1
2
3
4
5
6
 
zk-test 60 IN A 10.4.7.11
zk-prod 60 IN A 10.4.7.12
config-test	60 IN A 10.4.7.10
config-prod	60 IN A 10.4.7.10
demo-test 60 IN A 10.4.7.10
demo-prod 60 IN A 10.4.7.10

Apollo的k8s应用配置

  • 删除app命名空间内应用，创建test命名空间，创建prod命名空间
  • 删除infra命名空间内apollo-configservice，apollo-adminservice应用
  • 数据库内删除ApolloConfigDB，创建ApolloConfigTestDB，创建ApolloConfigProdDB
复制

1
2
3
4
5
6
7
8
9
10
11
12
13
 
mysql> drop database ApolloConfigDB;

mysql> create database ApolloConfigTestDB;
mysql> use ApolloConfigTestDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigTestDB.ServerConfig set ServerConfig.Value="http://config-test.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigTestDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";

mysql> create database ApolloConfigProdDB;
mysql> use ApolloConfigProdDB;
mysql> source ./apolloconfig.sql
mysql> update ApolloConfigProdDB.ServerConfig set ServerConfig.Value="http://config-prod.od.com/eureka" where ServerConfig.Key="eureka.service.url";
mysql> grant INSERT,DELETE,UPDATE,SELECT on ApolloConfigProdDB.* to "apolloconfig"@"10.4.7.%" identified by "123456";
  • 准备apollo-config，apollo-admin的资源配置清单（各2套）

注：apollo-config/apollo-admin的configmap配置要点

  • Test环境
复制

1
2
3
4
5
6
 
application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigTestDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-test.od.com/eureka
  • Prod环境
复制

1
2
3
4
5
6
 
application-github.properties: |
    # DataSource
    spring.datasource.url = jdbc:mysql://mysql.od.com:3306/ApolloConfigProdDB?characterEncoding=utf8
    spring.datasource.username = apolloconfig
    spring.datasource.password = 123456
    eureka.service.url = http://config-prod.od.com/eureka
  • 依次应用，分别发布在test和prod命名空间
  • 修改apollo-portal的configmap并重启portal
复制

1
2
3
 
apollo-env.properties: |
    TEST.meta=http://config-test.od.com
    PROD.meta=http://config-prod.od.com

Apollo的portal配置

管理员工具

删除应用、集群、AppNamespace，将已配置应用删除

系统参数

  • Key

    apollo.portal.envs

  • Value

    TEST,PROD

查询

  • Value

    TEST,PROD

保存

新建dubbo-demo-service和dubbo-demo-web项目

在TEST/PROD环境分别增加配置项并发布

发布dubbo微服务

  • 准备dubbo-demo-service和dubbo-demo-web的资源配置清单（各2套）
  • 依次应用，分别发布至app-test和app-prod命名空间
  • 使用dubbo-monitor查验

互联网公司技术部的日常

  • 产品经理整理需求，需求评审，出产品原型
  • 开发同学夜以继日的开发，提测
  • 测试同学使用Jenkins持续集成，并发布至测试环境
  • 验证功能，通过->待上线or打回->修改代码
  • 提交发版申请，运维同学将测试后的包发往生产环境
  • 无尽的BUG修复（笑cry）

（转）实验文档3：在kubernetes集群里集成Apollo配置中心的更多相关文章

  1. 在kubernetes集群里集成Apollo配置中心(6)之实战使用apollo分环境管理dubbo服务

    生产实践 1.迭代新需求/修复BUG(编码--->提git) 2.测试环境发版,测试(应用通过编译打包发布至test命名空间) 3.测试通过,上线(应用镜像直接发布至prod命名空间) 系统架构 ...

  2. 在kubernetes集群里集成Apollo配置中心(1)之交付Apollo-configservice至Kubernetes集群

    1.Apollo简介 Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境.不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限.流程治理等特性,适用于微 ...

  3. 在kubernetes集群里集成Apollo配置中心(5)之dubbo服务消费者连接apollo实战

    1.在Apollo的portal创建dubbo消费者项目 (1)添加dubbo消费者项目 (2)在dubbo消费者项目中添加配置项 (3)发布 2.通过jenkins构建dubbo消费者镜像 3.登录 ...

  4. 在kubernetes集群里集成Apollo配置中心(4)之dubbo服务提供者连接apollo实战

    1.登录portal.od.com(Apollo-portal),新建一个dubbo-demo-service项目 2.在Apollo项目中的dubbo-demo-service添加配置 (1)添加d ...

  5. 在kubernetes集群里集成Apollo配置中心(3)之交付Apollo-portal至Kubernetes集群

    1.执行apollo-portal数据库脚本 apollo-portal数据库脚本链接:https://raw.githubusercontent.com/ctripcorp/apollo/1.5.1 ...

  6. 在kubernetes集群里集成Apollo配置中心(1)之交付Apollo-adminservice至Kubernetes集群

    1.部署apollo-adminservice软件包 apollo-adminservice软件包链接地址:https://github.com/ctripcorp/apollo/releases/d ...

  7. （转）实验文档4：kubernetes集群的监控和日志分析

    改造dubbo-demo-web项目为Tomcat启动项目 Tomcat官网 准备Tomcat的镜像底包 准备tomcat二进制包 运维主机HDSS7-200.host.com上:Tomcat8下载链 ...

  8. 基于zookeeper集群的云平台-配置中心的功能设计

    最近准备找工作面试,就研究了下基于zookeeper集群的配置中心. 下面是自己设想的关于开源的基于zookeeper集群的云平台-配置中心的功能设计.大家觉得哪里有问题,请提出宝贵的意见和建议,谢谢 ...

  9. Kubernetes集群搭建之Master配置篇

    本次系列使用的所需部署包版本都使用的目前最新的或最新稳定版,安装包地址请到公众号内回复[K8s实战]获取 今天终于到正题了~~ 生成kubernets证书与私钥 1. 制作kubernetes ca证 ...

随机推荐

  1. sql注入测试(1)---概念概述

    在进行java web 测试时,经常会忽略的测试种类就是sql注入测试,这类缺陷造成的原因是开发技术在这方面欠缺的表现,虽然不常见,但一旦有这类缺陷,就很因此对运营的数据造成很多不必要的损失,所以,还 ...

  2. SpringFramework5.0 @Indexed注解 简单解析

    目录 使用场景 使用方法 原理说明 使用需注意点 案例说明 参考资料 纸上得来终觉浅 绝知此事要躬行 -陆游 最近在看SpringBoot核编程思想(核心篇),看到走向注解驱动编程这章,里面有讲解到: ...

  3. myeclipse 出现换行符和空格符 解决方案 换行出现乱码

    请参看百度经验 https://jingyan.baidu.com/article/acf728fd2639e4f8e510a399.html myeclipse 2014 自定义视图Customiz ...

  4. windows开机自动执行bat脚本启动cmd命令窗口并执行命令，最后自动关闭cmd命令窗

    先说bat脚本吧,随便在桌面建一个文本文档,在里面写脚本,内容是,先启动cmd命令窗,并执行一个命令,最后自动关闭cmd窗,如下: ---------------------------------- ...

  5. Eclipse 设置代码模板

    checkstyle插件 安装 从 http://sourceforge.jp/projects/sfnet_eclipse-cs/releases/ 下载 net.sf.eclipsecs-upda ...

  6. stm32 按键操作

    抖动时间的长短由按键的机械特性决定,一般为5ms-10ms void key() { static u8 flag = 1; if(flag == 1 && KEY_UP == 1) ...

  7. require.js 加载 js 文件 404 处理(配置无效)

    main.js 是 配置文件,data-main 是异步加载,如果在main.js未加载完成的时候,使用了require去加载文件,就会导致配置无效  main.js

  8. Orion (system-on-a-chip)

      From Wikipedia: Orion is a system-on-a-chip manufactured by Marvell Technology Group and used in n ...

  9. Django：总结setting中的配置

    一.Django setting配置说明 二.setting配置一览 一.Django setting配置说明 1.基础 DJANGO_SETTING_MODULE环境变量:让settings模块被包 ...

  10. idou老师教你学Istio 14：如何用K8S对Istio Service进行流量健康检查

    Istio利用k8s的探针对service进行流量健康检查,有两种探针可供选择,分别是liveness和readiness: liveness探针用来侦测什么时候需要重启容器.比如说当liveness ...