Kubernetes-基于helm安装部署高可用的Redis及其形态探索（二）

上一章，我们通过实践和其他文章的帮助，在k8s的环境安装了redis-ha，并且对其进行了一些实验来验证他的主从切换是否有效。本篇中将会分析，究竟是如何实现了redis-ha的主从切换，以及其与K8S平面进行的交互。

1.如何实现的redis的搭建

我曾经以为是在helm/chart中写入了脚本来完成这件事，但是仔细看过代码之后，并未发现明显的内容，关于搭建redis-ha和主从切换的脚本。

地址：https://github.com/helm/charts/tree/master/stable/redis-ha

后来，通过查看redis镜像的日志发现了一些内容，

地址：https://quay.io/repository/smile/redis/manifest/sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df?tab=layers

可以看到，在这个镜像构建日志中，有几个疑似相关内容的文件，/usr/local/bin目录下的promte.sh,redis-launcher.sh,label-updater.sh

进入到pod中，我们可以看到redis-launcher是作为启动时就运行的脚本，所以我就推测这一切都是这个文件引起的。

2.脚本内容

redis-launcher.sh:

bash-4.4# cat /usr/local/bin/redis-launcher.sh
#!/bin/bash
# Copyright 2017 Ismail KABOUBI
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script determines whether the pod that executes it will be a Redis Sentinel, Master, or Slave
# The redis-ha Helm chart signals Sentinel status with environment variables. If they are not set, the newly
# launched pod will scan K8S to see if there is an active master. If not, it uses a deterministic means of
# sensing whether it should launch as master then writes master or slave to the label called redis-role
# appropriately. It's this label that determines which LB a pod can be seen through.
#
# The redis-role=master pod is the key for the cluster to get started. Sentinels will wait for it to appear
# in the LB before they finish launching. All other pods wait for the Sentinels to ID the master.
#
# Pods also set the labels podIP and runID. RunID is the first few characters of the unique run_id value
# generated by each Redis sever.
#
# During normal operation, there should be only one redis-role=master pod. If it fails, the Sentinels
# will nominate a new master and change all the redis-role values appropriately.

echo "Starting redis launcher"
echo "Setting labels"
label-updater.sh & plabeler=$!

echo "Selecting proper service to execute"
# Define config file locations
SENTINEL_CONF=/etc/redis/sentinel.conf
MASTER_CONF=/etc/redis/master.conf
SLAVE_CONF=/etc/redis/slave.conf

# Adapt to dynamically named env vars
ENV_VAR_PREFIX=`echo $REDIS_CHART_PREFIX|awk '{print toupper($0)}'|sed 's/-/_/g'`
PORTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_PORT"
HOSTVAR="${ENV_VAR_PREFIX}MASTER_SVC_SERVICE_HOST"
MASTER_LB_PORT="${!PORTVAR}"
MASTER_LB_HOST="${!HOSTVAR}"
QUORUM=${QUORUM:-2}

# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""

# Launch master when `MASTER` environment variable is set
function launchmaster() {
  # If we know we're a master, update the labels right away
  kubectl label --overwrite pod $HOSTNAME redis-role="master"
  echo "Using config file $MASTER_CONF"
  if [[ ! -e /redis-master-data ]]; then
    echo "Redis master data doesn't exist, data won't be persistent!"
    mkdir /redis-master-data
  fi

  if [ -n "$REDIS_PASS" ]; then
    sed -i "s/# requirepass/requirepass ${REDIS_PASS} \n#/" $MASTER_CONF
  fi

  redis-server $MASTER_CONF --protected-mode no $@
}

# Launch sentinel when `SENTINEL` environment variable is set
function launchsentinel() {
  # If we know we're a sentinel, update the labels right away
  kubectl label --overwrite pod $HOSTNAME redis-role="sentinel"
  echo "Using config file $SENTINEL_CONF"

  while true; do
    # The sentinels must wait for a load-balanced master to appear then ask it for its actual IP.
    MASTER_IP=$(kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX|awk '{print $2}'|xargs)
    echo "Current master is $MASTER_IP"

    if [[ -z ${MASTER_IP} ]]; then
      continue
    fi

    timeout -t 3 redis-cli ${REDISAUTH} -h ${MASTER_IP} -p ${MASTER_LB_PORT} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 10
  done

  echo "sentinel monitor mymaster ${MASTER_IP} ${MASTER_LB_PORT} ${QUORUM}" > ${SENTINEL_CONF}
  echo "sentinel down-after-milliseconds mymaster 15000" >> ${SENTINEL_CONF}
  echo "sentinel failover-timeout mymaster 30000" >> ${SENTINEL_CONF}
  echo "sentinel parallel-syncs mymaster 10" >> ${SENTINEL_CONF}
  echo "bind 0.0.0.0" >> ${SENTINEL_CONF}
  echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}

  if [ -n "$REDIS_PASS" ]; then
   echo "sentinel auth-pass mymaster ${REDIS_PASS}" >> ${SENTINEL_CONF}
  fi

  redis-sentinel ${SENTINEL_CONF} --protected-mode no $@
}

# Launch slave when `SLAVE` environment variable is set
function launchslave() {
  kubectl label --overwrite pod $HOSTNAME redis-role="slave"
  echo "Using config file $SLAVE_CONF"
  if [[ ! -e /redis-master-data ]]; then
    echo "Redis master data doesn't exist, data won't be persistent!"
    mkdir /redis-master-data
  fi

  i=0
  while true; do
    master=${MASTER_LB_HOST}
    timeout -t 3 redis-cli ${REDISAUTH} -h ${master} -p ${MASTER_LB_PORT} INFO
    if [[ "$?" == "0" ]]; then
      break
    fi
    i=$((i+1))
    if [[ "$i" -gt "30" ]]; then
      echo "Exiting after too many attempts"
      kill $plabeler
      exit 1
    fi
    echo "Connecting to master failed.  Waiting..."
    sleep 1
  done

  if [ -n "$REDIS_PASS" ]; then
    sed -i "s/# masterauth/masterauth ${REDIS_PASS} \n#/" $SLAVE_CONF
    sed -i "s/# requirepass/requirepass ${REDIS_PASS} \n#/" $SLAVE_CONF
  fi

  sed -i "s/%master-ip%/${MASTER_LB_HOST}/" $SLAVE_CONF
  sed -i "s/%master-port%/${MASTER_LB_PORT}/" $SLAVE_CONF
  redis-server $SLAVE_CONF --protected-mode no $@
}

#Check if MASTER environment variable is set
if [[ "${MASTER}" == "true" ]]; then
  echo "Launching Redis in Master mode"
  launchmaster
  exit 0
fi

# Check if SENTINEL environment variable is set
if [[ "${SENTINEL}" == "true" ]]; then
  echo "Launching Redis Sentinel"
  launchsentinel
  echo "Launcsentinel action completed"
  exit 0
fi

# Determine whether this should be a master or slave instance
echo "Looking for pods running as master"
MASTERS=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master|grep running|grep $REDIS_CHART_PREFIX`
if [[ "$MASTERS" == "" ]]; then
  echo "No masters found: \"$MASTERS\" Electing first master..."
  SLAVE1=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.creationTimestamp} {.metadata.name} {.status.containerStatuses[0].state} {"\n"} {end}' -l redis-node=true |grep running|sort|awk '{print $2}'|grep $REDIS_CHART_PREFIX|head -n1`
  if [[ "$SLAVE1" == "$HOSTNAME" ]] || [[ "$SLAVE1" == "" ]]; then
    echo "Taking master role"
    launchmaster
  else
    echo "Electing $SLAVE1 master"
    launchslave
  fi
  exit 0
else
  echo "Found $MASTERS"
  echo "Launching Redis in Slave mode"
  launchslave
  exit 0
fi

echo "Launching Redis in Slave mode"
launchslave
echo "Launchslave action completed"

label-updater.sh

bash-4.4# cat /usr/local/bin/label-updater.sh
# Push some helpful vars into labels
PODIP=`hostname -i`
echo podIP $PODIP
kubectl label --overwrite pod $HOSTNAME podIP="$PODIP"

if [ "$SENTINEL" ]; then
    exit
fi

RUNID=""

# Only sets AUTH if the ENV var REDIS_PASS is set.
REDISAUTH=""
[ -n "$REDIS_PASS" ] && REDISAUTH="-a $REDIS_PASS" || REDISAUTH=""

while true; do
  RUNID=`redis-cli $REDISAUTH info server |grep run_id|awk -F: '{print $2}'|head -c6`
  if [ -n "$RUNID" ]; then
    kubectl label --overwrite pod $HOSTNAME runID="$RUNID"
    break
  else
    sleep 1
  fi
done

　　

promote.sh

bash-4.4# cat /usr/local/bin/promote.sh
#!/usr/bin/env bash
MASTERIP=$6

# Convert the IP of the promoted pod to a hostname
MASTERPOD=`kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {..podIP} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=slave --sort-by=.metadata.name|grep running|grep $MASTERIP|awk '{print $1}'`
echo "PROMO ARGS: $@"
echo "PROMOTING $MASTERPOD ($MASTERIP) TO MASTER"
kubectl label --overwrite pod $MASTERPOD redis-role="master"

# Demote anyone else who jumped to master
kubectl get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[0].state}{"\n"}{end}' -l redis-role=master --sort-by=.metadata.name|grep running|awk '{print $1}'|grep $REDIS_CHART_PREFIX|grep -v $MASTERPOD|xargs -n1 -I% kubectl label --overwrite pod % redis-role="slave"
echo "OTHER MASTERS $MASTERS"

　　

3.大致原理

详细的内容我还没有开始看，但是可以讲一下大致的原理，就是在每个pod在启动的时候都会起这样的一个redis-launcher的进程，这个就像一个agent一样，主要完成redis的master，slave和sentinel的配置，同时，他会将各个pod的角色反向通过kubectl命令传给K8S平面。

如果是发生了主从结构已经起来了，但是中途因为某种原因挂掉了，则会通过监控sentinel的状态来触发更改K8S平面pod的Role的过程。这个设定是在启动sentinel完成的，代码在这里：

  echo "sentinel client-reconfig-script mymaster /usr/local/bin/promote.sh" >> ${SENTINEL_CONF}

他会监控，如果对于mymaster这个集群中的sentinel发生了reconfig的事件的时候，就会去触发/usr/local/bin/promote.sh这个脚本。

所以可以看到，是redis的pod在控制，而不是K8S平面在进行控制。之后有时间，我会详细的读一下这个脚本，然后加上一些注释。

更多openstack/trove的文章：http://www.cnblogs.com/S-tec-songjian/

此文章属博客园用户S-tec原创作品，受国家《著作权法》保护，未经许可，任何单位及个人不得做营利性使用；若仅做个人学习、交流等非营利性使用，应当指明作者姓名、作品名称，原文地址，并且不得侵犯作者依法享有的其他权利。