Redis源码解析:23sentinel(四)故障转移流程
十:故障转移流程中的状态转换
当哨兵针对某个主节点进行故障转移时,该主节点的故障转移状态master->failover_state,要依次经历下面六个状态:
SENTINEL_FAILOVER_STATE_WAIT_START
SENTINEL_FAILOVER_STATE_SELECT_SLAVE
SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE
SENTINEL_FAILOVER_STATE_WAIT_PROMOTION
SENTINEL_FAILOVER_STATE_RECONF_SLAVES
SENTINEL_FAILOVER_STATE_UPDATE_CONFIG
在哨兵的“主函数”sentinelHandleRedisInstance中,通过sentinelFailoverStateMachine函数进行故障转移状态的转换。它的代码如下:
void sentinelFailoverStateMachine(sentinelRedisInstance *ri) {
redisAssert(ri->flags & SRI_MASTER); if (!(ri->flags & SRI_FAILOVER_IN_PROGRESS)) return; switch(ri->failover_state) {
case SENTINEL_FAILOVER_STATE_WAIT_START:
sentinelFailoverWaitStart(ri);
break;
case SENTINEL_FAILOVER_STATE_SELECT_SLAVE:
sentinelFailoverSelectSlave(ri);
break;
case SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE:
sentinelFailoverSendSlaveOfNoOne(ri);
break;
case SENTINEL_FAILOVER_STATE_WAIT_PROMOTION:
sentinelFailoverWaitPromotion(ri);
break;
case SENTINEL_FAILOVER_STATE_RECONF_SLAVES:
sentinelFailoverReconfNextSlave(ri);
break;
}
}
下面分别讲解每个状态及其处理函数。
1:SENTINEL_FAILOVER_STATE_WAIT_START
上一章讲过,在哨兵的“主函数”sentinelHandleRedisInstance中,调用sentinelStartFailoverIfNeeded函数判断是否可以开始一次故障转移流程。当条件满足后,就会调用sentinelStartFailover函数,开始新一轮的故障转移流程。在该函数中,就会将该主节点的故障转移状态置为SENTINEL_FAILOVER_STATE_WAIT_START。
一旦哨兵开始一次故障转移流程时,该哨兵第一件事就是向其他所有哨兵发送”is-master-down-by-addr”命令进行拉票。然后就是调用sentinelFailoverWaitStart函数处理当前状态。
sentinelFailoverWaitStart函数的代码如下:
void sentinelFailoverWaitStart(sentinelRedisInstance *ri) {
char *leader;
int isleader; /* Check if we are the leader for the failover epoch. */
leader = sentinelGetLeader(ri, ri->failover_epoch);
isleader = leader && strcasecmp(leader,server.runid) == 0;
sdsfree(leader); /* If I'm not the leader, and it is not a forced failover via
* SENTINEL FAILOVER, then I can't continue with the failover. */
if (!isleader && !(ri->flags & SRI_FORCE_FAILOVER)) {
int election_timeout = SENTINEL_ELECTION_TIMEOUT; /* The election timeout is the MIN between SENTINEL_ELECTION_TIMEOUT
* and the configured failover timeout. */
if (election_timeout > ri->failover_timeout)
election_timeout = ri->failover_timeout;
/* Abort the failover if I'm not the leader after some time. */
if (mstime() - ri->failover_start_time > election_timeout) {
sentinelEvent(REDIS_WARNING,"-failover-abort-not-elected",ri,"%@");
sentinelAbortFailover(ri);
}
return;
}
sentinelEvent(REDIS_WARNING,"+elected-leader",ri,"%@");
ri->failover_state = SENTINEL_FAILOVER_STATE_SELECT_SLAVE;
ri->failover_state_change_time = mstime();
sentinelEvent(REDIS_WARNING,"+failover-state-select-slave",ri,"%@");
}
当前哨兵,在调用sentinelStartFailover函数发起故障转移流程时,会将当前选举纪元sentinel.current_epoch记录到ri->failover_epoch中。因此,本函数首先根据ri->failover_epoch,调用函数sentinelGetLeader得到本界选举的结果leader。如果本界选举尚无人获得超过半数的选票,则leader为NULL;
如果当前哨兵还没有赢得选举,并且主节点标志位中没有设置SRI_FORCE_FAILOVER标记,说明当前哨兵还没有获得足够的选票,暂时不能继续进行接下来的故障转移流程,需要直接返回。
但是如果超过一定时间之后,当前哨兵还是没有赢得选举,则会终止当前的故障转移流程,因此如果当前距离开始故障转移的时间超过election_timeout,则调用函数sentinelAbortFailover,终止本次故障转移流程。
如果当前哨兵最终赢得了选举,则更新故障转移的状态,置ri->failover_state属性为下一个状态:SENTINEL_FAILOVER_STATE_SELECT_SLAVE,并更新ri->failover_state_change为当前时间;
2:SENTINEL_FAILOVER_STATE_SELECT_SLAVE
当故障转移状态转换为SENTINEL_FAILOVER_STATE_SELECT_SLAVE时,就需要在下线主节点的所有下属从节点中,按照一定的规则,选择一个从节点使其成为新的主节点。
该状态下的处理函数为sentinelFailoverSelectSlave,该函数的代码如下:
void sentinelFailoverSelectSlave(sentinelRedisInstance *ri) {
sentinelRedisInstance *slave = sentinelSelectSlave(ri); /* We don't handle the timeout in this state as the function aborts
* the failover or go forward in the next state. */
if (slave == NULL) {
sentinelEvent(REDIS_WARNING,"-failover-abort-no-good-slave",ri,"%@");
sentinelAbortFailover(ri);
} else {
sentinelEvent(REDIS_WARNING,"+selected-slave",slave,"%@");
slave->flags |= SRI_PROMOTED;
ri->promoted_slave = slave;
ri->failover_state = SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE;
ri->failover_state_change_time = mstime();
sentinelEvent(REDIS_NOTICE,"+failover-state-send-slaveof-noone",
slave, "%@");
}
}
该函数首先调用函数sentinelSelectSlave选择一个符合条件的从节点;
如果没有合适的从节点,则调用sentinelAbortFailover直接终止本次故障转移流程;
如果找到了合适的从节点slave,则首先将标记SRI_PROMOTED增加到该从节点的标志位中;并使主节点实例的ri->promoted_slave指针指向该从节点实例,并将故障转移状态置为SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE;然后更新ri->failover_state_change_time为当前时间;
函数sentinelSelectSlave用于在下线主节点的所有从节点实例中,按照一定的规则选择一个从节点。该函数的代码如下:
sentinelRedisInstance *sentinelSelectSlave(sentinelRedisInstance *master) {
sentinelRedisInstance **instance =
zmalloc(sizeof(instance[0])*dictSize(master->slaves));
sentinelRedisInstance *selected = NULL;
int instances = 0;
dictIterator *di;
dictEntry *de;
mstime_t max_master_down_time = 0; if (master->flags & SRI_S_DOWN)
max_master_down_time += mstime() - master->s_down_since_time;
max_master_down_time += master->down_after_period * 10; di = dictGetIterator(master->slaves);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *slave = dictGetVal(de);
mstime_t info_validity_time; if (slave->flags & (SRI_S_DOWN|SRI_O_DOWN|SRI_DISCONNECTED)) continue;
if (mstime() - slave->last_avail_time > SENTINEL_PING_PERIOD*5) continue;
if (slave->slave_priority == 0) continue; /* If the master is in SDOWN state we get INFO for slaves every second.
* Otherwise we get it with the usual period so we need to account for
* a larger delay. */
if (master->flags & SRI_S_DOWN)
info_validity_time = SENTINEL_PING_PERIOD*5;
else
info_validity_time = SENTINEL_INFO_PERIOD*3;
if (mstime() - slave->info_refresh > info_validity_time) continue;
if (slave->master_link_down_time > max_master_down_time) continue;
instance[instances++] = slave;
}
dictReleaseIterator(di);
if (instances) {
qsort(instance,instances,sizeof(sentinelRedisInstance*),
compareSlavesForPromotion);
selected = instance[0];
}
zfree(instance);
return selected;
}
首先创建数组instance,它将用于保存所有状态良好的从节点;
然后计算max_master_down_time,他表示所允许的从节点与主节点断链时间的最大值。它的值是主节点客观下线的时间加上10倍的master->down_after_period的值:
接下来,轮训字典master->slaves,针对其中的每一个从节点,判断其状态是否良好。从节点状态良好的条件是:
从节点没有处于主观下线、客观下线或者断链状态;
距离上一次收到该从节点对于"PING"命令的正常回复的时间,不超过5倍的SENTINEL_PING_PERIOD;
该从节点的优先级不是0;
距离上一次收到该从节点对于"INFO"命令的回复的时间,不超过3倍或5倍(根据主节点是否客观下线而定)的SENTINEL_PING_PERIOD;
从节点与主节点的断链时间(该时间值根据从节点的"INFO"命令回复中得到)不超过max_master_down_time;
满足以上条件的从节点,就认为是状态良好的从节点,将其记录到数组instance中;
所有从节点都轮训完毕之后,使用qsort快速排序算法,对数组instance进行排序。这里使用的比较函数compareSlavesForPromotion;排好序的instance数组,状态越好的从节点,其位置越靠前,因此,返回instance[0]作为选中的从节点;
下面就是快速排序算法中,使用的比较函数compareSlavesForPromotion的代码:
int compareSlavesForPromotion(const void *a, const void *b) {
sentinelRedisInstance **sa = (sentinelRedisInstance **)a,
**sb = (sentinelRedisInstance **)b;
char *sa_runid, *sb_runid; if ((*sa)->slave_priority != (*sb)->slave_priority)
return (*sa)->slave_priority - (*sb)->slave_priority; /* If priority is the same, select the slave with greater replication
* offset (processed more data frmo the master). */
if ((*sa)->slave_repl_offset > (*sb)->slave_repl_offset) {
return -1; /* a < b */
} else if ((*sa)->slave_repl_offset < (*sb)->slave_repl_offset) {
return 1; /* b > a */
} /* If the replication offset is the same select the slave with that has
* the lexicographically smaller runid. Note that we try to handle runid
* == NULL as there are old Redis versions that don't publish runid in
* INFO. A NULL runid is considered bigger than any other runid. */
sa_runid = (*sa)->runid;
sb_runid = (*sb)->runid;
if (sa_runid == NULL && sb_runid == NULL) return 0;
else if (sa_runid == NULL) return 1; /* a > b */
else if (sb_runid == NULL) return -1; /* a < b */
return strcasecmp(sa_runid, sb_runid);
}
该函数用于比较两个从节点的状态:如果a的状态要好于b,则返回-1,表示a小于b,否则返回0或1,表示a等于或大于b;
首先比较a和b的优先级:优先级越小(0除外),则状态越好;如果a和b的优先级相同,则比较它们的复制偏移量:复制偏移量越大,则状态越好;
如果以上的比较结果都是相同的,则比较a和b的运行ID的字母循序,另外如果某个从节点的运行ID为NULL,则它的状态更差。
3:SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE
当选择好一个从节点之后,接下来在状态为SENTINEL_FAILOVER_STATE_SEND_SLAVEOF_NOONE时,要做的就是向该从节点发送”SLAVEOF NO ONE”命令。
该状态下的处理函数为sentinelFailoverSendSlaveOfNoOne,该函数的代码如下:
void sentinelFailoverSendSlaveOfNoOne(sentinelRedisInstance *ri) {
int retval; /* We can't send the command to the promoted slave if it is now
* disconnected. Retry again and again with this state until the timeout
* is reached, then abort the failover. */
if (ri->promoted_slave->flags & SRI_DISCONNECTED) {
if (mstime() - ri->failover_state_change_time > ri->failover_timeout) {
sentinelEvent(REDIS_WARNING,"-failover-abort-slave-timeout",ri,"%@");
sentinelAbortFailover(ri);
}
return;
} /* Send SLAVEOF NO ONE command to turn the slave into a master.
* We actually register a generic callback for this command as we don't
* really care about the reply. We check if it worked indirectly observing
* if INFO returns a different role (master instead of slave). */
retval = sentinelSendSlaveOf(ri->promoted_slave,NULL,0);
if (retval != REDIS_OK) return;
sentinelEvent(REDIS_NOTICE, "+failover-state-wait-promotion",
ri->promoted_slave,"%@");
ri->failover_state = SENTINEL_FAILOVER_STATE_WAIT_PROMOTION;
ri->failover_state_change_time = mstime();
}
代码很简单。首先如果选中的从节点当前处于断链状态,则因无法向其发送命令,因此直接返回;如果该状态已经持续超过ri->failover_timeout的时间,则调用函数sentinelAbortFailover终止本次故障转移流程;
然后调用sentinelSendSlaveOf函数,向从节点发送"SLAVEOF NO ONE"命令,然后置故障转移状态为SENTINEL_FAILOVER_STATE_WAIT_PROMOTION,并且更新ri->failover_state_change_time为当前时间;
函数sentinelSendSlaveOf用于发送”SLAVEOF”命令,它的代码如下:
int sentinelSendSlaveOf(sentinelRedisInstance *ri, char *host, int port) {
char portstr[32];
int retval; ll2string(portstr,sizeof(portstr),port); /* If host is NULL we send SLAVEOF NO ONE that will turn the instance
* into a master. */
if (host == NULL) {
host = "NO";
memcpy(portstr,"ONE",4);
} /* In order to send SLAVEOF in a safe way, we send a transaction performing
* the following tasks:
* 1) Reconfigure the instance according to the specified host/port params.
* 2) Rewrite the configuraiton.
* 3) Disconnect all clients (but this one sending the commnad) in order
* to trigger the ask-master-on-reconnection protocol for connected
* clients.
*
* Note that we don't check the replies returned by commands, since we
* will observe instead the effects in the next INFO output. */
retval = redisAsyncCommand(ri->cc,
sentinelDiscardReplyCallback, NULL, "MULTI");
if (retval == REDIS_ERR) return retval;
ri->pending_commands++; retval = redisAsyncCommand(ri->cc,
sentinelDiscardReplyCallback, NULL, "SLAVEOF %s %s", host, portstr);
if (retval == REDIS_ERR) return retval;
ri->pending_commands++; retval = redisAsyncCommand(ri->cc,
sentinelDiscardReplyCallback, NULL, "CONFIG REWRITE");
if (retval == REDIS_ERR) return retval;
ri->pending_commands++; /* CLIENT KILL TYPE <type> is only supported starting from Redis 2.8.12,
* however sending it to an instance not understanding this command is not
* an issue because CLIENT is variadic command, so Redis will not
* recognized as a syntax error, and the transaction will not fail (but
* only the unsupported command will fail). */
retval = redisAsyncCommand(ri->cc,
sentinelDiscardReplyCallback, NULL, "CLIENT KILL TYPE normal");
if (retval == REDIS_ERR) return retval;
ri->pending_commands++; retval = redisAsyncCommand(ri->cc,
sentinelDiscardReplyCallback, NULL, "EXEC");
if (retval == REDIS_ERR) return retval;
ri->pending_commands++; return REDIS_OK;
}
如果参数host为NULL,则需要发送"SLAVEOF NO ONE"命令,否则,"SLAVEOF"后跟具体的ip和port信息;
为了安全的发送"SLAVEOF"命令,这里使用事务的方式进行发送。首先发送"MULTI"命令;然后发送"SLAVEOF"命令;然后发送"CONFIG REWRITE"命令,这样从节点会重写配置文件;然后发送"CLIENT KILL TYPEnormal"命令,从节点收到该命令后,会断开所有与之连接的normal客户端,包括与所有哨兵的命令连接;最后发送"EXEC"命令;
以上的命令,都不关心它们的回复,而是会在该实例的"INFO"命令回复中判断命令的执行结果;
4:SENTINEL_FAILOVER_STATE_WAIT_PROMOTION
该状态下的处理函数为sentinelFailoverWaitPromotion,代码如下:
void sentinelFailoverWaitPromotion(sentinelRedisInstance *ri) {
/* Just handle the timeout. Switching to the next state is handled
* by the function parsing the INFO command of the promoted slave. */
if (mstime() - ri->failover_state_change_time > ri->failover_timeout) {
sentinelEvent(REDIS_WARNING,"-failover-abort-slave-timeout",ri,"%@");
sentinelAbortFailover(ri);
}
}
本函数中,只是判断处于SENTINEL_FAILOVER_STATE_WAIT_PROMOTION状态的时间是否超过了阈值ri->failover_timeout。如果确实已经超过了,则调用函数sentinelAbortFailover终止本次故障转移流程;
从节点执行完"SLAVEOF NO ONE"命令之后,会在其发送的"INFO"命令回复中体现出来。因此相应的状态转换动作也就在"INFO"回复的回调函数sentinelRefreshInstanceInfo中执行。
在sentinelRefreshInstanceInfo中,处理这部分的代码为:
void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
...
/* Handle slave -> master role switch. */
if ((ri->flags & SRI_SLAVE) && role == SRI_MASTER) {
/* If this is a promoted slave we can change state to the
* failover state machine. */
if ((ri->flags & SRI_PROMOTED) &&
(ri->master->flags & SRI_FAILOVER_IN_PROGRESS) &&
(ri->master->failover_state ==
SENTINEL_FAILOVER_STATE_WAIT_PROMOTION))
{
/* Now that we are sure the slave was reconfigured as a master
* set the master configuration epoch to the epoch we won the
* election to perform this failover. This will force the other
* Sentinels to update their config (assuming there is not
* a newer one already available). */
ri->master->config_epoch = ri->master->failover_epoch;
ri->master->failover_state = SENTINEL_FAILOVER_STATE_RECONF_SLAVES;
ri->master->failover_state_change_time = mstime();
sentinelFlushConfig();
sentinelEvent(REDIS_WARNING,"+promoted-slave",ri,"%@");
sentinelEvent(REDIS_WARNING,"+failover-state-reconf-slaves",
ri->master,"%@");
sentinelCallClientReconfScript(ri->master,SENTINEL_LEADER,
"start",ri->master->addr,ri->addr);
sentinelForceHelloUpdateForMaster(ri->master);
}
...
}
...
}
属性ri->flags表示该实例原来的类型,而role表示该实例在”INFO”命令回复中,报告的自己当前的角色。
如果ri->flags中为从节点,但是role为主节点。这种情况下:如果当前实例确实是哨兵在进行故障转移流程中选中的新主节点,并且目前的故障转移状态为SENTINEL_FAILOVER_STATE_WAIT_PROMOTION,说明已经向其发送了"SLAVEOF NO ONE",这里收到该节点的"INFO"回复中,它已经报告自己为主节点,因此"SLAVEOF"命令执行成功了。
因此:更新ri->master中的config_epoch属性值,更新故障迁移状态为SENTINEL_FAILOVER_STATE_RECONF_SLAVES,更新failover_state_change_time属性为当前时间;并且更新配置文件,记录日志,发布消息,调用sentinelForceHelloUpdateForMaster函数,强制引发向所有实例节点发送"PUBLISH"命令;
5:SENTINEL_FAILOVER_STATE_RECONF_SLAVES
当故障转移状态变为SENTINEL_FAILOVER_STATE_RECONF_SLAVES时,选中的从节点已经升级为主节点,接下来要做的就是向其他从节点发送”SLAVEOF”命令,使它们与新的主节点进行同步。
该状态下的处理函数是sentinelFailoverReconfNextSlave,代码如下:
void sentinelFailoverReconfNextSlave(sentinelRedisInstance *master) {
dictIterator *di;
dictEntry *de;
int in_progress = 0; di = dictGetIterator(master->slaves);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *slave = dictGetVal(de); if (slave->flags & (SRI_RECONF_SENT|SRI_RECONF_INPROG))
in_progress++;
}
dictReleaseIterator(di); di = dictGetIterator(master->slaves);
while(in_progress < master->parallel_syncs &&
(de = dictNext(di)) != NULL)
{
sentinelRedisInstance *slave = dictGetVal(de);
int retval; /* Skip the promoted slave, and already configured slaves. */
if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE)) continue; /* If too much time elapsed without the slave moving forward to
* the next state, consider it reconfigured even if it is not.
* Sentinels will detect the slave as misconfigured and fix its
* configuration later. */
if ((slave->flags & SRI_RECONF_SENT) &&
(mstime() - slave->slave_reconf_sent_time) >
SENTINEL_SLAVE_RECONF_TIMEOUT)
{
sentinelEvent(REDIS_NOTICE,"-slave-reconf-sent-timeout",slave,"%@");
slave->flags &= ~SRI_RECONF_SENT;
slave->flags |= SRI_RECONF_DONE;
} /* Nothing to do for instances that are disconnected or already
* in RECONF_SENT state. */
if (slave->flags & (SRI_DISCONNECTED|SRI_RECONF_SENT|SRI_RECONF_INPROG))
continue; /* Send SLAVEOF <new master>. */
retval = sentinelSendSlaveOf(slave,
master->promoted_slave->addr->ip,
master->promoted_slave->addr->port);
if (retval == REDIS_OK) {
slave->flags |= SRI_RECONF_SENT;
slave->slave_reconf_sent_time = mstime();
sentinelEvent(REDIS_NOTICE,"+slave-reconf-sent",slave,"%@");
in_progress++;
}
}
dictReleaseIterator(di); /* Check if all the slaves are reconfigured and handle timeout. */
sentinelFailoverDetectEnd(master);
}
因为从节点在与主节点进行同步时,有可能无法响应客户端的查询。因此为了避免过多从节点因为同步而无法响应的问题,一个时间段内,最多只能允许master->parallel_syncs个从节点正在进行同步操作;
因此,首先轮训字典master->slaves,统计当前正在进行同步的从节点之和;只要从节点标志位中设置了SRI_RECONF_SENT或者SRI_RECONF_INPROG标记,就说明该从节点正在进行同步,将计数器in_progress加1;
接下来,只要in_progress还没超过master->parallel_syncs,就轮训字典master->slaves,向尚未发送过"SLAVEOF"命令的从节点发送该命令。在轮训过程中:
如果该从节点实例的标志位中设置了SRI_PROMOTED,说明它是"我"选中的新的主节点,因此直接跳过;
如果该从节点实例的标志位中设置了SRI_RECONF_DONE,说明该从节点已经完成了同步,因此直接跳过;
如果从节点处于SRI_RECONF_SENT状态的时间已经超过了SENTINEL_SLAVE_RECONF_TIMEOUT,则将该从节点的状态直接置为SRI_RECONF_DONE,当做其已经完成了同步。后续收到该从节点的"INFO"回复时,如果信息不正确,到时候会采取相应的动作;
如果从节点实例已经断链,则直接跳过;
如果从节点实例的状态为SRI_RECONF_SENT或SRI_RECONF_INPROG,说明该从节点正在进行同步,直接跳过;
经过以上判断之后,剩下的从节点就是还没有发送过"SLAVEOF"命令的节点,因此调用sentinelSendSlaveOf函数向其发送命令,发送成功之后,将其状态置为SRI_RECONF_SENT;
在函数的最后,调用函数sentinelFailoverDetectEnd,检查是否所有从节点实例都已经完成了同步;
在向从节点发送”SLAVEOF”命令之后,该从节点实例的状态会经过SRI_RECONF_SENT、SRI_RECONF_INPROG和SRI_RECONF_DONE这三种状态的转换。
当向从节点发送完”SLAVEOF”命令之后,该从节点实例的状态为SRI_RECONF_SENT,剩下的状态转换是根据该从节点发来的”INFO”命令回复中的信息进行判断的。
在收到从节点的”INFO”命令回复的回调函数sentinelRefreshInstanceInfo中,处理这部分的代码如下:
void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
...
/* Detect if the slave that is in the process of being reconfigured
* changed state. */
if ((ri->flags & SRI_SLAVE) && role == SRI_SLAVE &&
(ri->flags & (SRI_RECONF_SENT|SRI_RECONF_INPROG)))
{
/* SRI_RECONF_SENT -> SRI_RECONF_INPROG. */
if ((ri->flags & SRI_RECONF_SENT) &&
ri->slave_master_host &&
strcmp(ri->slave_master_host,
ri->master->promoted_slave->addr->ip) == 0 &&
ri->slave_master_port == ri->master->promoted_slave->addr->port)
{
ri->flags &= ~SRI_RECONF_SENT;
ri->flags |= SRI_RECONF_INPROG;
sentinelEvent(REDIS_NOTICE,"+slave-reconf-inprog",ri,"%@");
} /* SRI_RECONF_INPROG -> SRI_RECONF_DONE */
if ((ri->flags & SRI_RECONF_INPROG) &&
ri->slave_master_link_status == SENTINEL_MASTER_LINK_STATUS_UP)
{
ri->flags &= ~SRI_RECONF_INPROG;
ri->flags |= SRI_RECONF_DONE;
sentinelEvent(REDIS_NOTICE,"+slave-reconf-done",ri,"%@");
}
}
}
如果该从节点标志位中设置了SRI_RECONF_SENT标记,并且它的"INFO"回复中"master_host:"和"master_port:"的信息与新主节点的ip和port相同,则将从节点标志中的SRI_RECONF_SENT标记清除,并增加SRI_RECONF_INPROG标记;
如果该从节点的标志位中设置了SRI_RECONF_INPROG标记,并且它的"INFO"回复中的"master_link_status:"的信息为"up",则说明该从节点已经完成了与新主节点间的同步,因此,将将从节点标志中的SRI_RECONF_INPROG标记清除,并增加SRI_RECONF_DONE标记。
在函数sentinelFailoverReconfNextSlave的最后,会调用函数sentinelFailoverDetectEnd,检查是否所有从节点实例都已经完成了同步。该函数的代码如下:
void sentinelFailoverDetectEnd(sentinelRedisInstance *master) {
int not_reconfigured = 0, timeout = 0;
dictIterator *di;
dictEntry *de;
mstime_t elapsed = mstime() - master->failover_state_change_time; /* We can't consider failover finished if the promoted slave is
* not reachable. */
if (master->promoted_slave == NULL ||
master->promoted_slave->flags & SRI_S_DOWN) return; /* The failover terminates once all the reachable slaves are properly
* configured. */
di = dictGetIterator(master->slaves);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *slave = dictGetVal(de); if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE)) continue;
if (slave->flags & SRI_S_DOWN) continue;
not_reconfigured++;
}
dictReleaseIterator(di); /* Force end of failover on timeout. */
if (elapsed > master->failover_timeout) {
not_reconfigured = 0;
timeout = 1;
sentinelEvent(REDIS_WARNING,"+failover-end-for-timeout",master,"%@");
} if (not_reconfigured == 0) {
sentinelEvent(REDIS_WARNING,"+failover-end",master,"%@");
master->failover_state = SENTINEL_FAILOVER_STATE_UPDATE_CONFIG;
master->failover_state_change_time = mstime();
} /* If I'm the leader it is a good idea to send a best effort SLAVEOF
* command to all the slaves still not reconfigured to replicate with
* the new master. */
if (timeout) {
dictIterator *di;
dictEntry *de; di = dictGetIterator(master->slaves);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *slave = dictGetVal(de);
int retval; if (slave->flags &
(SRI_RECONF_DONE|SRI_RECONF_SENT|SRI_DISCONNECTED)) continue; retval = sentinelSendSlaveOf(slave,
master->promoted_slave->addr->ip,
master->promoted_slave->addr->port);
if (retval == REDIS_OK) {
sentinelEvent(REDIS_NOTICE,"+slave-reconf-sent-be",slave,"%@");
slave->flags |= SRI_RECONF_SENT;
}
}
dictReleaseIterator(di);
}
}
首先,如果"我"选中的新主节点目前处于主观下线的状态,则直接返回;
接下来,轮训字典master->slaves,查看当前尚未完成同步的从节点的个数not_reconfigured:如果该从节点的标志位中还没有设置SRI_RECONF_DONE标记,则表示它还没有完成同步操作;
如果故障转移流程处于当前状态的时间,已经超过了master->failover_timeout的时间,则将not_reconfigured置为0,表示接下来会强制进入下一状态;并且置timeout为1,表示接下来会重新发送一次"SLAVEOF"命令;
接下来,如果not_reconfigured为0,要么表示所有从节点已经完成了与新主节点间的同步,要么表示超时了。不管哪种情况,都将故障转移状态置为SENTINEL_FAILOVER_STATE_UPDATE_CONFIG,表示进入故障转移流程的最后状态;
接下来,如果timeout为1,表示发生了超时。向所有未完成同步的从节点发送一次"SLAVEOF"命令:轮训字典master->slaves,只要从节点标志位中没有设置SRI_RECONF_DONE,SRI_RECONF_SENT或SRI_DISCONNECTED标记,就调用sentinelSendSlaveOf函数重新向从节点发送一次"SLAVEOF"命令;
6:SENTINEL_FAILOVER_STATE_UPDATE_CONFIG
故障转移流程的最后一个状态,就是要更新当前哨兵节点中的主节点实例,及其下属从节点实例的信息。
需要注意的是,该状态的处理并非在sentinelFailoverStateMachine函数中完成的。而是在sentinelHandleDictOfRedisInstances函数中,轮训完所有实例之后,一旦发现某个主节点的故障转移状态为SENTINEL_FAILOVER_STATE_UPDATE_CONFIG,则调用函数sentinelFailoverSwitchToPromotedSlave进行处理。
sentinelFailoverSwitchToPromotedSlave函数的代码很简单,就是调用函数sentinelResetMasterAndChangeAddress,将主节点的信息更新为选中的从节点的信息。sentinelResetMasterAndChangeAddress函数的代码如下:
int sentinelResetMasterAndChangeAddress(sentinelRedisInstance *master, char *ip, int port) {
sentinelAddr *oldaddr, *newaddr;
sentinelAddr **slaves = NULL;
int numslaves = 0, j;
dictIterator *di;
dictEntry *de; newaddr = createSentinelAddr(ip,port);
if (newaddr == NULL) return REDIS_ERR; /* Make a list of slaves to add back after the reset.
* Don't include the one having the address we are switching to. */
di = dictGetIterator(master->slaves);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *slave = dictGetVal(de); if (sentinelAddrIsEqual(slave->addr,newaddr)) continue;
slaves = zrealloc(slaves,sizeof(sentinelAddr*)*(numslaves+1));
slaves[numslaves++] = createSentinelAddr(slave->addr->ip,
slave->addr->port);
}
dictReleaseIterator(di); /* If we are switching to a different address, include the old address
* as a slave as well, so that we'll be able to sense / reconfigure
* the old master. */
if (!sentinelAddrIsEqual(newaddr,master->addr)) {
slaves = zrealloc(slaves,sizeof(sentinelAddr*)*(numslaves+1));
slaves[numslaves++] = createSentinelAddr(master->addr->ip,
master->addr->port);
} /* Reset and switch address. */
sentinelResetMaster(master,SENTINEL_RESET_NO_SENTINELS);
oldaddr = master->addr;
master->addr = newaddr;
master->o_down_since_time = 0;
master->s_down_since_time = 0; /* Add slaves back. */
for (j = 0; j < numslaves; j++) {
sentinelRedisInstance *slave; slave = createSentinelRedisInstance(NULL,SRI_SLAVE,slaves[j]->ip,
slaves[j]->port, master->quorum, master);
releaseSentinelAddr(slaves[j]);
if (slave) sentinelEvent(REDIS_NOTICE,"+slave",slave,"%@");
}
zfree(slaves); /* Release the old address at the end so we are safe even if the function
* gets the master->addr->ip and master->addr->port as arguments. */
releaseSentinelAddr(oldaddr);
sentinelFlushConfig();
return REDIS_OK;
}
因为某个从节点实例升级为主节点了。因此首先遍历字典master->slaves,根据其中的每一个从节点实例,只要它的ip或port与新主节点的ip或port不同,就将其ip和port记录到数组slaves中;
并且,当前主节点的ip和port与新的主节点的ip和port不同的情况下,也把当前主节点的地址记录到数组slaves中(因为该主节点后续上线后,会转换成从节点);
然后,调用sentinelResetMaster函数,重置主节点实例的信息,比如释放并重建从节点字典ri->slaves;断开异步上下文cc和pc上的连接;重置实例结构中的各个属性等;
最后,轮训数组slaves,根据其中记录的每一个ip和port信息,创建从节点实例,增加到字典master->slaves中;
另外,如果哨兵A收到其他哨兵发布的HELLO消息后,发现HELLO消息中的主节点信息,与本地的不一致。说明其他哨兵刚刚完成了一次故障转移流程,并升级了某个从节点使其成为了新的主节点。因此,哨兵A也会调用sentinelResetMasterAndChangeAddress函数,重置主节点信息。
最后,当前处于下线状态的旧的主节点B,已经被放到新的主节点的master->slaves字典中了。因此哨兵会不断尝试向其建链。一旦B恢复上线后,哨兵与其的命令连接和订阅连接就会建立。在向其发送”INFO”命令,并得到其回复后,就会发现它的角色还是主节点,因此需要向其发送”SLAVEOF”命令,使其成为从节点。
这是在收到”INFO”命令回复的回调函数sentinelRefreshInstanceInfo中进行处理的。这部分的代码如下:
void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
...
if ((ri->flags & SRI_SLAVE) && role == SRI_MASTER) {
/* If this is a promoted slave we can change state to the
* failover state machine. */
if ((ri->flags & SRI_PROMOTED) &&
(ri->master->flags & SRI_FAILOVER_IN_PROGRESS) &&
(ri->master->failover_state ==
SENTINEL_FAILOVER_STATE_WAIT_PROMOTION))
{
...
} else {
/* A slave turned into a master. We want to force our view and
* reconfigure as slave. Wait some time after the change before
* going forward, to receive new configs if any. */
mstime_t wait_time = SENTINEL_PUBLISH_PERIOD*4; if (!(ri->flags & SRI_PROMOTED) &&
sentinelMasterLooksSane(ri->master) &&
sentinelRedisInstanceNoDownFor(ri,wait_time) &&
mstime() - ri->role_reported_time > wait_time)
{
redisLog(REDIS_WARNING, "[%s]%s report it is master, send SLAVEOF %s %d",
__func__, getinstanceinfo(ri), ri->master->addr->ip, ri->master->addr->port); int retval = sentinelSendSlaveOf(ri,
ri->master->addr->ip,
ri->master->addr->port);
if (retval == REDIS_OK)
sentinelEvent(REDIS_NOTICE,"+convert-to-slave",ri,"%@");
}
}
}
...
}
如果ri->flags中为从节点,但是role为主节点,但是该实例不是在在进行故障转移流程中选中的新主节点。这种情况一般是,之前下线的老的主节点又重新上线了。
因此,在调用sentinelMasterLooksSane函数判断当前主节点状态正常,并且该实例在近期并未主观下线或客观下线,并且该实例上报自己是主节点已经有一段时间了,则调用函数sentinelSendSlaveOf,向该实例发送"SLAVE OF"命令,使其成为从节点。
至此,故障转移流程就介绍完了。但是,因为sentinel是分布式系统,涉及到多个主机,以及网络环境的不稳定等因素,现实中肯定会有很多边界情况的发生,sentinel的代码也肯定是踩过很多坑之后才更新到现在的样子。所以,这里只是介绍了一些主体流程,剩下的,只能在实际的场景中去感受代码的巧妙。
Redis源码解析:23sentinel(四)故障转移流程的更多相关文章
- Redis源码阅读(四)集群-请求分配
Redis源码阅读(四)集群-请求分配 集群搭建好之后,用户发送的命令请求可以被分配到不同的节点去处理.那Redis对命令请求分配的依据是什么?如果节点数量有变动,命令又是如何重新分配的,重分配的过程 ...
- Cwinux源码解析(四)
我在我的 薛途的博客 上发表了新的文章,欢迎各位批评指正. Cwinux源码解析(四)
- VueJs 源码解析 (四) initRender.Js
vueJs 源码解析 (四) initRender.Js 在之前的文章中提到了 vuejs 源码中的 架构部分,以及 谈论到了 vue 源码三要素 vm.compiler.watcher 这三要素,那 ...
- Netty 源码解析(四): Netty 的 ChannelPipeline
今天是猿灯塔“365篇原创计划”第四篇. 接下来的时间灯塔君持续更新Netty系列一共九篇 Netty 源码解析(一): 开始 Netty 源码解析(二): Netty 的 Channel Netty ...
- Flink 源码解析 —— Standalone Session Cluster 启动流程深度分析之 Job Manager 启动
Job Manager 启动 https://t.zsxq.com/AurR3rN 博客 1.Flink 从0到1学习 -- Apache Flink 介绍 2.Flink 从0到1学习 -- Mac ...
- Flink 源码解析 —— Standalone session 模式启动流程
Standalone session 模式启动流程 https://t.zsxq.com/EemAEIi 博客 1.Flink 从0到1学习 -- Apache Flink 介绍 2.Flink 从0 ...
- Flink 源码解析 —— Standalone Session Cluster 启动流程深度分析之 Task Manager 启动
Task Manager 启动 https://t.zsxq.com/qjEUFau 博客 1.Flink 从0到1学习 -- Apache Flink 介绍 2.Flink 从0到1学习 -- Ma ...
- Redis源码解析:28集群(四)手动故障转移、从节点迁移
一:手动故障转移 Redis集群支持手动故障转移.也就是向从节点发送"CLUSTER FAILOVER"命令,使其在主节点未下线的情况下,发起故障转移流程,升级为新的主节点,而原 ...
- Redis源码解析:27集群(三)主从复制、故障转移
一:主从复制 在集群中,为了保证集群的健壮性,通常设置一部分集群节点为主节点,另一部分集群节点为这些主节点的从节点.一般情况下,需要保证每个主节点至少有一个从节点. 集群初始化时,每个集群节点都是以独 ...
随机推荐
- 面试系列13 redis都有哪些数据类型
(1)string 这是最基本的类型了,没啥可说的,就是普通的set和get,做简单的kv缓存 (2)hash 这个是类似map的一种结构,这个一般就是可以将结构化的数据,比如一个对象(前提是这个对象 ...
- amazeUI表单提交验证--input框required
效果: html: <!DOCTYPE html> <html> <head> <meta charset="utf-8"> < ...
- vue+layui制作列表页
优点: 1.选用layui国产. 2.layui有一套完整的前端框架,基本哪来就可以用. 3.选用vue去掉了很多页面元素js拼接的繁琐,及不易修改. 4.vue里面还有一些过滤器等,用起来很方便. ...
- Vue项目在Docker的自动化部署
操作系统:CentOS 部署环境:Docker CI/CD工具:Jenkins 1.环境配置 安装Jenkins:Centos安装Jenkins 安装Docker:Centos安装Git.DotNet ...
- 利用HttpWebRequest模拟提交图片
利用HttpWebRequest模拟提交图片 最近在做排量post工具, 以前做的都是提交文字 这次需要post图片过去,弄了半天终于弄好了: /// <summary> /// Post ...
- 【珍惜时间】vue-websocket
这个项目可能是个有始无终的项目?跟我一起分析吧,比较简单的一个项目 另外,我也想跟自己说,我好像失去了那个努力的自己了.要珍惜时间,好好加油啊~ 项目地址为:https://github.com/xi ...
- 0906NOIP模拟测试赛后总结
我进前十辣.然而有10个大佬去学LCT了于是没有考试. Dybala神和exzkt神分-rank1,奶风神和林哥分-rank3,wc.miemeng和DuanYue神140分-rank5. 我.ooo ...
- Django项目:CMDB(服务器硬件资产自动采集系统)--11--07CMDB文件模式测试采集硬件数据
#settings.py # ————————01CMDB获取服务器基本信息———————— import os BASEDIR = os.path.dirname(os.path.dirname(o ...
- natapp出现Invalid Host header
前端是vue 3.x 项目,需要更改vue的配置文件 vue.config.js,在module.exports中添加devServer:{disableHostCheck:true}
- 深入浅出 Java Concurrency (37): 并发总结 part 1 死锁与活跃度[转]
死锁与活跃度 前面谈了很多并发的特性和工具,但是大部分都是和锁有关的.我们使用锁来保证线程安全,但是这也会引起一些问题. 锁顺序死锁(lock-ordering deadlock):多个线程试图通 ...