MySQL高可用之PXC安装部署（续）

Preface

Yesterday I implemented a three-nodes PXC,but there were some errors when proceeding it.So I decide to research what's wrong with it.The operations and evidences shows below.

Procedure

Fisrt of all,I find out that yesterday's error is due to lack of "socat" package on first node zlm2.

 //Here's the error log of it.

 --08T08::18.198487+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')

 --08T08::18.200272+:  [Note] WSREP: DONOR thread signaled with

 which: no socat in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)

     --08T06::.782344Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************

     --08T06::.784043Z WSREP_SST: [ERROR] socat not found in path: /usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin

     --08T06::.785926Z WSREP_SST: [ERROR] ******************************************************

 --08T08::18.786496+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21':  (No such file or directory)

 --08T08::18.786587+:  [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'

 --08T08::18.787610+:  [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (No such file or directory)

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #rpm -qa|grep socat

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #yum install socat

 ... //Omitted.

 Installed:

   socat.x86_64 :1.7.3.2-.el7                                                                                                                       

 Complete!

Restart the PXC mysqld on zlm2.

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #mysqladmin -P3308 shutdown

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 mysql       0.1 22.4   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

 [] 

 [root@zlm2 :: /data/mysql/mysql3308/data]

 #ps aux|grep mysqld

 mysql       0.4 22.1   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

It still show the ERROR output when node zlm3 startup.

 --08T09::14.789691+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') connection established to 2bcb8dba tcp://192.168.1.102:4567

 --08T09::14.790669+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --08T09::15.289238+:  [Note] WSREP: declaring 2bcb8dba at tcp://192.168.1.102:4567 stable

 --08T09::15.290161+:  [Note] WSREP: Node e783b5a9 state primary

 --08T09::15.291214+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,2bcb8dba,)

 memb {

     2bcb8dba,

     e783b5a9,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --08T09::15.291254+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T09::15.291707+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T09::15.291726+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --08T09::15.791844+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b

 --08T09::15.793169+:  [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from  (zlm3)

 --08T09::15.793198+:  [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from  (zlm2)

 --08T09::15.793205+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T09::15.793211+:  [Note] WSREP: Flow-control interval: [, ]

 --08T09::15.793213+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T09::15.793314+:  [Note] WSREP: REPL Protocols:  (, )

 --08T09::15.793324+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T09::15.793327+:  [Note] WSREP: Setting wsrep_ready to true

 --08T09::15.793330+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T09::15.793332+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T09::15.793337+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T09::15.793351+:  [Note] WSREP: Service thread queue flushed.

 --08T09::16.392574+:  [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.

 --08T09::16.392599+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --08T09::16.392659+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T09::16.392879+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21')

 --08T09::16.396838+:  [Note] WSREP: DONOR thread signaled with

 --08T09::17.917352+:  [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --08T07::.003790Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --08T09::27.124206+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --08T09::27.138281+:  [Note] Aborted connection  to db: 'unconnected' user: 'sst' host: 'localhost' (Got an error reading communication packets)

     --08T07::.139336Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************

     --08T07::.140923Z WSREP_SST: [ERROR] xtrabackup finished with error: .  Check /data/mysql/mysql3308/data//innobackup.backup.log

 --------------- innobackup.backup.log (START) --------------------

 xtrabackup: recognized server arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel=

 xtrabackup: recognized client arguments: --datadir=/data/mysql/mysql3308/data --server-id= --open_files_limit= --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group= --innodb_log_file_size=100M --innodb_file_per_table= --innodb_open_files= --innodb_page_size=16k --innodb_read_io_threads= --innodb_write_io_threads= --innodb_io_capacity= --innodb_doublewrite= --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit= --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel= --port= --user=sst --password=* --socket=/tmp/mysql3308.sock --lock-ddl= --backup= --galera-info= --binlog-info=ON --stream=xbstream --target-dir=/tmp/pxc_sst_sn2p/donor_xb_qh5x

  ::  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock' as 'sst'  (using password: YES).

 Failed to connect to MySQL server: DBI connect(';mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock','sst',...) failed: Access denied for user 'sst'@'192.168.1.101' (using password: YES) at - line .

  :: Connecting to MySQL server host: localhost, user: sst, password: set, port: , socket: /tmp/mysql3308.sock

 Using server version 5.7.--29.26-log

  :: Executing LOCK TABLES FOR BACKUP...

 Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation

 --------------- innobackup.backup.log (END) ----------------------

     --08T07::.143691Z WSREP_SST: [ERROR] ******************************************************

     --08T07::.145779Z WSREP_SST: [ERROR] Cleanup after exit with status:

 --08T09::27.158932+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21':  (Invalid argument)

 --08T09::27.159041+:  [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21'

 --08T09::27.160430+:  [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: - (Invalid argument)

 //Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation.

 //It seams the user sst@localhost is lack of privilege of "PROCESS".

Modify the privilege of user sst@localhost.

 (zlm@192.168.1.101 )[(none)]>select user,host from mysql.user;

 +---------------+-------------+

 | user          | host        |

 +---------------+-------------+

 | zlm           | 192.168..% |

 | mysql.session | localhost   |

 | mysql.sys     | localhost   |

 | root          | localhost   |

 | sst           | localhost   |

 +---------------+-------------+

  rows in set (0.02 sec)

 (zlm@192.168.1.101 )[(none)]>show grants for sst@localhost;

 +---------------------------------------------------------------------------+

 | Grants for sst@localhost                                                  |

 +---------------------------------------------------------------------------+

 | GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |

 +---------------------------------------------------------------------------+

  row in set (0.00 sec)

 (zlm@192.168.1.101 )[(none)]>grant process on *.* to 'sst'@'localhost';

 ERROR  (): Access denied for user 'zlm'@'192.168.1.%' (using password: YES)

 (zlm@192.168.1.101 )[(none)]>exit

 Bye

 [root@zlm2 :: ~]

 #mysql -uroot -pPassw0rd -hlocalhost -S /tmp/mysql3308.sock

 mysql: [Warning] Using a password on the command line interface can be insecure.

 Welcome to the MySQL monitor.  Commands end with ; or \g.

 Your MySQL connection id is

 Server version: 5.7.--29.26-log Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 Copyright (c) - Percona LLC and/or its affiliates

 Copyright (c) , , Oracle and/or its affiliates. All rights reserved.

 Oracle is a registered trademark of Oracle Corporation and/or its

 affiliates. Other names may be trademarks of their respective

 owners.

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 (root@localhost mysql3308.sock)[(none)]>grant process on *.* to sst@localhost;

 Query OK,  rows affected,  warning (0.01 sec)

 (root@localhost mysql3308.sock)[(none)]>show grants for sst@localhost;

 +------------------------------------------------------------------------------------+

 | Grants for sst@localhost                                                           |

 +------------------------------------------------------------------------------------+

 | GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' |

 +------------------------------------------------------------------------------------+

  row in set (0.00 sec)

Restart PXC again on zlm2.

[root@zlm2 :: ~]

#ps aux|grep mysqld

mysql       0.1 22.2   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

[root@zlm2 :: ~]

#mysqladmin -uzlm -pzlmzlm -h192.168.1. -P3308 shutdown

mysqladmin: [Warning] Using a password on the command line interface can be insecure.

[root@zlm2 :: ~]

#ps aux|grep mysqld

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

[root@zlm2 :: /data/mysql/mysql3308/data]

#mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

[]

[]   Done                    mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

[root@zlm2 :: /data/mysql/mysql3308/data]

#ps aux|grep mysqld

mysql       5.8 20.6   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster

root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

//No ERROR messages were found in error log.

Restart PXC again on zlm3 and check the output of error log.

 [root@zlm3 :: ~]

 #ps aux|grep mysqld

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 []+  Exit                   mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf

 [root@zlm3 :: /data/mysql/mysql3308/data]

 #cat error.log |grep ERROR

     --08T08::.819918Z WSREP_SST: [ERROR] Cleanup after exit with status:

 --08T10::19.826527+:  [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' :  (Operation not permitted)

 --08T10::19.826557+:  [ERROR] WSREP: Failed to read uuid:seqno from joiner script.

 --08T10::19.826565+:  [ERROR] WSREP: SST script aborted with error  (Operation not permitted)

 --08T10::19.826592+:  [ERROR] WSREP: SST failed:  (Operation not permitted)

 --08T10::19.826595+:  [ERROR] Aborting

Check the error log on donor node zlm2 begin with the time of starting mysqld on joiner zlm3.

 --08T10::47.539174+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567

 --08T10::47.539620+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --08T10::48.038167+:  [Note] WSREP: declaring b3de093b at tcp://192.168.1.102:4567 stable

 --08T10::48.038656+:  [Note] WSREP: Node d4006db4 state primary

 --08T10::48.039842+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,b3de093b,)

 memb {

     b3de093b,

     d4006db4,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --08T10::48.039890+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T10::48.040138+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T10::48.040154+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --08T10::48.540351+:  [Note] WSREP: STATE EXCHANGE: sent state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067

 --08T10::48.540644+:  [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from  (zlm3)

 --08T10::48.541101+:  [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from  (zlm2)

 --08T10::48.541140+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T10::48.541150+:  [Note] WSREP: Flow-control interval: [, ]

 --08T10::48.541155+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T10::48.541224+:  [Note] WSREP: REPL Protocols:  (, )

 --08T10::48.541234+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T10::48.541238+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::48.541241+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T10::48.541245+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::48.541252+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T10::48.541264+:  [Note] WSREP: Service thread queue flushed.

 --08T10::49.007421+:  [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor.

 --08T10::49.007441+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --08T10::49.007480+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::49.007626+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')

 --08T10::49.012384+:  [Note] WSREP: DONOR thread signaled with

 --08T10::50.911071+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --08T08::.535108Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --08T10::59.626864+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --08T10::05.537365+:  [Note] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) complete.

 --08T10::05.537399+:  [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )

 --08T10::05.542864+:  [Note] WSREP: Member 1.0 (zlm2) synced with group.

 --08T10::05.542949+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --08T10::05.543015+:  [Note] WSREP: Synchronized with group, ready for connections

 --08T10::05.543022+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::05.543027+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::22.477724+:  [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)

 --08T10::22.479542+:  [Note] WSREP: Node d4006db4 state primary

 --08T10::22.479612+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,d4006db4,)

 memb {

     d4006db4,

     }

 joined {

     }

 left {

     }

 partitioned {

     b3de093b,

     }

 )

 --08T10::22.479621+:  [Note] WSREP: Save the discovered primary-component to disk

 --08T10::22.479763+:  [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567)

 --08T10::22.479875+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --08T10::22.479976+:  [Note] WSREP: STATE_EXCHANGE: sent state UUID: c8aef280-9ae8-11e8-80c9-d74e514aa266

 --08T10::22.479976+:  [Note] WSREP: STATE EXCHANGE: sent state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266

 --08T10::22.479976+:  [Note] WSREP: STATE EXCHANGE: got state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266 from  (zlm2)

 --08T10::22.479978+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --08T10::22.479983+:  [Note] WSREP: Flow-control interval: [, ]

 --08T10::22.480000+:  [Note] WSREP: Trying to continue unpaused monitor

 --08T10::22.480032+:  [Note] WSREP: REPL Protocols:  (, )

 --08T10::22.480037+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --08T10::22.480040+:  [Note] WSREP: Setting wsrep_ready to true

 --08T10::22.480043+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --08T10::22.480046+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --08T10::22.480051+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --08T10::22.480060+:  [Note] WSREP: Service thread queue flushed.

 --08T10::26.085271+:  [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567

 --08T10::26.085306+:  [Warning] WSREP: discarding established (time wait) b3de093b (tcp://192.168.1.102:4567)

 --08T10::27.948973+:  [Note] WSREP:  cleaning up b3de093b (tcp://192.168.1.102:4567)

 //There's no ERROR messages at all this time.

Suplemented on August 9.

The error above was due to the parameter of "innodb_log_group_home_dir",after I omitted it,there was no error message any more.

Evidence on node zlm2.

 //Start the PCX on node zlm2.

 [root@zlm2 :: ~]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster &

 [] 

 //The error log of node zlm2 when node zlm3 is joining the cluster.

 --09T04::43.185049+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T04::43.185703+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T04::43.683941+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T04::43.684321+:  [Note] WSREP: Node 13eae368 state primary

 --09T04::43.684954+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T04::43.684992+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T04::43.685286+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T04::43.685750+:  [Note] WSREP: STATE_EXCHANGE: sent state UUID: -9b79-11e8-b078-6a52474ad103

 --09T04::43.688438+:  [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103

 --09T04::43.688976+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm2)

 --09T04::44.194166+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm3)

 --09T04::44.194203+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = ,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.194210+:  [Note] WSREP: Flow-control interval: [, ]

 --09T04::44.194213+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T04::44.194278+:  [Note] WSREP: REPL Protocols:  (, )

 --09T04::44.194285+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T04::44.194288+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::44.194315+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T04::44.194320+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.194327+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T04::44.194337+:  [Note] WSREP: Service thread queue flushed.

 --09T04::44.870138+:  [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T04::44.870160+:  [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: )

 --09T04::44.870209+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.870406+:  [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix ''  --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22')

 --09T04::44.872063+:  [Note] WSREP: DONOR thread signaled with

 --09T04::46.270023+:  [Note] WSREP: (13eae368, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --09T02::.527214Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102

 --09T04::55.751008+:  [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES)

 --09T04::03.636264+:  [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.

 --09T04::03.636295+:  [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: )

 --09T04::03.647735+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

 --09T04::03.647805+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T04::03.647828+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T04::03.647833+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::03.647837+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::23.739120+:  [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.

 --09T04::23.739901+:  [Note] WSREP: Member 1.0 (zlm3) synced with group.

Evidence on node zlm3(using xtrabackup-v2 mothed of sst).

 //Start the PCX on node zlm3.

 [root@zlm3 :: /data/mysql/mysql3308]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 [] 

 [root@zlm3 :: /data/mysql/mysql3308]

 #ps aux|grep mysqld

 mysql       3.1 21.1   pts/  Sl   :   : mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf

 root        0.0  0.0     pts/    R+   :   : grep --color=auto mysqld

 //The intact error log of node zlm3 when joining the cluster.

 [root@zlm3 :: /data/mysql/mysql3308/data]

 #tail -f error.log 

 --09T04::43.084230+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T04::43.089848+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T04::43.089906+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T04::43.109751+:  [Note] WSREP: Setting wsrep_ready to false

 --09T04::43.109768+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T04::43.109772+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T04::43.129981+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T04::43.130124+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T04::43.130414+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T04::43.141875+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.102; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T04::43.179573+:  [Note] WSREP: GCache history reset: bd5525ab-9a15-11e8-aa0f-4b830c783fc7: -> ----:-

 --09T04::43.183306+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T04::43.183338+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T04::43.183342+:  [Note] WSREP: Starting replication

 --09T04::43.183355+:  [Note] WSREP: Setting initial position to ----:-

 --09T04::43.183507+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T04::43.183568+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T04::43.183674+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T04::43.183679+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T04::43.184132+:  [Note] WSREP: GMCast version

 --09T04::43.184337+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T04::43.184344+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T04::43.184483+:  [Note] WSREP: EVS version

 --09T04::43.184957+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.103:'

 --09T04::43.186863+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T04::43.186895+:  [Warning] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.102:4567' points to own listening address, blacklisting

 --09T04::43.188042+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T04::43.188190+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T04::43.686789+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T04::43.687321+:  [Note] WSREP: Node 13eae368 state primary

 --09T04::43.690123+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T04::43.690159+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T04::43.690256+:  [Note] WSREP: discarding pending addr without UUID: tcp://192.168.0.103:4567

 --09T04::43.690263+:  [Note] WSREP: discarding pending addr proto entry 0x28b0b70

 --09T04::44.185955+:  [Note] WSREP: gcomm: connected

 --09T04::44.186036+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T04::44.186098+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T04::44.186339+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T04::44.186355+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T04::44.186383+:  [Note] WSREP: STATE EXCHANGE: sent state msg: -9b79-11e8-b078-6a52474ad103

 --09T04::44.186388+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm2)

 --09T04::44.189234+:  [Note] WSREP: STATE EXCHANGE: got state msg: -9b79-11e8-b078-6a52474ad103 from  (zlm3)

 --09T04::44.189263+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.189270+:  [Note] WSREP: Flow-control interval: [, ]

 --09T04::44.189273+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T04::44.189277+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T04::44.189468+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T04::44.189478+:  [Note] WSREP: REPL Protocols:  (, )

 --09T04::44.189485+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T04::44.189489+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::44.189491+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T04::44.189493+:  [Note] WSREP: Setting wsrep_ready to false

 --09T04::44.192795+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

     --09T02::.755132Z WSREP_SST: [WARNING] Found a stale sst_in_progress file: /data/mysql/mysql3308/data//sst_in_progress

 --09T04::44.870937+:  [Note] WSREP: Prepared SST/IST request: xtrabackup-v2|192.168.1.102:/xtrabackup_sst//

 --09T04::44.871047+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T04::44.871090+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T04::44.871140+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T04::44.871282+:  [Note] WSREP: Service thread queue flushed.

 --09T04::44.871758+:  [Note] WSREP: Check if state gap can be serviced using IST

 --09T04::44.871782+:  [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T04::44.871834+:  [Note] WSREP: State gap can't be serviced using IST. Switching to SST

 --09T04::44.871842+:  [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7):  (Operation not permitted)

      at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.

 --09T04::44.872523+:  [Note] WSREP: Member 1.0 (zlm3) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T04::44.872545+:  [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )

 --09T04::44.872679+:  [Note] WSREP: Requesting state transfer: success, donor:

 --09T04::44.872720+:  [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     --09T02::.589512Z WSREP_SST: [INFO] WARNING: Stale temporary SST directory: /data/mysql/mysql3308/data//.sst from previous state transfer. Removing

     --09T02::.608370Z WSREP_SST: [INFO] Proceeding with SST.........

 removed ‘/data/mysql/mysql3308/logs/mysql-bin.’

     --09T02::.637863Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete!

 --09T04::46.189014+:  [Note] WSREP: (23fb8f7a, 'tcp://0.0.0.0:4567') turning message relay requesting off

     --09T02::.636108Z WSREP_SST: [INFO] Preparing the backup at /data/mysql/mysql3308/data//.sst

 --09T04::03.649250+:  [Note] WSREP: 0.0 (zlm2): State transfer to 1.0 (zlm3) complete.

 --09T04::03.659780+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

     --09T02::.887135Z WSREP_SST: [INFO] Moving the backup to /data/mysql/mysql3308/data/

     --09T02::.003073Z WSREP_SST: [INFO] Galera co-ords from recovery: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T04::22.015642+:  [Note] WSREP: SST complete, seqno:

 --09T04::22.020459+:  [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html

 --09T04::22.020620+:  [Note] InnoDB: PUNCH HOLE support available

 --09T04::22.020665+:  [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

 --09T04::22.020674+:  [Note] InnoDB: Uses event mutexes

 --09T04::22.020679+:  [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

 --09T04::22.020684+:  [Note] InnoDB: Compressed tables use zlib 1.2.

 --09T04::22.020688+:  [Note] InnoDB: Using Linux native AIO

 --09T04::22.020708+:  [Note] InnoDB: Adjusting innodb_buffer_pool_instances from  to  since innodb_buffer_pool_size is less than  MiB

 --09T04::22.021332+:  [Note] InnoDB: Number of pools:

 --09T04::22.021561+:  [Note] InnoDB: Using CPU crc32 instructions

 --09T04::22.024636+:  [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M

 --09T04::22.027579+:  [Note] InnoDB: Completed initialization of buffer pool

 --09T04::22.031734+:  [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

 --09T04::22.056548+:  [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite

 --09T04::22.058938+:  [Note] InnoDB: Highest supported file format is Barracuda.

 --09T04::22.176951+:  [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size  bytes

 --09T04::22.311846+:  [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

 --09T04::22.311867+:  [Note] InnoDB: Creating shared tablespace for temporary tables

 --09T04::22.311914+:  [Note] InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...

 --09T04::22.753228+:  [Note] InnoDB: File './ibtmp1' size is now  MB.

 --09T04::22.753891+:  [Note] InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.

 --09T04::22.753902+:  [Note] InnoDB:  non-redo rollback segment(s) are active.

 --09T04::22.754607+:  [Note] InnoDB: Waiting for purge to start

 --09T04::22.805815+:  [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605608

 --09T04::22.806243+:  [Note] Plugin 'FEDERATED' is disabled.

 --09T04::22.808992+:  [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool

 --09T04::22.854750+:  [Note] InnoDB: Buffer pool(s) load completed at   ::

 --09T04::22.865970+:  [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin

 --09T04::22.866002+:  [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T04::22.866021+:  [Note] Starting crash recovery...

 --09T04::22.866046+:  [Note] Crash recovery finished.

 --09T04::22.866057+:  [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T04::22.875798+:  [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,

 --09T04::22.875871+:  [Note] Generated uuid: '3ba41aa7-9b79-11e8-ad75-080027de0e0e', server_start_time: , bytes_sent:

 --09T04::22.875894+:  [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 3ba41aa7-9b79-11e8-ad75-080027de0e0e.

 --09T04::23.579815+:  [Note] Auto generated SSL certificates are placed in data directory.

 --09T04::23.580319+:  [Warning] CA certificate ca.pem is self signed.

 --09T04::23.708603+:  [Note] Auto generated RSA key files are placed in data directory.

 --09T04::23.708726+:  [Note] Server hostname (bind-address): '*'; port:

 --09T04::23.708750+:  [Note] IPv6 is available.

 --09T04::23.708759+:  [Note]   - '::' resolves to '::';

 --09T04::23.708775+:  [Note] Server socket created on IP: '::'.

 --09T04::23.715377+:  [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715477+:  [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715490+:  [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715499+:  [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715539+:  [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715548+:  [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.715559+:  [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.716731+:  [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.716749+:  [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T04::23.739950+:  [Note] Event Scheduler: Loaded  events

 --09T04::23.740507+:  [Note] WSREP: Signalling provider to continue on SST completion.

 --09T04::23.740531+:  [Note] WSREP: Initialized wsrep sidno

 --09T04::23.740553+:  [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T04::23.740631+:  [Note] mysqld: ready for connections.

 Version: '5.7.22-22-29.26-log'  socket: '/tmp/mysql3308.sock'  port:   Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 --09T04::23.741168+:  [Note] WSREP: 1.0 (zlm3): State transfer from 0.0 (zlm2) complete.

 --09T04::23.741191+:  [Note] WSREP: SST leaving flow control

 --09T04::23.741196+:  [Note] WSREP: Shifting JOINER -> JOINED (TO: )

 --09T04::23.742029+:  [Note] WSREP: Member 1.0 (zlm3) synced with group.

 --09T04::23.742059+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T04::23.742077+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T04::23.742082+:  [Note] WSREP: Setting wsrep_ready to true

 --09T04::23.742087+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Evidence on node zlm4(using rsync mothed of sst).

 //Change the parameter "wsrep_sst_method" into "rysnc" in my3308.cnf file.

 #loose-wsrep_sst_method=xtrabackup-v2

 loose-wsrep_sst_method=rsync

 [root@zlm4 :: /data/mysql/mysql3308/data]

 #tail -f error.log

 --09T07::21.038606+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T07::21.041387+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T07::21.041490+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T07::21.044740+:  [Note] WSREP: Setting wsrep_ready to false

 --09T07::21.044761+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T07::21.044767+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T07::21.050035+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T07::21.050186+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T07::21.050565+:  [Warning] WSREP: Could not open state file for reading: '/data/mysql/mysql3308/data//grastate.dat'

 --09T07::21.050585+:  [Warning] WSREP: No persistent state found. Bootstraping with default state

 --09T07::21.050637+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T07::21.055209+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T07::21.069257+:  [Note] WSREP: GCache history reset: ----: -> ----:-

 --09T07::21.071550+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T07::21.071584+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T07::21.071589+:  [Note] WSREP: Starting replication

 --09T07::21.071606+:  [Note] WSREP: Setting initial position to ----:-

 --09T07::21.071812+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T07::21.071888+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T07::21.072021+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T07::21.072030+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T07::21.072800+:  [Note] WSREP: GMCast version

 --09T07::21.072942+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T07::21.072942+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T07::21.073357+:  [Note] WSREP: EVS version

 --09T07::21.073926+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'

 --09T07::21.076189+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 61aad7a7 tcp://192.168.1.103:4567

 --09T07::21.076220+:  [Warning] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting

 --09T07::21.078146+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T07::21.078312+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T07::21.080311+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T07::21.583579+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T07::21.583614+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T07::21.584818+:  [Note] WSREP: Node 13eae368 state primary

 --09T07::21.589478+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     61aad7a7,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T07::21.589519+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T07::22.075191+:  [Note] WSREP: gcomm: connected

 --09T07::22.075272+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T07::22.075348+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T07::22.075642+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T07::22.075658+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T07::22.075689+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05

 --09T07::22.075697+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm2)

 --09T07::22.075704+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm3)

 --09T07::22.077812+:  [Note] WSREP: STATE EXCHANGE: got state msg: 61f621e1-9b98-11e8-a772-bfae300f8d05 from  (zlm4)

 --09T07::22.077845+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T07::22.077853+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::22.077858+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::22.077864+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T07::22.077926+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T07::22.077935+:  [Note] WSREP: REPL Protocols:  (, )

 --09T07::22.077943+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T07::22.077948+:  [Note] WSREP: Setting wsrep_ready to true

 --09T07::22.077951+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T07::22.077954+:  [Note] WSREP: Setting wsrep_ready to false

 --09T07::22.078033+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

 which: no lsof in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.-rel22-29.26..Linux.x86_64.ssl102/bin:/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin)

 --09T07::22.254544+:  [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin'

     Read: ''lsof' not found in PATH' //Here's the important message of error but has not "ERROR" keyword in the same line.So I've almost missed it.

 --09T07::22.254571+:  [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' :  (No such file or directory)

 --09T07::22.254631+:  [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

 --09T07::22.254639+:  [ERROR] Aborting

 --09T07::22.254643+:  [Note] Giving  client threads a chance to die gracefully

 --09T07::24.254907+:  [Note] WSREP: Closing send monitor...

 --09T07::24.254992+:  [Note] WSREP: Closed send monitor.

 --09T07::24.255028+:  [Note] WSREP: gcomm: terminating thread

 --09T07::24.255042+:  [Note] WSREP: gcomm: joining thread

 --09T07::24.255061+:  [Note] WSREP: rollbacker thread exiting

 --09T07::24.255377+:  [Note] WSREP: gcomm: closing backend

 --09T07::24.260692+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(NON_PRIM,13eae368,)

 memb {

     61aad7a7,

     }

 joined {

     }

 left {

     }

 partitioned {

     13eae368,

     23fb8f7a,

     }

 )

 --09T07::24.260770+:  [Note] WSREP: (61aad7a7, 'tcp://0.0.0.0:4567') turning message relay requesting off

 --09T07::24.260814+:  [Note] WSREP: Current view of cluster as seen by this node

 view ((empty))

 --09T07::24.261439+:  [Note] WSREP: gcomm: closed

 --09T07::24.261690+:  [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = , memb_num =

 --09T07::24.261741+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::24.261746+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::24.261753+:  [Note] WSREP: Received NON-PRIMARY.

 --09T07::24.261758+:  [Note] WSREP: Shifting PRIMARY -> OPEN (TO: )

 --09T07::24.261767+:  [Note] WSREP: Received self-leave message.

 --09T07::24.261774+:  [Note] WSREP: Flow-control interval: [, ]

 --09T07::24.261777+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T07::24.261780+:  [Note] WSREP: Received SELF-LEAVE. Closing connection.

 --09T07::24.261784+:  [Note] WSREP: Shifting OPEN -> CLOSED (TO: )

 --09T07::24.261794+:  [Note] WSREP: RECV thread exiting : Success

 --09T07::24.262101+:  [Note] WSREP: recv_thread() joined.

 --09T07::24.262113+:  [Note] WSREP: Closing replication queue.

 --09T07::24.262117+:  [Note] WSREP: Closing slave action queue.

 --09T07::24.262162+:  [Note] WSREP: Waiting for active wsrep applier to exit

 --09T07::24.262170+:  [Note] WSREP: Service disconnected.

 --09T07::24.262173+:  [Note] WSREP: Waiting to close threads......

 --09T07::29.262290+:  [Note] WSREP: Some threads may fail to exit.

 --09T07::29.262337+:  [Note] Binlog end

 --09T07::29.262596+:  [Note] mysqld: Shutdown complete

 //Install lsof tool.

 [root@zlm4 :: /data/mysql/mysql3308]

 #yum -y install lsof

 ... //Omitted.

 Installed:

   lsof.x86_64 :4.87-.el7                                                                                                                           

 Complete!

 //Start the node again.

 [root@zlm4 :: /data/mysql/mysql3308]

 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 [] 

 [root@zlm4 :: /data/mysql/mysql3308]

 #jobs

 []+  Running                 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf &

 //Check the error log again.

 [root@zlm4 :: /data/mysql/mysql3308/data]

 #tail -f error.log

 --09T08::02.899495+:  [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

 --09T08::02.900253+:  [Warning] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.

 --09T08::02.900326+:  [Note] mysqld (mysqld 5.7.--29.26-log) starting as process  ...

 --09T08::02.902758+:  [Note] WSREP: Setting wsrep_ready to false

 --09T08::02.902772+:  [Note] WSREP: No pre-stored wsrep-start position found. Skipping position initialization.

 --09T08::02.902776+:  [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/libgalera_smm.so'

 --09T08::02.906503+:  [Note] WSREP: wsrep_load(): Galera 3.26(r) by Codership Oy <info@codership.com> loaded successfully.

 --09T08::02.906567+:  [Note] WSREP: CRC-32C: using hardware acceleration.

 --09T08::02.906904+:  [Note] WSREP: Found saved state: ----:-, safe_to_bootstrap:

 --09T08::02.911237+:  [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/mysql3308/data/; base_host = 192.168.1.103; base_port = ; cert.log_conflicts = no; debug = no; evs.auto_evict = ; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = ; evs.send_window = ; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = ; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/mysql3308/data/; gcache.freeze_purge_at_seqno = -; gcache.keep_pages_count = ; gcache.keep_pages_size = ; gcache.mem_size = ; gcache.name = /data/mysql/mysql3308/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 8; socket.checksum = 2; socket.recv_buf_size = 212992;

 --09T08::02.921480+:  [Note] WSREP: GCache history reset: ----: -> ----:-

 --09T08::02.923600+:  [Note] WSREP: Assign initial position for certification: -, protocol version: -

 --09T08::02.923636+:  [Note] WSREP: Preparing to initiate SST/IST

 --09T08::02.923639+:  [Note] WSREP: Starting replication

 --09T08::02.923652+:  [Note] WSREP: Setting initial position to ----:-

 --09T08::02.923845+:  [Note] WSREP: Using CRC-32C for message checksums.

 --09T08::02.923912+:  [Note] WSREP: gcomm thread scheduling priority set to other:

 --09T08::02.923991+:  [Warning] WSREP: Fail to access the file (/data/mysql/mysql3308/data//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown

 --09T08::02.923996+:  [Note] WSREP: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown

 --09T08::02.924673+:  [Note] WSREP: GMCast version

 --09T08::02.924779+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567

 --09T08::02.924779+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') multicast: , ttl:

 --09T08::02.925300+:  [Note] WSREP: EVS version

 --09T08::02.925433+:  [Note] WSREP: gcomm: connecting to group 'pxc_wubx', peer '192.168.1.101:,192.168.1.102:,192.168.1.103:'

 --09T08::02.928124+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 74f3db69 tcp://192.168.1.103:4567

 --09T08::02.928158+:  [Warning] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') address 'tcp://192.168.1.103:4567' points to own listening address, blacklisting

 --09T08::02.928505+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 13eae368 tcp://192.168.1.101:4567

 --09T08::02.929670+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:

 --09T08::02.929755+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') connection established to 23fb8f7a tcp://192.168.1.102:4567

 --09T08::03.429892+:  [Note] WSREP: declaring 13eae368 at tcp://192.168.1.101:4567 stable

 --09T08::03.429916+:  [Note] WSREP: declaring 23fb8f7a at tcp://192.168.1.102:4567 stable

 --09T08::03.431850+:  [Note] WSREP: Node 13eae368 state primary

 --09T08::03.434752+:  [Note] WSREP: Current view of cluster as seen by this node

 view (view_id(PRIM,13eae368,)

 memb {

     13eae368,

     23fb8f7a,

     74f3db69,

     }

 joined {

     }

 left {

     }

 partitioned {

     }

 )

 --09T08::03.434779+:  [Note] WSREP: Save the discovered primary-component to disk

 --09T08::03.927902+:  [Note] WSREP: gcomm: connected

 --09T08::03.927980+:  [Note] WSREP: Shifting CLOSED -> OPEN (TO: )

 --09T08::03.928042+:  [Note] WSREP: Waiting for SST/IST to complete.

 --09T08::03.928251+:  [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = , memb_num =

 --09T08::03.928261+:  [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.

 --09T08::03.928285+:  [Note] WSREP: STATE EXCHANGE: sent state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8

 --09T08::03.928290+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm2)

 --09T08::03.928294+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm3)

 --09T08::03.929335+:  [Note] WSREP: STATE EXCHANGE: got state msg: 753e350b-9b99-11e8-be59-b7e1945f65e8 from  (zlm4)

 --09T08::03.929361+:  [Note] WSREP: Quorum results:

     version    = ,

     component  = PRIMARY,

     conf_id    = ,

     members    = / (primary/total),

     act_id     = ,

     last_appl. = -,

     protocols  = // (gcs/repl/appl),

     group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T08::03.929366+:  [Note] WSREP: Flow-control interval: [, ]

 --09T08::03.929369+:  [Note] WSREP: Trying to continue unpaused monitor

 --09T08::03.929372+:  [Note] WSREP: Shifting OPEN -> PRIMARY (TO: )

 --09T08::03.929413+:  [Note] WSREP: State transfer required:

     Group state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

     Local state: ----:-

 --09T08::03.929419+:  [Note] WSREP: REPL Protocols:  (, )

 --09T08::03.929425+:  [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:, view# : Primary, number of nodes: , my index: , protocol version

 --09T08::03.929428+:  [Note] WSREP: Setting wsrep_ready to true

 --09T08::03.929430+:  [Warning] WSREP: Gap in state sequence. Need state transfer.

 --09T08::03.929432+:  [Note] WSREP: Setting wsrep_ready to false

 --09T08::03.929496+:  [Note] WSREP: Initiating SST/IST transfer on JOINER side (wsrep_sst_rsync --role 'joiner' --address '192.168.1.103' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' )

     --09T06::.047845Z WSREP_SST: [INFO] Waiting for data-dir through rsync................ //Begin to transfer data using rsync mode.

 --09T08::04.319479+:  [Note] WSREP: Prepared SST/IST request: rsync|192.168.1.103:/rsync_sst

 --09T08::04.319504+:  [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset:  -> ) (Increment:  -> )

 --09T08::04.319510+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 --09T08::04.319527+:  [Note] WSREP: Assign initial position for certification: , protocol version:

 --09T08::04.319546+:  [Note] WSREP: Service thread queue flushed.

 --09T08::04.319558+:  [Note] WSREP: Check if state gap can be serviced using IST

 --09T08::04.319566+:  [Note] WSREP: Local UUID: ---- != Group UUID: bd5525ab-9a15-11e8-aa0f-4b830c783fc7

 --09T08::04.319598+:  [Note] WSREP: State gap can't be serviced using IST. Switching to SST

 --09T08::04.319602+:  [Note] WSREP: Failed to prepare for incremental state transfer: Local state UUID (----) does not match group state UUID (bd5525ab-9a15-11e8-aa0f-4b830c783fc7):  (Operation not permitted)

      at galera/src/replicator_str.cpp:prepare_for_IST():. IST will be unavailable.

 --09T08::04.321186+:  [Note] WSREP: Member 2.0 (zlm4) requested state transfer from '*any*'. Selected 0.0 (zlm2)(SYNCED) as donor.

 --09T08::04.321208+:  [Note] WSREP: Shifting PRIMARY -> JOINER (TO: )

 --09T08::04.321275+:  [Note] WSREP: Requesting state transfer: success, donor:

 --09T08::04.321291+:  [Note] WSREP: GCache history reset: ----: -> bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T08::05.939711+:  [Note] WSREP: (74f3db69, 'tcp://0.0.0.0:4567') turning message relay requesting off

 --09T08::26.343649+:  [Note] WSREP: 0.0 (zlm2): State transfer to 2.0 (zlm4) complete.

 --09T08::26.344408+:  [Note] WSREP: Member 0.0 (zlm2) synced with group.

     --09T06::.399740Z WSREP_SST: [INFO] ..............rsync completed //Finished rsync transfer.

 --09T08::26.929160+:  [Note] WSREP: SST complete, seqno:

 --09T08::27.003561+:  [Warning] InnoDB: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead; Please refer to http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html

 --09T08::27.003682+:  [Note] InnoDB: PUNCH HOLE support available

 --09T08::27.003696+:  [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

 --09T08::27.003701+:  [Note] InnoDB: Uses event mutexes

 --09T08::27.003705+:  [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

 --09T08::27.003710+:  [Note] InnoDB: Compressed tables use zlib 1.2.

 --09T08::27.003715+:  [Note] InnoDB: Using Linux native AIO

 --09T08::27.003734+:  [Note] InnoDB: Adjusting innodb_buffer_pool_instances from  to  since innodb_buffer_pool_size is less than  MiB

 --09T08::27.004212+:  [Note] InnoDB: Number of pools:

 --09T08::27.004325+:  [Note] InnoDB: Using CPU crc32 instructions

 --09T08::27.010477+:  [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = , chunk size = 100M

 --09T08::27.019767+:  [Note] InnoDB: Completed initialization of buffer pool

 --09T08::27.023312+:  [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

 --09T08::27.066815+:  [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite

 --09T08::27.070460+:  [Note] InnoDB: Highest supported file format is Barracuda.

 --09T08::27.071136+:  [Note] InnoDB: Log scan progressed past the checkpoint lsn

 --09T08::27.071168+:  [Note] InnoDB: Doing recovery: scanned up to log sequence number

 --09T08::27.071185+:  [Note] InnoDB: Database was not shutdown normally!

 --09T08::27.071190+:  [Note] InnoDB: Starting crash recovery.

 --09T08::27.306683+:  [Note] InnoDB: Created parallel doublewrite buffer at /data/mysql/mysql3308/data/xb_doublewrite, size  bytes

 --09T08::27.558968+:  [Note] InnoDB: Last MySQL binlog file position  , file name mysql-bin.

 --09T08::27.575777+:  [Note] InnoDB: Creating shared tablespace for temporary tables

 --09T08::27.575872+:  [Note] InnoDB: Setting file './ibtmp1' size to  MB. Physically writing the file full; Please wait ...

 --09T08::28.063310+:  [Note] InnoDB: File './ibtmp1' size is now  MB.

 --09T08::28.064352+:  [Note] InnoDB:  redo rollback segment(s) found.  redo rollback segment(s) are active.

 --09T08::28.064370+:  [Note] InnoDB:  non-redo rollback segment(s) are active.

 --09T08::28.065740+:  [Note] InnoDB: Waiting for purge to start

 --09T08::28.116905+:  [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.22-rel22 started; log sequence number 2605219

 --09T08::28.117131+:  [Note] Plugin 'FEDERATED' is disabled.

 --09T08::28.123701+:  [Note] InnoDB: Loading buffer pool(s) from /data/mysql/mysql3308/data/ib_buffer_pool

 --09T08::28.123749+:  [ERROR] InnoDB: Cannot open '/data/mysql/mysql3308/data/ib_buffer_pool' for reading: No such file or directory

 --09T08::28.126137+:  [Note] Recovering after a crash using /data/mysql/mysql3308/logs/mysql-bin

 --09T08::28.126167+:  [Note] WSREP: Before binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T08::28.128885+:  [Note] Starting crash recovery...

 --09T08::28.128936+:  [Note] Crash recovery finished.

 --09T08::28.128955+:  [Note] WSREP: After binlog recovery (wsrep position bd5525ab-9a15-11e8-aa0f-4b830c783fc7:)

 --09T08::28.135097+:  [Note] Salting uuid generator variables, current_pid: , server_start_time: , bytes_sent: ,

 --09T08::28.135163+:  [Note] Generated uuid: '83fac5b7-9b99-11e8-9f3a-080027de0e0e', server_start_time: , bytes_sent:

 --09T08::28.135180+:  [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 83fac5b7-9b99-11e8-9f3a-080027de0e0e.

 --09T08::28.649143+:  [Note] Auto generated SSL certificates are placed in data directory.

 --09T08::28.651737+:  [Warning] CA certificate ca.pem is self signed.

 --09T08::28.812659+:  [Note] Auto generated RSA key files are placed in data directory.

 --09T08::28.812836+:  [Note] Server hostname (bind-address): '*'; port:

 --09T08::28.812866+:  [Note] IPv6 is available.

 --09T08::28.816928+:  [Note]   - '::' resolves to '::';

 --09T08::28.821691+:  [Note] Server socket created on IP: '::'.

 --09T08::28.846925+:  [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.846986+:  [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847001+:  [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847028+:  [Warning] 'user' entry 'sst@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847089+:  [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847097+:  [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.847124+:  [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.858902+:  [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.858929+:  [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

 --09T08::28.873861+:  [Note] Event Scheduler: Loaded  events

 --09T08::28.874109+:  [Note] WSREP: Signalling provider to continue on SST completion.

 --09T08::28.874133+:  [Note] WSREP: Initialized wsrep sidno

 --09T08::28.874156+:  [Note] WSREP: SST received: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:

 --09T08::28.874266+:  [Note] mysqld: ready for connections.

 Version: '5.7.22-22-29.26-log'  socket: '/tmp/mysql3308.sock'  port:   Percona XtraDB Cluster binary (GPL) 5.7.-29.26, Revision da86071, wsrep_29.

 --09T08::28.875367+:  [Note] WSREP: 2.0 (zlm4): State transfer from 0.0 (zlm2) complete.

 --09T08::28.875409+:  [Note] WSREP: SST leaving flow control

 --09T08::28.875418+:  [Note] WSREP: Shifting JOINER -> JOINED (TO: )

 --09T08::28.876587+:  [Note] WSREP: Member 2.0 (zlm4) synced with group.

 --09T08::28.876618+:  [Note] WSREP: Shifting JOINED -> SYNCED (TO: )

 --09T08::28.876639+:  [Note] WSREP: Synchronized with group, ready for connections

 --09T08::28.876645+:  [Note] WSREP: Setting wsrep_ready to true

 --09T08::28.876649+:  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

 //It's okay now.No error is found this time.

MySQL高可用之PXC安装部署（续）的更多相关文章

MySQL高可用之PXC安装部署
Preface Today,I'm gonna implement a PXC,Let's see the procedure. Framework Hostname IP P ...
MySQL高可用方案MHA的部署和原理
MHA(Master High Availability)是一套相对成熟的MySQL高可用方案,能做到在0~30s内自动完成数据库的故障切换操作,在master服务器不宕机的情况下,基本能保证数据的一 ...
MySQL高可用架构-MHA环境部署记录
一.MHA介绍 MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Facebook公司) ...
mysql高可用之PXC（Percona XtraDB Cluster）
简介 Percona XtraDB Cluster是MySQL高可用性和可扩展性的解决方案,Percona XtraDB Cluster提供的特性如下: 1).同步复制,事务要么在所有节点提交或不提交 ...
MySQL高可用架构-MMM环境部署记录
MMM介绍MMM(Master-Master replication manager for MySQL)是一套支持双主故障切换和双主日常管理的脚本程序.MMM使用Perl语言开发,主要用来监控和管理 ...
MySQL高可用架构-MMM安装教程
安装指南: 一.架构以及服务器信息基本安装包含至少2个数据库服务器和1个监视服务器.本例中使用2个监视服务器和5个数据库服务器(服务器系统为CentOS 7) 用途 IP 主机名 Server-id ...
hbase学习（二）hbase单机和高可用完全分布式安装部署
hbase版本 2.0.4 与hadoop兼容表http://hbase.apache.org/book.html#hadoop 我的 hadoop版本是3.1 1.单机版hbase 1.1解 ...
MySQL高可用之MGR安装测试（续）
Preface I've implemented the Group Replication with three servers yesterday,What a shame it ...
（5.2）mysql高可用系列——测试环境部署
关键词环境部署: ############## 测试环境机器架构 #########[1]策划[1.1]linux服务器A组 8台 192.168.1.200~192.168.1.207,主机名db, ...

随机推荐

使用canvas上传图片+上传进度
实现效果: 速度过快,调式浏览器方式:F12 后台java代码 public String imageshangchuan(@RequestPart("xxx") Multipar ...
使用Dapper处理多个结果集和多重映射的教程
在本文中,我们将介绍如何使用DAPPER从单个数据库调用中读取数据库中的多个结果集.我们将看看我们可能希望这样做的场景,以及如何使用它的Query和QueryMultiple方法更简洁地实现这一点. ...
数字三角形W
题目描述 Description 数字三角形要求走到最后mod 100最大输入描述 Input Description 第1行n,表示n行第2到n+1行为每个的权值输出描述 Output De ...
C++指针数组，二级指针和函数指针的练习
1.编一程序,将字符串“Hello,C++!”赋给一个字符数组, 然后从第一个字母开始间隔地输出该串(请用指针完成). 代码如下 #include<iostream> #include&l ...
你应该要知道的JS中的this
前言 this 是 JavaScript 中不可不谈的一个知识点,它非常重要但又不容易理解.因为 JavaScript 中的 this 不同于其他语言.不同场景下的 this 指向不同(当函数被调用执 ...
用状态机表示SFC中的并行分支
过去一直认为,状态机表示SFC会不会是任务复杂化,这次简单实验了一下,感觉还可以.请看下面的控制. 在SFC中,A和B是一对并行分支,汇合后转移到C分支中,怎么了用状态机表示呢?这里我们在状态机里分别 ...
python学习——函数
一.在python的世界里什么是函数: 答:函数通常是用来实现某一个功能二被封装成的一个对象,是用来实现代码复用的常用方式现在有一个需求,假如你在不知道len()方法的情况下,要你计算字符串‘he ...
python基础之初识
一. 计算机是什么基本组成: 主板+cpu+内存 cpu: 主频, 核数(16) 内存:大小(8G, 16G, 32G) 型号: DDR3, DDR4, DDR5, 主频(海盗船,玩家国度) 显卡: ...
计蒜客-----跳跃游戏（C语言）
/********************************************************给定一个非负整数数组,假定你的初始位置为数组第一个下标.数组中的每个元素代表你在那个位 ...
ts packet解析
(1)TS流是基于Packet的位流格式,每个包是188字节或者204字节(一般是188字节,204字节的格式仅仅是在188字节的Packet后部加上16字节的CRC数据,其他格式是一样的),整个TS ...

MySQL高可用之PXC安装部署（续）

MySQL高可用之PXC安装部署（续）的更多相关文章

随机推荐

热门专题