centos7.5系统elasticsearch使用滚动和全新安装升级到最新的elasticsearch7.4.2版本

背景：
    生产环境大量使用 elasticsearch 集群，不同的业务使用不同版本的elasticsearch
    es经常曝出一些大的漏洞，需要进行版本升级，并且使用x-pack的基本验证功能，避免用户数据泄露

x-pack免费版本特征：

基本的TLS 功能，可对通信进行加密
文件和原生 Realm，可用于创建和管理用户
基于角色的访问控制，可用于控制用户对集群 API 和索引的访问权限；
通过针对 Kibana Spaces 的安全功能，还可允许在 Kibana 中实现多租户。

升级的两种策略：
.滚动性升级，即不中断业务服务，一台一台进行升级
.全新部署新版本，然后将数据迁移到新版本的es集群中

这两种方式都需要将数据恢复到新版本的es集群中，可以先进行快照备份

.升级前先备份低版本的elasticserch的数据：快照方式

原理：即将老版本的es数据打个快照备份出来写入到 /opt/esback 目录中并进行nfs挂载到某台服务器中(作为nfs服务端的机器要有足够量的磁盘空间，最好磁盘性能比较好)，新、旧两个es集群的配置文件中都引用配置 path.repo: ["/opt/esback/"]，
这样新的集群也能对这个目录进行操作了，等待新集群搭建好后，直接把 /opt/esback 目录中的文件恢复到新集群的 索引 index 中即可

使用Mount nfs进行挂载共享(所有的es集群节点都可以访问)：

目标：将本地es备份出来的数据目录/opt/esback 目录挂载到nfs的共享目录 /opt/es_snapshot，这样恢复的时候就都可以访问这个共享目录进行恢复了

// 在10.10.18.92上创建共享目录
创建共享目录，即作为nfs的共享目录
mkdir /opt/es_snapshot

创建本地备份出来的目录
/opt/esback

# 在集群所有节点中创建 /opt/esback 目录，即将es数据备份出来的目标目录

# 将其中一台es客户端作为nfs服务端
#nfs服务端的操作
# vim /etc/exports

# 注意此处的anonuid和gid要和运行es程序的用户保持一致

# 添加指定 uid 和 gid 的用户
groupadd -g  elastic
useradd -u  -g elastic elastic

# 修改 gid和 uid为500 命令示例：
usermod -u  es
groupmod -g  es

/opt/es_snapshot *(insecure,rw,no_root_squash,sync,anonuid=,anongid=)

// 查看共享文件夹
yum install -y exportfs
exportfs -rv

// nfs服务端修改nfs配置
 vim /etc/sysconfig/nfs
修改如下：
RPCNFSDARGS="-N 2 -N 3"
        ----->启用
# Turn off v4 protocol support
RPCNFSDARGS="-N 4"     ---->启用
重启生效
systemctl restart nfs

// 客户端操作
yum install -y nfs-utils

// 重启启动新集群机器的NFS服务
systemctl restart nfs

//每一台es节点服务器上进行Mount挂载
mount -t nfs 10.10.18.90:/opt/es_snapshot /opt/esback -o proto=tcp -o nolock

列出nfs服务端共享的目录:
[root@sz_kp_wanghong_dev02_18_93:/home/wanxing]# showmount -e 10.10.18.90
Export list for 10.10.18.92:
/opt/es_snapshot *

// 在旧机器上将共享目录的权限付给ES的运行用户
chown elastic:elastic -R /opt/esback

.创建ES仓库my_backup

修改配置文件：
vim elasticsearch.yml
# 添加如下配置(需要在旧集群的每个节点上添加)，重新启动集群
path.repo: ["/opt/esback"]

创建快照仓库 my_backup 命令：
curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup -d '
{
    "type": "fs",
    "settings": {
        "location": "/opt/esback",
    "compress": true
    }
}
'
# 返回值
{"acknowledged":true}

# 报错的处理
'RemoteTransportException[[ictr_node1][10.10.18.93:9300][internal:admin/repository/verify]]

# 权限不够
chown -R es.es /opt/es_snapshot/
chown -R es.es /opt/esback_20191104/

# 创建所有索引的备份

# curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107
{"accepted":true}

查看备份

[elastic@szyyelk01t slave02]$ curl -XGET http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107?pretty
{
  "snapshots" : [
    {
      "snapshot" : "snapshot20191107",
      "uuid" : "0_4SOntVS1GH-7irHjKBMQ",
      "version_id" : ,
      "version" : "6.3.2",
      "indices" : [
        "support_faq_categorys",
        "ticket_list",
        "templates_search",
        "site_page_search",
        "support",
        "templates_page_search",
        "support_new_articles",
        "article_version",
        "blocks_version",
        "search",
        "version",
        "article_search",
        "templates",
        "learn",
        "templates_version",
        "blocks_search",
        "templates_page_version"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2019-11-07T01:35:00.811Z",
      "start_time_in_millis" : ,
      "end_time" : "2019-11-07T01:35:03.702Z",
      "end_time_in_millis" : ,
      "duration_in_millis" : ,
      "failures" : [ ],
      "shards" : {
        "total" : ,
        "failed" : ,
        "successful" :
      }
    }
  ]
}

升级方式1：滚动升级 elasticsearch5.6.16 --> elasticsearch6.8.4

.备份数据，避免出现问题后回滚
.先升级到新版本，然后安装x-pack，此时再要求开发同事修改代码适配
a.先下载新版本的6.8.4

①关闭自动分片
curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
>   "persistent": {
>     "cluster.routing.allocation.enable": "none"
>   }
> }'
* Hostname was NOT found in DNS cache
*   Trying 10.10.18.92...
* Connected to 10.10.18.92 (10.10.18.92) port  (#)
> PUT /_cluster/settings HTTP/1.1
> User-Agent: curl/7.36.
> Host: 10.10.18.92:
> Accept: */*
> Content-Length: 73
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 73 out of 73 bytes
< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-5.6.15-fe7575a "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 05 Nov 2019 08:14:44 GMT"
< content-type: application/json; charset=UTF-8
< content-length: 106
<
* Connection #0 to host 10.10.18.92 left intact
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"none"}}}},"transient":{}}

②暂时禁用非必要的索引并执行同步刷新

curl -XPOST http://10.10.18.92:9200/_flush/synced

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]# curl -XPOST http://10.10.18.92:9200/_flush/synced
{"_shards":{"total":28,"successful":28,"failed":0},"channel_rel":{"total":4,"successful":4,"failed":0},".kibana":{"total":2,"successful":2,"failed":0},"channel":{"total":6,"successful":6,"failed":0},"video":{"total":4,"successful":4,"failed":0},"channel_list":{"total":6,"successful":6,"failed":0},"influecer":{"total":6,"successful":6,"failed":0}}

注意: 如果是从6.3之前的版本升级上来的，需要注意提前要移除X-Pack插件，然后再去升级版本。执行bin/elasticsearch-plugin remove x-pack

a. 备份原来的elasticsearch目录，然后解压新版的elasticsearch。
b. 如果使用外部的配置路径，配置ES_PATH_CONF环境变量到那个位置。如果没有的话，拷贝老的配置目录过来新的elasticsearch目录就可以了。
c. 检查path.data是否指向正确的数据目录
d. 检查path.log是否指向正确的日志目录

新集群的配置文件
[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ more config/elasticsearch.yml
cluster.name: kp-dev-application
node.name: ictr_node2
node.master: true
node.attr.rack: r1
node.max_local_storage_nodes: 3
network.host: 10.10.18.92
http.port: 9200
transport.tcp.port: 9300
path.repo: ["/opt/esback_20191104"]
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"
# 新集群的数据还是指向老版本 es5.6.15 的数据存储目录
path.data: /opt/es-node/elasticsearch-5.6.15/data
path.logs: /opt/es-node/elasticsearch-5.6.15/logs
# 启用安全认证
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

③关闭节点
④重新启动节点，注意要切换到es用户，不能使用root用户
chown -R es.es elasticsearch-6.8.4

[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.4]$ bin/elasticsearch -d

在其他节点重复以上过程

启动升级后的节点，并通过查看日志和使用下面命令来检查节点是否正确加入到集群
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/nodes
10.10.18.93 16 98 56 1.22 0.50 0.29 di  - ictr_node1
10.10.18.92 16 88  8 0.08 0.26 0.31 mdi * ictr_node2
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl http://10.10.18.92:9200/_cat/indices
yellow open channel      vRFQoIhmT8WmSbDCfph0ag 3 1   53374      0  44.2mb  44.2mb
yellow open channel_rel  ZeeBbkogT5KtxzziUYtu_Q 2 1  459528      0 168.8mb 168.8mb
yellow open channel_list 1dk8uH8bTeikez0lFR2mJg 3 1 5509390  78630     7gb     7gb
yellow open video        HNhyt9ioSEayAotGVXRCVg 2 1  798369 228155   1.6gb   1.6gb
yellow open .kibana      lY82G_-XSniyd_bnMOLuQg 1 1      15      1 146.3kb 146.3kb
yellow open influecer    RQtQWXKIRE2UYyZlCvv7bA 3 1  148526  48641 272.8mb 272.8mb

节点加入集群后，删除cluster.routing.allocation.enable设置以启用分片分配并开始使用节点：

curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'

重新打开分片报错：
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.15]# curl -v -XPUT http://10.10.18.92:9200/_cluster/settings -d '{
>   "persistent": {
>     "cluster.routing.allocation.enable": "true"
>   }
> }'

* Hostname was NOT found in DNS cache
*   Trying 10.10.18.92...
* Connected to 10.10.18.92 (10.10.18.92) port 9200 (#0)
> PUT /_cluster/settings HTTP/1.1
> User-Agent: curl/7.36.0
> Host: 10.10.18.92:9200
> Accept: */*
> Content-Length:
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off:  out of  bytes
< HTTP/1.1  Not Acceptable
< content-type: application/json; charset=UTF-
< content-length:
<
* Connection # to host 10.10.18.92 left intact
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":}

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-5.6.]# curl http://10.10.18.92:9200/_cluster/health?pretty
{
  "cluster_name" : "kp-dev-application",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : ,
  "number_of_data_nodes" : ,
  "active_primary_shards" : ,
  "active_shards" : ,
  "relocating_shards" : ,
  "initializing_shards" : ,
  "unassigned_shards" : ,
  "delayed_unassigned_shards" : ,
  "number_of_pending_tasks" : ,
  "number_of_in_flight_fetch" : ,
  "task_max_waiting_in_queue_millis" : ,
  "active_shards_percent_as_number" : 100.0
}

安装新版本中文分词插件

https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.8.4/elasticsearch-analysis-ik-6.8.4.zip

# 解压在plugin目录重新启动elasticsearch即可
cd /opt/es-node/elasticsearch-6.8./plugins
unzip -d elasticsearch-analysis-ik elasticsearch-analysis-ik-6.8..zip

ot@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8./plugins]# curl http://10.10.18.92:9200/_xpack?pretty
{
  "build" : {
    "hash" : "bca0c8d",
    "date" : "2019-10-16T06:19:49.319352Z"
  },
  "license" : {
    "uid" : "4de9d1c1-59f6-4dfd-8d48-baefd0a583d0",
    "type" : "basic",
    "mode" : "basic",
    "status" : "active"
  },
  "features" : {
    "ccr" : {
      "description" : "Cross Cluster Replication",
      "available" : false,
      "enabled" : true
    },
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : false,
      "enabled" : true
    },
    "ilm" : {
      "description" : "Index lifecycle management for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "logstash" : {
      "description" : "Logstash management component for X-Pack",
      "available" : false,
      "enabled" : true
    },
    "ml" : {
      "description" : "Machine Learning for the Elastic Stack",
      "available" : false,
      "enabled" : true,
      "native_code_info" : {
        "version" : "6.8.4",
        "build_hash" : "93ad89b02ff490"
      }
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "rollup" : {
      "description" : "Time series pre-aggregation and rollup",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : false
    },
    "sql" : {
      "description" : "SQL access to Elasticsearch",
      "available" : true,
      "enabled" : true
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : false,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}

.启用x-pack的密码验证

# 生成证书

[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]# bin/elasticsearch-certutil ca
This tool assists you in the generation of X. certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X. certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS# output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]:
Enter password for elastic-stack-ca.p12 :
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]# ls
bin  config  elastic-stack-ca.p12  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile
[root@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]# bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X. certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X. certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.

    * All certificates generated by this tool will be signed by a certificate authority (CA).
    * The tool can automatically generate a new CA for you, or you can provide your own with the
         -ca or -ca-cert command line options.

By default the 'cert' mode produces a single PKCS# output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -keep-ca-key (retain generated CA key)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) :
Please enter the desired output file [elastic-certificates.p12]:
Enter password for elastic-certificates.p12 : 

Certificates written to /opt/es-node/elasticsearch-6.8./elastic-certificates.p12

This file should be properly secured as it contains the private key for
your instance.

This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.

# 修改config/elasticsearch.yml配置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12

# 配置密码
[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

密码：espass

[es@sz_kp_wanghong_dev01_18_92:/opt/es-node/elasticsearch-6.8.]$ curl --user elastic:espass -XGET 'http://10.10.18.92:9200/_cat/indices'
green open channel_rel  ZeeBbkogT5KtxzziUYtu_Q           .7mb .8mb
green open .security-  iQHndFBqRe2Ss2o7KMxyFg                 .3kb  .1kb
green open .kibana      lY82G_-XSniyd_bnMOLuQg               .6kb .3kb
green open influecer    RQtQWXKIRE2UYyZlCvv7bA       .6mb .8mb
green open channel      vRFQoIhmT8WmSbDCfph0ag             .4mb  .2mb
green open channel_list 1dk8uH8bTeikez0lFR2mJg         14gb     7gb
green open video        HNhyt9ioSEayAotGVXRCVg        .3gb   .6gb

升级方式2：完全重启集群升级
即配置好全新的elasticsearch7..2集群，然后把数据恢复到新集群中

下载地址：wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-linux-x86_64.tar.gz

老版本的配置
# cms elasticsearch

[root@szyyelk01t opt]# egrep -v '^#|^$' elk-master/config/elasticsearch.yml
cluster.name: cms-uat-elastic
node.name: master
path.data: /opt/elk-master/data/data01,/opt/elk-master/data/data02
path.logs: /opt/elk-master/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port:
http.cors.enabled: true
http.cors.allow-origin: "*"

[root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave01/config/elasticsearch.yml
cluster.name: cms-uat-elastic
node.name: slave01
path.data: /opt/elk-slave/slave01/data/data01,/opt/elk-slave/slave01/data/data02
path.logs: /opt/elk-slave/slave01/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port:
discovery.zen.ping.unicast.hosts: ["10.10.18.90"]
http.cors.enabled: true
http.cors.allow-origin: "*"

[root@szyyelk01t elk-slave]# egrep -v '^$|^#' slave02/config/elasticsearch.yml
cluster.name: cms-uat-elastic
node.name: slave02
path.data: /opt/elk-slave/slave02/data/data01,/opt/elk-slave/slave02/data/data02
path.logs: /opt/elk-slave/slave02/logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.port:
discovery.zen.ping.unicast.hosts: ["10.10.18.90"]
http.cors.enabled: true
http.cors.allow-origin: "*"

# 已经升级的线上配置参考：

[root@eus_filmora_db01:/usr/local/elasticsearch-7.4.]# egrep -v '^$|^#' config/elasticsearch.yml
cluster.name: UOS_CLUSTER_ES
node.name: uos_node_1
path.data: /data/elasticsearch_data/data
path.logs: /data/elasticsearch_data/logs
bootstrap.memory_lock: true
network.host: 172.20.103.199
http.port:
transport.tcp.port:
node.master: true
node.data: true
discovery.seed_hosts: ["172.20.103.199:9300", "172.20.73.200:9300", "172.20.73.212:9300"]
cluster.initial_master_nodes: ["172.20.103.199", "172.20.73.200", "172.20.73.212"]
gateway.recover_after_nodes:
transport.tcp.compress: true
path.repo: ["/data/bak_es"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch/config/elastic-certificates.p12

############## cms 系统的 elasticsearch 6升级7.4.2
整体策略：
.升级cms测试环境的es到7.4.2，然后做适配性的开发，再次升级内网的生产环境 --> cms海外环境(找海外业务不繁忙的时候操作，先和开发协商好)

测试环境其他人依赖进行测试，所以升级需要两套并存，新版本的es7..2使用自带的openjdk13.

.配置新版本的elasticsearch使用指定的jdk环境
# vim bin/elasticsear

export JAVA_HOME=/opt/elk7_onenode/elasticsearch-7.4./jdk
export PATH=$JAVA_HOME/bin:$PATH

mkdir /opt/elk7_onenode/elasticsearch-7.4./data

# 主节点配置
[elastic@szyyelk01t elasticsearch-7.4.]$ more config/elasticsearch.yml
cluster.name: cms-uat-elastic7
node.name: cms_node01
node.master: true
node.data: true
discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"]
cluster.initial_master_nodes: ["10.10.18.90"]
path.data: /opt/cms_elk7/elasticsearch-7.4./data
path.logs: /opt/cms_elk7/elasticsearch-7.4./logs
discovery.zen.minimum_master_nodes:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.90
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.compress: true
path.repo: ["/opt/esback/"]
gateway.recover_after_nodes: 

# 增加新的参数head插件可以访问es
http.port:
transport.tcp.port: 

gateway.recover_after_time: 8m

# 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries:
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

# 启用安全认证
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

# 第二个节点配置

[elastic@cms-test:/opt/cms_elk7/elasticsearch-7.4.]$ more config/elasticsearch.yml
cluster.name: cms-uat-elastic7
node.name: cms_node02
node.master: false
node.data: true
discovery.seed_hosts: ["10.10.18.90:19300", "10.10.18.117:19300"]
cluster.initial_master_nodes: ["10.10.18.90"]
path.data: /opt/cms_elk7/elasticsearch-7.4./data
path.logs: /opt/cms_elk7/elasticsearch-7.4./logs
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
network.host: 10.10.18.117
http.cors.enabled: true
http.cors.allow-origin: "*"
transport.tcp.compress: true
path.repo: ["/opt/esback/"]
gateway.recover_after_nodes: 

# 增加新的参数head插件可以访问es
http.port:
transport.tcp.port: 

gateway.recover_after_time: 8m

# 以下配置可以减少当es节点短时间宕机或重启时shards重新分布带来的磁盘io读写浪费
discovery.zen.fd.ping_timeout: 300s
discovery.zen.fd.ping_retries:
discovery.zen.fd.ping_interval: 30s
discovery.zen.ping_timeout: 180s

# 启用安全认证
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

# 设置密码

# 启用安全认证，只添加这个选项
xpack.security.enabled: true

#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
#xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

elastic密码：
espass

在集群上配置TLS:
如果你在操作单节点ES则可以跳过本内容。

.生成CA证书 :
bin/elasticsearch-certutil ca
将产生新文件 elastic-stack-ca.p12。该 elasticsearch-certutil 命令还会提示你输入密码以保护文件和密钥，请保留该文件的副本并记住其密码，此处我们设置为空

.为集群中的每个节点生成证书和私钥
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

将产生新文件 elastic-certificates.p12。系统还会提示你输入密码，你可以输入证书和密钥的密码，也可以按Enter键将密码留空。默认情况下 elasticsearch-certutil 生成没有主机名信息的证书，这意味着你可以将证书用于集群中的每个节点，另外要关闭主机名验证。
将 elastic-certificates.p12 文件复制到每个节点上Elasticsearch配置目录中
无需将 elastic-stack-ca.p12 文件复制到此目录。
mkdir config/certs
mv elastic-certificates.p12 config/certs/

配置集群中的每个节点以使用其签名证书标识自身并在传输层上启用TLS
启用TLS并指定访问节点证书所需的信息，将以下信息添加到每个节点的 elasticsearch.yml 文件中：

xpack.security.enabled: true

.设置密码

# 报错
[elastic@szyyelk01t elasticsearch-7.4.]$ bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]y

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y               

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]: 

Unexpected response code [] from calling PUT http://10.10.18.90:19200/_security/user/apm_system/_password?pretty
Cause: Cluster state has not been recovered yet, cannot write to the [null] index

Possible next steps:
* Try running this tool again.
* Try running with the --verbose parameter for additional messages.
* Check the elasticsearch logs for additional error details.
* Use the change password API manually. 

ERROR: Failed to set password for user [apm_system].

[elastic@szyyelk01t elasticsearch-7.4.]$  bin/elasticsearch-setup-passwords interactive

Connection failure to: http://10.10.18.90:19200/_security/_authenticate?pretty failed: Connection refused

ERROR: Failed to connect to elasticsearch at http://10.10.18.90:19200/_security/_authenticate?pretty. Is the URL correct and elasticsearch running?
[elastic@szyyelk01t elasticsearch-7.4.]$  bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]^C[elastic@szyyelk01t elasticsearch-7.4.]$  bin/elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at http://10.10.18.90:19200
Unexpected response code [] from calling GET http://10.10.18.90:19200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]

解决办法：只配置一个主节点：cluster.initial_master_nodes: ["10.10.18.90"]

# 错误2处理

[--07T16::,][INFO ][o.e.c.c.JoinHelper       ] [cms_node02] failed to join {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:}{dilm}{ml.machine_memory=, ml.max_open_jobs=, xpack.installed=true} with JoinRequest{sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:}{dil}{ml.machine_memory=, xpack.installed=true, ml.max_open_jobs=}, optionalJoin=Optional[Join{term=, lastAcceptedTerm=, lastAcceptedVersion=, sourceNode={cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:}{dil}{ml.machine_memory=, xpack.installed=true, ml.max_open_jobs=}, targetNode={cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:}{dilm}{ml.machine_memory=, ml.max_open_jobs=, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [cms_node01][10.10.18.90:][internal:cluster/coordination/join]
Caused by: java.lang.IllegalArgumentException: can't add node {cms_node02}{765pAegcS8S0Y3OrE9taMA}{ki1VVW27TnakEEFagCoDlg}{10.10.18.117}{10.10.18.117:19300}{dil}{ml.machine_memory=16853446656, ml.max_open_jobs=20, xpack.installed=true}, found existing node {cms_node01}{765pAegcS8S0Y3OrE9taMA}{Up16Gw9pQlyXg3n1wCHE8g}{10.10.18.90}{10.10.18.90:19300}{dilm}{ml.machine_memory=8362151936, xpack.installed=true, ml.max_open_jobs=20} with the same id but is a different node instance
    at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.add(DiscoveryNodes.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.coordination.JoinTaskExecutor.execute(JoinTaskExecutor.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.coordination.JoinHelper$.execute(JoinHelper.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.MasterService.access$(MasterService.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:) ~[elasticsearch-7.4..jar:7.4.]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:) ~[elasticsearch-7.4..jar:7.4.]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:) ~[?:?]

# 拷贝连着目录 都拷贝，删除 data 下面的所有文件重启即可

# 最终密码配置成功

[elastic@szyyelk01t elasticsearch-7.4.]$ bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

# 查看集群状态
[elastic@szyyelk01t elasticsearch-7.4.]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cluster/health?pretty
{
  "cluster_name" : "cms-uat-elastic7",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : ,
  "number_of_data_nodes" : ,
  "active_primary_shards" : ,
  "active_shards" : ,
  "relocating_shards" : ,
  "initializing_shards" : ,
  "unassigned_shards" : ,
  "delayed_unassigned_shards" : ,
  "number_of_pending_tasks" : ,
  "number_of_in_flight_fetch" : ,
  "task_max_waiting_in_queue_millis" : ,
  "active_shards_percent_as_number" : 100.0
}

# 查看刚才创建所有索引的备份

# curl -H "Content-Type: application/json" -v -XPUT http://10.10.18.90:9200/_snapshot/my_backup/snapshot20191107
{"accepted":true}

# 恢复全索引快照

#保证elasticsearch用户拥有快照目录的权限
chown -R elastic.elastic /opt/esback

#创建仓库
curl -H "Content-Type: application/json" -XPUT -u elastic:espass http://10.10.18.90:19200/_snapshot/backup -d '
{
"type":"fs",
"settings":{"location":"/opt/esback"}
}'

#查询全索引快照备份
$ curl -XGET -u elastic:espass "http://10.10.18.90:19200/_snapshot/backup/_all" | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
                    160k       --:--:-- --:--:-- --:--:--  170k
{
    "snapshots": [
        {
            "duration_in_millis": ,
            "end_time": "2019-11-07T01:35:03.702Z",
            "end_time_in_millis": ,
            "failures": [],
            "include_global_state": true,
            "indices": [
                "support_faq_categorys",
                "ticket_list",
                "templates_search",
                "site_page_search",
                "support",
                "templates_page_search",
                "support_new_articles",
                "article_version",
                "blocks_version",
                "search",
                "version",
                "article_search",
                "templates",
                "learn",
                "templates_version",
                "blocks_search",
                "templates_page_version"
            ],
            "shards": {
                "failed": ,
                "successful": ,
                "total":
            },
            "snapshot": "snapshot20191107",
            "start_time": "2019-11-07T01:35:00.811Z",
            "start_time_in_millis": ,
            "state": "SUCCESS",
            "uuid": "0_4SOntVS1GH-7irHjKBMQ",
            "version": "6.3.2",
            "version_id":
        }
    ]
}

#恢复全索引快照
[elastic@szyyelk01t elasticsearch-7.4.]$ curl -XPOST -u elastic:espass 'http://10.10.18.90:19200/_snapshot/backup/snapshot20191107/_restore'?wait_for_completion=true

{"snapshot":{"snapshot":"snapshot20191107","indices":["templates_page_search","article_search","blocks_version","learn","templates_page_version","templates","version","site_page_search","support_new_articles","support_faq_categorys","search","templates_search","blocks_search","ticket_list","article_version","support","templates_version"],"shards":{"total":,"failed":,"successful":}}}

# 查看已经恢复成功
[elastic@szyyelk01t elasticsearch-7.4.]$ curl -H "Content-Type: application/json" -u elastic:espass http://10.10.18.90:19200/_cat/indices
green open templates_page_search  tUKh1vaHRla6QamphIByLQ       .3kb .6kb
green open article_search         _LE5n_-KRSGVH6Z3I1YLNQ           .5mb .8kb
green open blocks_version         VRmv8fyESY6iclBYkhKJ_w          .5kb  .7kb
green open learn                  W4RyJnkrStaRJwQgS4MAug         .6kb .8kb
green open templates_page_version _hHckKOfRuCPEojviySxVw          .5mb   777kb
green open templates              7iJqDoBwTbOEHcyEzPLHbA            2mb     1mb
green open version                mLbfHoA7SAu4RWHSHM3vtw           .9kb  .9kb
green open support_new_articles   HvGe-CklRU-iua-_T1pLNA        12mb     6mb
green open site_page_search       xxk8IetTSr2HF2tEe2Vc1w          .5mb .2kb
green open .security-            xdRnCeykQGGPcqM3-_WFCw           .5kb  .8kb
green open search                 fOteaZd0QfaU_2fKBaWPdA            .5kb    783b
green open support_faq_categorys  h61nZp5bSQqV1UGVyHL7WA            .5kb    783b
green open templates_search       ru8oFeQDTtKovOmkjP6A0w          .5mb .8kb
green open blocks_search          8vMOY6ebTs-4iJIwM2VG0Q            .5kb   .2kb
green open article_version        qcF3Nft6QMezKqtPHyYLlA            5mb   .5mb
green open ticket_list            xpvXuhlqRFq5Y_zugq0qKw          .1mb     1mb
green open support                LypmJq0pRDy428-TKOy6Yg            .5kb    783b
green open templates_version      gI28sYWJT3GVgfBeyJhSLg          .2mb   .1mb