ELK之elasticsearch6.5集群

前面介绍并初试了es6.5系列的单节点的操作,现在搭建es6.5系列的集群:

环境:三节点:master-172.16.23.128.node1-172.16.23.129.node2-172.16.23.130,首先查看es的服务状态:

[root@master ~]# ansible all_nodes -m shell -a "systemctl status elasticsearch"|grep -i running
   Active: active (running) since 六 2018-12-29 12:06:55 CST; 3h 33min ago
   Active: active (running) since 六 2018-12-29 12:07:43 CST; 3h 32min ago
   Active: active (running) since 六 2018-12-29 15:38:47 CST; 1min 42s ago

查看各节点上面的es的配置文件:

[root@master ~]# ansible all_nodes -m shell -a 'cat /etc/elasticsearch/elasticsearch.yml|egrep -v "^$|^#"'
172.16.23.128 | CHANGED | rc=0 >>
cluster.name: estest
node.name: esnode2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.23.128", "172.16.23.131"]

172.16.23.130 | CHANGED | rc=0 >>
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

172.16.23.129 | CHANGED | rc=0 >>
cluster.name: es
node.name: node1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200

现在基于discovery.zen做集群配置参考:https://www.elastic.co/guide/en/elasticsearch/reference/6.5/modules-discovery-zen.html,具体配置如下:

[root@master ~]# ansible all_nodes -m shell -a 'cat /etc/elasticsearch/elasticsearch.yml|egrep -v "^$|^#"'
172.16.23.128 | CHANGED | rc=0 >>
cluster.name: estest
node.name: master
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.23.128", "172.16.23.129", "172.16.23.130"]

172.16.23.130 | CHANGED | rc=0 >>
cluster.name: estest
node.name: node2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.23.128", "172.16.23.129", "172.16.23.130"]

172.16.23.129 | CHANGED | rc=0 >>
cluster.name: estest
node.name: node1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["172.16.23.128", "172.16.23.129", "172.16.23.130"]

重启elasticsearch服务:

[root@master ~]# ansible all_nodes -m shell -a "systemctl restart elasticsearch"
172.16.23.130 | CHANGED | rc=0 >>

172.16.23.128 | CHANGED | rc=0 >>

172.16.23.129 | CHANGED | rc=0 >>

然后查看集群状态:

[root@master ~]# curl -X GET "localhost:9200/_cluster/health" -s|python -m json.tool
{
    "active_primary_shards": 0,
    "active_shards": 0,
    "active_shards_percent_as_number": 100.0,
    "cluster_name": "estest",
    "delayed_unassigned_shards": 0,
    "initializing_shards": 0,
    "number_of_data_nodes": 3,
    "number_of_in_flight_fetch": 0,
    "number_of_nodes": 3,
    "number_of_pending_tasks": 0,
    "relocating_shards": 0,
    "status": "green",
    "task_max_waiting_in_queue_millis": 0,
    "timed_out": false,
    "unassigned_shards": 0
}

查看节点个数:

[root@master ~]# curl -X GET "localhost:9200/_cat/nodes?v"
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.16.23.128           28          71   3    0.04    0.11     0.08 mdi       *      master
172.16.23.130           29          67   4    0.04    0.11     0.10 mdi       -      node2
172.16.23.129           28          58   4    0.12    0.20     0.13 mdi       -      node1

单单只看master节点:

[root@master ~]# curl -X GET "localhost:9200/_cat/master?v"
id                     host          ip            node
hVY-U_ocQueMtcryoGGbTg 172.16.23.128 172.16.23.128 master

查看集群health:

[root@master ~]# curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1546070536 08:02:16  estest  green           3         3      0   0    0    0        0             0                  -                100.0%

查看nodeattrs属性:

[root@master ~]# curl -X GET "localhost:9200/_cat/nodeattrs?v"
node   host          ip            attr              value
master 172.16.23.128 172.16.23.128 ml.machine_memory 3956293632
master 172.16.23.128 172.16.23.128 xpack.installed   true
master 172.16.23.128 172.16.23.128 ml.max_open_jobs  20
master 172.16.23.128 172.16.23.128 ml.enabled        true
node2  172.16.23.130 172.16.23.130 ml.machine_memory 3956293632
node2  172.16.23.130 172.16.23.130 ml.max_open_jobs  20
node2  172.16.23.130 172.16.23.130 xpack.installed   true
node2  172.16.23.130 172.16.23.130 ml.enabled        true
node1  172.16.23.129 172.16.23.129 ml.machine_memory 3956293632
node1  172.16.23.129 172.16.23.129 ml.max_open_jobs  20
node1  172.16.23.129 172.16.23.129 xpack.installed   true
node1  172.16.23.129 172.16.23.129 ml.enabled        true

现在手动创建一个index为test:

# curl -X PUT "localhost:9200/test"

然后查看各节点index情况:

[root@master ~]# ansible all_nodes -m shell -a 'curl -X GET "localhost:9200/_cat/indices?v" -s'
 [WARNING]: Consider using the get_url or uri module rather than running curl.  If you need to use command because get_url or uri is insufficient you can add
warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.

172.16.23.128 | CHANGED | rc=0 >>
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test  l0Js1PJLTPSFEdXhanVSHA   5   1          0            0      1.7kb          1.1kb

172.16.23.130 | CHANGED | rc=0 >>
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test  l0Js1PJLTPSFEdXhanVSHA   5   1          0            0      1.7kb          1.1kb

172.16.23.129 | CHANGED | rc=0 >>
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test  l0Js1PJLTPSFEdXhanVSHA   5   1          0            0      1.7kb          1.1kb

查看index的分片情况:

[root@master ~]# curl -X GET "localhost:9200/_cat/shards?v"
index shard prirep state      docs store ip            node
test  3     p      STARTED       0  230b 172.16.23.128 master
test  3     r      STARTED       0  230b 172.16.23.130 node2
test  2     r      STARTED       0  230b 172.16.23.129 node1
test  2     p      STARTED       0  230b 172.16.23.130 node2
test  1     p      STARTED       0  230b 172.16.23.129 node1
test  1     r      UNASSIGNED
test  4     p      STARTED       0  230b 172.16.23.129 node1
test  4     r      UNASSIGNED
test  0     p      STARTED       0  230b 172.16.23.128 master
test  0     r      STARTED       0  230b 172.16.23.130 node2

由上面可以看出有两个分片是UNASSIGNED状态,查看集群health:

[root@master ~]# curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1546071645 08:20:45  estest  yellow          3         3      8   5    0    0        2             0                  -                 80.0%

使用下面的命令定位有问题的分片以及原因:

[root@master ~]# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason -s| grep UNASSIGNED
test 1 r UNASSIGNED INDEX_CREATED
test 4 r UNASSIGNED INDEX_CREATED

获取分片更多信息:

[root@master ~]# curl -XGET localhost:9200/_cluster/allocation/explain?pretty
{
  "index" : "test",
  "shard" : 1,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2018-12-29T08:14:47.378Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "hVY-U_ocQueMtcryoGGbTg",
      "node_name" : "master",
      "transport_address" : "172.16.23.128:9300",
      "node_attributes" : {
        "ml.machine_memory" : "3956293632",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20",
        "ml.enabled" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "node_version",
          "decision" : "NO",
          "explanation" : "cannot allocate replica shard to a node with version [6.5.2] since this is older than the primary version [6.5.4]"
        }
      ]
    },
    {
      "node_id" : "q95yZ4W4Tj6PaXyzLZZYDQ",
      "node_name" : "node1",
      "transport_address" : "172.16.23.129:9300",
      "node_attributes" : {
        "ml.machine_memory" : "3956293632",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[test][1], node[q95yZ4W4Tj6PaXyzLZZYDQ], [P], s[STARTED], a[id=j7V8PBUvQnOZzISPAxK9Uw]]"
        }
      ]
    },
    {
      "node_id" : "_ADSWG04TEqNfX_88ejtzQ",
      "node_name" : "node2",
      "transport_address" : "172.16.23.130:9300",
      "node_attributes" : {
        "ml.machine_memory" : "3956293632",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "ml.enabled" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 3,
      "deciders" : [
        {
          "decider" : "node_version",
          "decision" : "NO",
          "explanation" : "cannot allocate replica shard to a node with version [6.5.2] since this is older than the primary version [6.5.4]"
        }
      ]
    }
  ]
}

由上面结果可知node1,node2的es版本不同于master的es版本:

[root@master ~]# ansible all_nodes -m shell -a 'rpm -qa|grep elasticsearch'
 [WARNING]: Consider using the yum, dnf or zypper module rather than running rpm.  If you need to use command because yum, dnf or zypper is insufficient you can
add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message.

172.16.23.128 | CHANGED | rc=0 >>
elasticsearch-6.5.2-1.noarch

172.16.23.130 | CHANGED | rc=0 >>
elasticsearch-6.5.2-1.noarch

172.16.23.129 | CHANGED | rc=0 >>
elasticsearch-6.5.4-1.noarch

将其中上面版本不一致的替换掉后,开启es服务,然后观察集群以及shards情况:

[root@master ~]# curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1546073143 08:45:43  estest  red             1         1      2   2    0    0        8             0                  -                 20.0%
[root@master ~]# curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1546073274 08:47:54  estest  green           3         3     10   5    0    0        0             0                  -                100.0%

[root@master ~]# curl -X GET "localhost:9200/_cat/shards?v"
index shard prirep state   docs store ip            node
test  3     p      STARTED    0  261b 172.16.23.128 master
test  3     r      STARTED    0  261b 172.16.23.130 node2
test  4     r      STARTED    0  261b 172.16.23.128 master
test  4     p      STARTED    0  261b 172.16.23.129 node1
test  2     r      STARTED    0  261b 172.16.23.129 node1
test  2     p      STARTED    0  261b 172.16.23.130 node2
test  1     p      STARTED    0  261b 172.16.23.129 node1
test  1     r      STARTED    0  261b 172.16.23.130 node2
test  0     p      STARTED    0  261b 172.16.23.128 master
test  0     r      STARTED    0  261b 172.16.23.130 node2

索引test由10个分片组成,五个主分片,5个replica shard,replica shard是primary shard的副本，负责容错，以及承担读请求负载,primary shard的数量在创建索引的时候就固定了，replica shard的数量可以随时修改,primary shard的默认数量是5，replica默认是1

[root@master ~]# curl -XGET localhost:9200/test?pretty
{
  "test" : {
    "aliases" : { },
    "mappings" : { },
    "settings" : {
      "index" : {
        "creation_date" : "1546071287243",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "l0Js1PJLTPSFEdXhanVSHA",
        "version" : {
          "created" : "6050299"
        },
        "provided_name" : "test"
      }
    }
  }
}

[root@master ~]# curl -XGET localhost:9200/_cat/indices?v
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test  l0Js1PJLTPSFEdXhanVSHA   5   1          0            0      2.5kb          1.2kb

primary shard不能和自己的replica shard放在同一个节点上（否则节点宕机，primary shard和副本都丢失，起不到容错的作用），但是可以和其他primary shard的replica shard放在    同一个节点上

节点以及shards数分配参考:https://blog.csdn.net/qq_38486203/article/details/80077844

然后这里梳理一下es中一些基础概念:

1.cluster:

集群，一个ES集群由一个或多个节点（Node）组成，每个集群都有一个cluster name作为标识

2.node:

节点，一个ES实例就是一个node，一个机器可以有多个实例，一个集群由多个节点构成,大多数情况下每个node运行在一个独立的环境或虚拟机上。

3.index:

索引，即一系列documents的集合

3.shard:

分片，ES是分布式搜索引擎，每个索引有一个或多个分片，索引的数据被分配到各个分片上，相当于一桶水用了N个杯子装

分片有助于横向扩展，N个分片会被尽可能平均地（rebalance）分配在不同的节点上（例如你有2个节点，4个主分片(不考虑备份)，那么每个节点会分到2个分片，后来你增加了2个节点，那么你这4个节点上都会有1个分片，这个过程叫relocation，ES感知后自动完成)

分片是独立的，对于一个Search Request的行为，每个分片都会执行这个Request.另外每个分片都是一个Lucene Index，所以一个分片只能存放 Integer.MAX_VALUE - 128 = 2,147,483,519 个docs

4.replica:

复制，可以理解为备份分片，相应地有primary shard（主分片）

主分片和备分片不会出现在同一个节点上（防止单点故障），默认情况下一个索引创建5个分片一个备份（即5primary+5replica=10个分片）

如果你只有一个节点，那么5个replica都无法分配（unassigned），此时cluster status会变成Yellow。

ES集群的三种状态:

Green: 所有主分片和备份分片都准备就绪,分配成功, 即使有一台机器挂了(假设一台机器实例),数据都不会丢失,但是会变成yellow状态.

Yellow: 所有主分片准备就绪,但至少一个主分片(假设是A)对应的备份分片没有就绪,此时集群处于告警状态,意味着高可用和容灾能力下降.如果刚好A所在的机器挂了,并且你只设置了一个备份(已处于未继续状态), 那么A的数据就会丢失(查询不完整),此时集群处于Red状态.

Red:至少有一个主分片没有就绪(直接原因是找不到对应的备份分片成为新的主分片),此时查询的结果会出现数据丢失(不完整).

容灾：primary分片丢失，replica分片就会被顶上去成为新的主分片，同时根据这个新的主分片创建新的replica，集群数据安然无恙

提高查询性能：replica和primary分片的数据是相同的，所以对于一个query既可以查主分片也可以查备分片，在合适的范围内多个replica性能会更优（但要考虑资源占用也会提升[cpu/disk/heap]），另外index request只能发生在主分片上，replica不能执行index request。

对于一个索引，除非重建索引否则不能调整分片的数目（主分片数, number_of_shards），但可以随时调整replica数(number_of_replicas)。