Elasticsearch使用小结之冷热分离

Elasticsearch使用小结之冷热分离

索引迁移

索引setting中的index.routing.allocation.exclude和index.routing.allocation.include可以用于指定索引分配与哪些节点。同时，这两个配置是可以在index存在的时候修改的，我们可以通过修改这两个配置的方式来迁移索引。
比如：
ES集群存在5个节点，ip分别为：
    192.168.1.101，
    192.168.1.102，
    192.168.1.103，
    192.168.1.104，
    192.168.1.105
1. 我们先创建一个索引，使其分配在192.168.1.101，192.168.1.102上
PUT test_index
{
    "settings":{
        "index":{
            "routing.allocation.exclude._ip":"192.168.1.101,192.168.1.102"
        }
    }
}
2. 使用cat shards API查看test_index的分片分布
GET _cat/shards/test_index
返回如下：
    test_index                   2  p STARTED            0     162b 192.168.1.105  Z1SgiFF
    test_index                   0  p STARTED            0     162b 192.168.1.103  qEu6eMp
    test_index                   3  p STARTED            0     162b 192.168.1.104   xlTltSO
    test_index                   5  p STARTED            0     162b 192.168.1.104  xlTltSO
    test_index                   4  p STARTED            0     162b 192.168.1.105   Z1SgiFF
    test_index                   1  p STARTED            0     162b 192.168.1.103  qEu6eMp

3. 修改配置
PUT test_index/_settings
{
    "settings":{
        "index":{
            "routing.allocation.exclude._ip":"192.168.1.103,192.168.1.104,192.168.1.105",
            "number_of_shards": 6
        }
    }
}
4. 查看分片
GET _cat/shards/test_index
    test_index                   2  p STARTED            0     162b 192.168.1.102  9OHYhSa
    test_index                   4  p STARTED            0     162b 192.168.1.101   -RdAJHx
    test_index                   0  p STARTED            0     162b 192.168.1.102  9OHYhSa
    test_index                   5  p STARTED            0     162b 192.168.1.104  xlTltSO -> 192.168.1.101   -RdAJHx
    test_index                   3  p STARTED            0     162b 192.168.1.101   -RdAJHx
    test_index                   5  p STARTED            0     162b 192.168.1.102  9OHYhSa

节点tag

在节点启动时，在192.168.1.101和192.168.1.102上指定了
bin/elasticsearch -Enode.attr.rack=rack1
其余三台指定了
bin/elasticsearch -Enode.attr.rack=rack2
使用 GET _nodes/stats 查看：
{
  "_nodes": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "cluster_name": "elasticsearch",
  "nodes": {
    "-RdAJHxHTf2kHNyfUwHHBw": {
      "timestamp": 1559467108509,
      "name": "-RdAJHx",
      "transport_address": "192.168.1.101:9200",
      "host": "192.168.1.101",
      "ip": "192.168.1.101:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack1"
      }
      ....
    },
    "9OHYhSaxRgKKu_H0q18KyA": {
      "timestamp": 1559467108628,
      "name": "9OHYhSaxRgKKu_H0q18KyA",
      "transport_address": "192.168.1.102:9200",
      "host": "192.168.1.102",
      "ip": "192.168.1.102:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack1"
      }
      ....
    },
    "qEu6eMp9SEK7mlB9HmjCFA": {
      "timestamp": 1559467110507,
      "name": "qEu6eMp",
      "transport_address": "192.168.1.103:9200",
      "host": "192.168.1.103",
      "ip": "192.168.1.103:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    },
    "xlTltSOCQmibG9HICbnQyw": {
      "timestamp": 1559467110507,
      "name": "xlTltSO",
      "transport_address": "192.168.1.104:9200",
      "host": "192.168.1.104",
      "ip": "192.168.1.104:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    },
    "Z1SgiFFrQMqrPKKGir1EXg": {
      "timestamp": 1559467110507,
      "name": "Z1SgiFF",
      "transport_address": "192.168.1.105:9200",
      "host": "192.168.1.105",
      "ip": "192.168.1.105:9200",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "tag": "rack2"
      }
      ....
    }
我们也可以通过指定attributes来控制索引的迁移，这里需要配置index.routing.allocation.exclude.tag或者index.routing.allocation.include.tag

冷热分离

结合上述，我们可以很方便的进行冷热分离，先将节点分类，划分为两块，分别用于存储热数据和冷数据，在新建索引时指定索引的index.routing.allocation.exclude.tag为冷节点(或者index.routing.allocation.include.tag为热节点，效果一样)，然后在索引不再有数据写入时指定index.routing.allocation.exclude.tag为热节点(与前面用的对应, 建议统一使用exclude或者include，两者混用时，由于两者不能覆盖，需要注意置空)
示例：
在上述的ES中，我们将rack1的作为热节点，rack2的作为冷节点。
1. PUT test_index
   {
      "settings":{
        "routing.allocation.exclude.tag":"rack2",
            "number_of_shards": 6
      }
   }
2. 数据写入
3. PUT test_index/_settings
   {
      "settings":{
        "routing.allocation.exclude.tag":"rack1"
      }
   }