elastersearch7.6.1搭建及基本操作详解

搭建

修改系统参数

vim /etc/security/limits.conf

* soft nofile 655350
* hard nofile 655350
* soft nofile 655350
* hard nofile 655350
es soft memlock unlimited
es hard memlock unlimited

vim /etc/sysctl.conf

vm.max_map_count=262144
fs.file-max=655360
vm.swappiness=1

保存后执行 sysctl -p

编辑配置文件

vim elastersearch.yaml

cluster.name: elastersearch
node.name: cs-es1
path.data: /home/es/data
path.logs: /home/es/logs
​
# 是否锁住内存，避免交换内存带来性能损失，默认为false
bootstrap.memory_lock: true
​
# 本机ip
network.host: 192.168.1.2
http.port: 9200
transport.tcp.port: 9300
​
# 节点角色，默认data master client 都有,如果分开，则改为 false
node.data: true
node.master: true
​
# 所有节点ip
discovery.seed_hosts: ["192.168.1.2","192.168.1.3","192.168.1.4"]
​
# 初始化 master 节点
cluster.initial_master_nodes: ["192.168.1.2","192.168.1.3","192.168.1.4"]
discovery.zen.ping_timeout: 60s
discovery.zen.join_timeout: 60s
​
# 忽略不符合master的节点
discovery.zen.master_election.ignore_non_master_pings: true
​
# 选举最多同意数
discovery.zen.mininum_master_nodes: 1
gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1
​
# xpack,如果不需要开启，注释掉
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

此配置文件为模板，如需多台服务器组集群，则要将本模板的node.name、network.host字段更改为实际的名称和地址，discovery.seed_hosts和cluster.initial_master_nodes也要做相应的更改

若开启 xpath ,需要创建证书

执行命令生成证书

bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""

需要将此证书放在每台节点的 config/ 目录下

然后重启es，然后设置密码

需要为4个用户分别设置密码：elastic、kibana、logstash_system、beats_system

1.可以自动设置密码

bin/elasticsearch-setup-passwords auto

2.也可以手动设置密码

 bin/elasticsearch-setup-passwords interactive 

访问时加上密码访问即可

curl -u elastic:baidu@123 '192.168.1.2/_cat/nodes'

如果kibina等需要链接ES，需要在连接的配置文件中指定密码

加入systemd管理

vim /usr/lib/systemd/system/elasticsearch.service

[Unit]
Description=es user elasticsearch.service
After=network.target
​
[Service]
Type=simple
User=es
Group=es
ExecStart=/home/es/elasticsearch7/bin/elasticsearch
ExecStop=/bin/kill -9 $MAINPID
Restart=no
LimitMEMLOCK=infinity
LimitNOFILE=1048576
LimitNPROC=1048576
PrivateTmp=true
​
[Install]
WantedBy=multi-user.target

其中参数不太理解可以参考此博文：http://www.jinbuguo.com/systemd/systemd.service.html

重新加载systemd

systemctl daemon-reload
systemctl start elasticsearch.service

基本操作

1、查看集群健康状态

curl '10.231.13.177:9200/_cluster/health?pretty'
{
  "cluster_name" : "elasticsearch",
  "status" : "green",  //集群的状态红绿灯，绿：健康，黄：亚健康，红：病态
  "timed_out" : false,
  "number_of_nodes" : 4,  //节点数
  "number_of_data_nodes" : 1,  //数据节点数
  "active_primary_shards" : 0,  //分片数，0个Index库
  "active_shards" : 0,  //正常的分片数
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

## 查看指定索引库的健康状态
http://localhost:9200/_cluster/health/index_name?pretty

2、查询集群所有节点

curl '10.231.13.177:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.231.66.181           18          96  14    3.12    4.11     4.44 di        -      ngd-es5-elasticsearch-data-0
10.231.66.172           26          96  14    3.12    4.11     4.44 -         -      ngd-es5-elasticsearch-client-7795566789-6qc5x
10.231.66.183           40          96  14    3.12    4.11     4.44 m         -      ngd-es5-elasticsearch-master-0
10.231.64.85            35          99   2    0.37    0.63     0.76 m         *      ngd-es5-elasticsearch-master-1

 heap.percent：堆内存占的内存百分比
 ram.percent：物理内存占用百分比
cpu：表示使用的cpu核心
load_1m load_5m load_15m：1分钟 5分钟 15分钟 占用系统cpu百分比
node.role：表示节点能充当的角色主(m)、数据(di) 、client(-)节点，节点未区分开为dilm
master：表示当前是否为主节点，*表示当前为主

## 查看MASTER节点
curl '10.233.0.93:9200/_cat/master?v'
id                     host          ip            node
2oaDofmzQ6KQK9BlorvO0Q 10.233.110.41 10.233.110.41 ngd-es5-elasticsearch-master-1

3、列出所有索引

curl '10.231.13.177:9200/_cat/indices?v'
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   vfantest acyI9dArRAqTOHNAwGO_oQ   5   1          0            0       650b           650b

当前只有一个索引

4、查看各个节点的总分片数及分布情况，及磁盘使用率

curl '10.233.8.243:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
    28       84.8kb   284.4gb     57.8gb    342.2gb           83 10.233.64.133 10.233.64.133 ngd-es7-master-1
    29       68.2kb   140.3gb    201.9gb    342.2gb           40 10.233.66.72  10.233.66.72  ngd-es7-master-2
    29      106.4kb    91.4gb    250.8gb    342.2gb           26 10.233.65.219 10.233.65.219 ngd-es7-master-0

5、查看集群分片的详细分布，分片的大小等

curl '10.233.8.243:9200/_cat/shards?v'
index                           shard prirep state   docs  store ip            node
dialog-statistics-scene-202106  1     p      STARTED    0   283b 10.233.64.133 ngd-es7-master-1
dialog-statistics-scene-202106  1     r      STARTED    0   283b 10.233.65.219 ngd-es7-master-0
dialog-statistics-scene-202106  2     p      STARTED    0   283b 10.233.66.72  ngd-es7-master-2
dialog-statistics-scene-202106  2     r      STARTED    0   283b 10.233.65.219 ngd-es7-master-0
dialog-statistics-scene-202106  0     r      STARTED    0   283b 10.233.64.133 ngd-es7-master-1
dialog-statistics-scene-202106  0     p      STARTED    0   283b 10.233.66.72  ngd-es7-master-2
search-suggest-online-data      1     r      STARTED    0   283b 10.233.64.133 ngd-es7-master-1
search-suggest-online-data      1     p      STARTED    0   283b 10.233.65.219 ngd-es7-master-0
search-suggest-online-data      2     p      STARTED    0   283b 10.233.66.72  ngd-es7-master-2
search-suggest-online-data      2     r      STARTED    0   283b 10.233.65.219 ngd-es7-master-0
search-suggest-online-data      0     r      STARTED    0   283b 10.233.64.133 ngd-es7-master-1
search-suggest-online-data      0     p      STARTED    0   283b 10.233.66.72  ngd-es7-master-2

6、查看 segments 分布信息

curl '10.233.8.243:9200/_cat/segments?v'
index                    shard prirep ip            segment generation docs.count docs.deleted   size size.memory committed searchable version compound
data-record-index-202106 0     p      10.233.64.133 _0               0          9            0 19.4kb        6477 true      true       8.4.0   true
data-record-index-202106 0     r      10.233.66.72  _0               0          9            0 19.4kb        6477 true      true       8.4.0   true
data-record-index-202106 1     p      10.233.64.133 _0               0          7            0 19.4kb        6461 true      true       8.4.0   true
data-record-index-202106 1     p      10.233.64.133 _1               1          9            0 38.1kb        6621 true      true       8.4.0   true
data-record-index-202106 1     r      10.233.65.219 _0               0          7            0 19.4kb        6461 true      true       8.4.0   true
data-record-index-202106 1     r      10.233.65.219 _1               1          9            0 38.1kb        6621 true      true       8.4.0   true
data-record-index-202106 2     p      10.233.66.72  _0               0          4            0 16.2kb        6437 true      true       8.4.0   true
data-record-index-202106 2     p      10.233.66.72  _1               1          5            0 24.4kb        6541 true      true       8.4.0   true
data-record-index-202106 2     r      10.233.65.219 _0               0          4            0 16.2kb        6437 true      true       8.4.0   true
data-record-index-202106 2     r      10.233.65.219 _1               1          5            0 24.4kb        6541 true      true       8.4.0   true

7、创建一个索引：

## 可以在创建中指定settings 和 mappings，加-d '{}' 参数指定即可
curl -XPUT '10.231.13.177:9200/20210122?pretty' -H 'content-type: application/json' -d
'{
  "acknowledged" : true,
  "shards_acknowledged" : true
}'

settings格式：
   "settings": {
     "number_of_shards": 5,
     "number_of_replicas": 1
   }

mappings格式：
    "mappings": {
      "sms_up": {
        "dynamic_templates": [
          {
            "strings_as_keywords": {
              "match_mapping_type": "string",
              "mapping": {
                "type": "keyword"
              }
            }
          }
        ],

## 再次查看索引
curl '10.231.13.177:9200/_cat/indices?v'
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   vfantest acyI9dArRAqTOHNAwGO_oQ   5   1          0            0       650b           650b
yellow open   20210122 urFLBfkQRGmIQVcXqITG5w   5   1          0            0       260b           260b

看到名称为"20210122"的索引已经创建，它有5个私有的分片以及一个副本

8、插入和获取

现在来插入一些数据到集群索引，插入时必须要指定数据类型，例如 external 类型 ID为 1 的数据

curl -XPUT 'localhost:9200/20210122/external/1?pretty' -d '
> {
>     "name": "Tony",
>     "sex": "boy"
> }'

回显：
{
  "_index" : "20210122",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}

返回为""created" : true" 代表创建成功

获取 20210122 索引下类型为 external 、ID =1 的数据，pretty参数表示返回结果格式美观。

curl 'localhost:9200/20210122/external/1?pretty'
{
  "_index" : "20210122",
  "_type" : "external",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "name" : "Tony",
    "sex" : "boy"
  }
}

9、删除索引

curl -XDELETE 'localhost:9200/20210122'
{"acknowledged":true}

10、查看ES有哪些插件

curl '10.233.0.93:9200/_cat/plugins?v'
name                                          component                   version
ngd-es5-elasticsearch-master-2                analysis-ik                 5.4.2
ngd-es5-elasticsearch-master-2                discovery-kubernetes        5.4.2
ngd-es5-elasticsearch-master-2                elasticsearch-analysis-ansj 5.4.2.0
ngd-es5-elasticsearch-master-2                ingest-geoip                5.4.2
ngd-es5-elasticsearch-master-2                ingest-user-agent           5.4.2
ngd-es5-elasticsearch-master-2                x-pack                      5.4.2
...
ngd-es5-elasticsearch-client-7795566789-6w2ns x-pack                      5.4.2

11、查看当前ES节点信息

curl '10.233.0.93:9200'
{
  "name" : "ngd-es5-elasticsearch-client-7795566789-6w2ns",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "KJAcgz-xQ1awOxG4T-WOlA",
  "version" : {
    "number" : "5.4.2",
    "build_hash" : "929b078",
    "build_date" : "2017-06-15T02:29:28.122Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"
}

You Know，for Search 表示节点正常