Elasticsearch7.6 集群部署、集群认证及使用、数据备份

window 环境部署集群

注意：window下载解压elasticsearch一定需要解压多次。例如搭建的3节点的，需要解压3次，防止生成 cluster UUID 一致导致只能看到一个节点

1、elasticsearch.yml配置：

node.name区别：elastic_node1、 elastic_node2、 elastic_node3

cluster.name: elastic_cluster
 
node.name: elastic_node1
 
node.master: true
node.data: true
 
#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs
 
bootstrap.memory_lock: true
 
network.host: 0.0.0.0
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb
 
transport.tcp.port:
transport.tcp.compress: true
 
http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 
 
discovery.seed_hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.initial_master_nodes: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.fault_detection.leader_check.interval: 15s
discovery.cluster_formation_warning_timeout: 30s
cluster.join.timeout: 30s
cluster.publish.timeout: 90s
cluster.routing.allocation.cluster_concurrent_rebalance:
cluster.routing.allocation.node_concurrent_recoveries:
cluster.routing.allocation.node_initial_primaries_recoveries:

2、依次运行生成集群

浏览器打开：http://127.0.0.1:9201/_cat/nodes?v

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1                                                 dilm      -      elastic_node3
127.0.0.1                                                 dilm      -      elastic_node1
127.0.0.1                                                 dilm      *      elastic_node2

3、生成证书

es集群通过证书来安全的组成集群

运行
```
bin/elasticsearch-certutil cert
```

注意：密码后面需要单独设置，这里是集群安全认证，建议密码不设置，成功后生成的证书默认在es的config目录里面 elastic-certificates.p12；分别copy一份到其他节点的config里面（默认目录）

在elasticsearch.yml配置添加

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

4、给认证的集群创建用户密码

bin/elasticsearch-setup-passwords interactive

elastic 账号：拥有 superuser 角色，是内置的超级用户。
kibana 账号：拥有 kibana_system 角色，用户 kibana 用来连接 elasticsearch 并与之通信。Kibana 服务器以该用户身份提交请求以访问集群监视 API 和 .kibana 索引。不能访问 index。
logstash_system 账号：拥有 logstash_system 角色。用户 Logstash 在 Elasticsearch 中存储监控信息时使用。
beats_system账号：拥有 beats_system 角色。用户 Beats 在 Elasticsearch 中存储监控信息时使用。

5、配置kibana认证

elasticsearch.username: "kibana"
elasticsearch.password: ""

完整的elasticsearch.yml配置，注意不同节点node.name区别

cluster.name: elastic_cluster
node.name: elastic_node1
node.master: true
node.data: true
 
#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs
 
bootstrap.memory_lock: true
 
network.host: 0.0.0.0
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb
 
transport.tcp.port:
transport.tcp.compress: true
 
http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 
 
discovery.seed_hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.initial_master_nodes: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.fault_detection.leader_check.interval: 15s
discovery.cluster_formation_warning_timeout: 30s
cluster.join.timeout: 30s
cluster.publish.timeout: 90s
cluster.routing.allocation.cluster_concurrent_rebalance:
cluster.routing.allocation.node_concurrent_recoveries:
cluster.routing.allocation.node_initial_primaries_recoveries: 
 
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

centos(docker-compose) 环境部署集群

1、docker-compose.yml配置

version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:
      - cluster.initial_master_nodes=es01,192.168.43.128:
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -
        hard: -
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
    ports:
      - :
      - :
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts..host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local
 
networks:
  elastic:
    driver: bridge

权限问题执行 chmod -R 777 mnt/*

2、生成证书文件创建密码

进入容器 docker exec -it 5144d3b1dd56 /bin/bash
生成证书 bin/elasticsearch-certutil cert
复制证书并cp到其他节点 docker cp 09f57b6067e0:/usr/share/elasticsearch/elastic-certificates.p12 .

3、修改配置&&动态添加测试

version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:
      - cluster.initial_master_nodes=es01,192.168.43.128:
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.license.self_generated.type=basic
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=elastic-certificates.p12
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -
        hard: -
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
      - ./mnt/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - :
      - :
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts..host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local
 
networks:
  elastic:
    driver: bridge

注意证书的位置，给权限 chmod -R 777 mnt/*

设置密码(建议进入主节点容器中) bin/elasticsearch-setup-passwords interactive -u 'http://es01:9200'
通用配置与window类似

springboot使用测试

1、引入pom

  <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.6.</version>
        </dependency>
 
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.6.</version>
        </dependency>
 
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.6.</version>
        </dependency>

2、代码

EsConfiguration.class

/**
 * @author hdy
 */
@Configuration
public class EsConfiguration {
    /**
     * 集群地址
     */
    private static String hosts = "192.168.43.128";
    private static String hosts1 = "192.168.43.129";
    private static String hosts2 = "192.168.43.130";
    /**
     * 使用的端口号
     */
    private static int port = ;
    /**
     * // 使用的协议
     */
    private static String schema = "http";
    private static ArrayList<HttpHost> hostList = null;
    /**
     * 连接超时时间
     */
    private static int connectTimeOut = ;
    /**
     * 连接超时时间
     */
    private static int socketTimeOut = ;
    /**
     * 获取连接的超时时间
     */
    private static int connectionRequestTimeOut = ;
    /**
     * 最大连接数
     */
    private static int maxConnectNum = ;
    /**
     * 最大路由连接数
     */
    private static int maxConnectPerRoute = ;
 
    private RestClientBuilder builder;
 
    private final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
 
    static {
        hostList = new ArrayList<>();
        hostList.add(new HttpHost(hosts, port, schema));
        hostList.add(new HttpHost(hosts1, port, schema));
        hostList.add(new HttpHost(hosts2, port, schema));
    }
 
    @Bean("restHighLevelClient")
    public RestHighLevelClient client() {
        builder = RestClient.builder(hostList.toArray(new HttpHost[]));
        setConnectTimeOutConfig();
        setMutiConnectConfig();
        return new RestHighLevelClient(builder);
    }
 
    /**
     * 异步httpclient的连接延时配置
     */
    private void setConnectTimeOutConfig() {
        builder.setRequestConfigCallback(requestConfigBuilder -> {
            requestConfigBuilder.setConnectTimeout(connectTimeOut);
            requestConfigBuilder.setSocketTimeout(socketTimeOut);
            requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
            return requestConfigBuilder;
        });
    }
 
    /**
     * 异步httpclient的连接数配置
     */
    private void setMutiConnectConfig() {
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", ""));
        builder.setHttpClientConfigCallback(httpClientBuilder -> {
            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            httpClientBuilder.setMaxConnTotal(maxConnectNum);
            httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute);
            return httpClientBuilder;
        });
    }
 
}

ElasticsearchApplicationTests.class

@Log4j2
@RunWith(SpringRunner.class)
@SpringBootTest
public class ElasticsearchApplicationTests {
 
    @Autowired
    RestHighLevelClient restHighLevelClient;
 
    @Test
    public void contextLoads() {
 
        for (int i = ; i >= ; i--) {
            Map<String, Object> jsonMap = new HashMap<>();
            jsonMap.put("name", "测试" + i);
            jsonMap.put("age", "" + i);
            jsonMap.put("des", "啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦拉了拉");
            jsonMap.put("des1", "des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1");
            jsonMap.put("des2", "des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2");
            jsonMap.put("des3", "des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3");
            jsonMap.put("des4", "des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4");
            jsonMap.put("postDate", new Date());
            jsonMap.put("message", "trying out Elasticsearch");
            IndexRequest indexRequest = new IndexRequest("test").id("" + i).source(jsonMap);
            try {
                IndexResponse response = null;
                try {
                    response = restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
                } catch (IOException e) {
                    e.printStackTrace();
                }
                log.info(response.toString());
            } catch (ElasticsearchException e) {
                if (e.status() == RestStatus.CONFLICT) {
                    System.out.println("e = " + e);
                }
            }
        }
 
        GetRequest getRequest = new GetRequest("posts", "");
        GetResponse response = null;
        try {
            response = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
            log.info(response.toString());
        } catch (IOException e) {
            e.printStackTrace();
        }
 
    }
 
}

错误信息,少引入elasticsearch-rest-client pom包

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'restHighLevelClient' defined in class path resource [com/dy/client/EsConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:) ~[spring-beans-5.1..RELEASE.jar:5.1..RELEASE]
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:) [junit-rt.jar:na]
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:) [junit-rt.jar:na]
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:) ~[spring-core-5.1..RELEASE.jar:5.1..RELEASE]
    ...  common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/client/Cancellable
    at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_191]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:) ~[spring-core-5.1..RELEASE.jar:5.1..RELEASE]
    ...  common frames omitted
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.client.Cancellable
    at java.net.URLClassLoader.findClass(URLClassLoader.java:) ~[na:1.8.0_191]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:) ~[na:1.8.0_191]
    ...  common frames omitted

ElasticSearch 6.x数据迁移ElasticSearch7.4

安装nodejs

yum install nodejs -y --registry=https://registry.npm.taobao.org

安装elasticdump
```
npm install elasticdump -g
```
升级nodejs
```
npm install -g n
n latest
```

mapping拷贝（建议手动拷贝）

elasticdump --input=http://127.0.0.1:9200/index(修改为真实的索引名称) --output=http://10.0.1.236:9200/index --type=mapping(修改为真实的索引名称与需要导入的IP)

data拷贝

# 创建认证文件
cat > authFile <<EOF
user=elastic
password=xxxxxx
EOF
 
# dump数据
elasticdump --input=http://192.168.0.110:9200/test-20200510 --output=http://192.168.0.215:9201/test-20200510 --type=data --limit=100000 --httpAuthFile=authFile

Elasticsearch7.6 集群部署、集群认证及使用、数据备份的更多相关文章

linux运维、架构之路-Kubernetes集群部署TLS双向认证
一.kubernetes的认证授权 Kubernetes集群的所有操作基本上都是通过kube-apiserver这个组件进行的,它提供HTTP RESTful形式的API供集群内外客户端调 ...
【集群实战】Rsync试题-异机数据备份解决方案
企业案例:Rsync上机实战考试题: 某公司有一台Web服务器,里面的数据很重要,但是如果硬盘坏了,数据就会丢失,现在领导要求你把数据在其它机器上做一个周期性定时备份. 要求如下: 每天晚上00点整在 ...
elk 系列：Elasticsearch 7.2 集群部署+TLS 加密+认证登陆
背景 2019年5月21日,Elastic官方发布消息: Elastic Stack 新版本6.8.0 和7.1.0的核心安全功能现免费提供. 这意味着用户现在能够对网络流量进行加密.创建和管理用户. ...
面试连环炮系列（二）：你们的项目Redis做了集群部署吗
你们的项目Redis做了集群部署吗? 我们有大量数据需要缓存,而单实例的容量毕竟是有限的,于是做了Redis集群部署. 采取的方案是什么,Codis还是Redis Cluster,为什么要选择这个方案 ...
使用kubeadm进行单master(single master)和高可用(HA)kubernetes集群部署
kubeadm部署k8s 使用kubeadm进行k8s的部署主要分为以下几个步骤: 环境预装: 主要安装docker.kubeadm等相关工具. 集群部署: 集群部署分为single master(单 ...
部署Redis Cluster 6.0 集群并开启密码认证和 Redis-cluster-proxy负载
部署Redis Cluster集群并开启密码认证如果只想简单的搭建Redis Cluster,不需要设置密码和公网访问,可以参考官方文档. 节点介绍 Cluster模式推荐最少有6个节点,本次实验搭 ...
CentOS6.8部署MongoDB集群及支持auth认证
三个节点的副本集如下图所示: 实验目的: 配置MongoDB的3节点副本集 3个节点的副本集都要开启auth认证,并且开启认证后,能互相通信第一步 - 准备环境准备三个虚拟机,其中一个用作Prim ...
ELK + filebeat集群部署
ELK + filebeat集群部署一.ELK简介 1. Elasticsearch Elasticsearch是一个实时的分布式搜索分析引擎, 它能让你以一个之前从未有过的速度和规模,去探索你的数 ...
OpenStack Swift集群部署流程与简单使用
之前介绍了<OpenStack Swift All In One安装部署流程与简单使用>,那么接下来就说一说Swift集群部署吧. 1. 简介本文档详细描述了使用两台PC部署一个小型Sw ...

随机推荐

Sharding-JDBC实现水平拆分-单库分表
参考资料:猿天地 https://mp.weixin.qq.com/s/901rNhc4WhLCQ023zujRVQ 作者:尹吉欢当单表的数量急剧上升,超过了1千万以上,这个时候就要对表进行水平 ...
Sharding-JDBC主键生成策略
当使用分库分表等功能之后,就不能再依赖数据库自带的主键生成机制了,一方面主键ID不能重复,另外需要在新增之前就知道主键ID,才能保证ID能够均匀分布到不同的数据库或数据表中,所以要使用一个合理的主键生 ...
Idea快捷生成serialVersionUID
Java对象实现了Serializable接口,是需要创建serialVersionUID,避免此对象在序列化.反序列化时出现问题.但idea默认没有生成serialVersionUID的设置,需要手 ...
8月1日起全部无版号游戏下架，ios手游想上架看这里！
在苹果至中国游戏开发者的邮件中声明:如果开发者不能在7月31日前提交版号及相关文件,付费游戏将不可以在中国AppStore供应.也就是说: 从8月1日开始,苹果将正式下架全部.所有的ios付费 ...
018_go语言中的结构体
代码演示 package main import "fmt" type person struct { name string age int } func main() { fm ...
three.js 着色器材质之变量（三）
这篇郭先生在练习一下着色器变量,在度娘上面或者官网上经常看到类似水波一样的效果,这篇就试着做一个这样的效果,顺便巩固一下顶点着色器和片元着色器,毕竟多多练习才能更好地掌握.效果如下图,在线案例请点击博 ...
Java 二维数组及方法概况
数组数组是指一组数据的集合,数组中的每个数据被称作元素.在数组中可以存放任意类型的元素,但同一个数组里存放的元素类型必须一致. 数组的定义在Java中,可以使用以下格式来定义一个数组. 数据类型[ ...
2020-06-07：mysql中varchar类型的id，where id=1，会用到索引吗？int 类型的id，where id="1"，会用到索引吗？为什么？
福哥答案2020-06-07: 答案来自群员:对于int类型id,查询的varchar 类型 ‘1’会隐式转换成 1,‘1’和 1都能正常走索引:对于varchar类型id,查询的int 类型 1不会 ...
CSS卡片右上角标记样式设计
template <div class="each-one-in-list"> <div class="show-icon">进行中&l ...
hdfs学习（一）
一.hdfs概述介绍: 在现代的企业环境中,单机容量往往无法存储大量数据,需要跨机器存储.统一管理分布在集群上的文件系统称为分布式文件系统 .HDFS(Hadoop Distributed File ...