Now posted on the Elastic blog

December 12, 2018 update: This article has been published on Elastic’s website as: https://www.elastic.co/blog/elasticsearch-security-configure-tls-ssl-pki-authentication

Introduction

When Elasticsearch security is enabled for a cluster that is running with a production license, the use of TLS/SSL for transport communications is obligatory and must be correctly setup. Additionally, once security has been enabled, all communications to an Elasticsearch cluster must be authenticated, including communications from Kibana and/or application servers.

The simplest way that Kibana and/or application servers can authenticate to an Elasticsearch cluster is by embedding a username and password in their configuration files or source code. However, in many organizations, it is forbidden to store usernames and passwords in such locations. In this case, one alternative is to use Public Key Infrastructure (PKI) (client certificates) for authenticating to an Elasticsearch cluster.

Configuring security along with TLS/SSL and PKI can seem daunting at first, and so this blog gives step-by-step instructions on how to: enable security; configure TLS/SSL; set passwords for built-in users; use PKI for authentication; and finally, how to authenticate Kibana to an Elasticsearch cluster using PKI.

Enabling security

In order to enable security it is necessary to have either a Gold or Platinum subscription, or a trial license enabled via Kibana or API. For example, the following command would enable a trial license via the API:

curl -X POST "localhost:9200/_xpack/license/start_trial?acknowledge=true"

Where localhost must be replaced with the name of a node in our Elasticsearch cluster.

After enabling a license, security can be enabled. We must modify the elasticsearch.yml file on each node in the cluster with the following line:

xpack.security.enabled: true

For a cluster that is running in production mode with a production license, once security is enabled, transport TLS/SSL must also be enabled. However, if we are running with a trial license, then transport TLS/SSL is not obligatory.

If we are running with a production license and we attempt to start the cluster with security enabled before we have enabled transport TLS/SSL, we will see the following error message:

Transport SSL must be enabled for setups with production licenses. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

Configuration of TLS/SSL is covered in the following sections.

TLS/SSL encryption

Elasticsearch has two levels of communications, transport communications and http communications. The transport protocol is used for internal communications between Elasticsearch nodes, and the http protocol is used for communications from clients to the Elasticsearch cluster. Securing these communications will be discussed in the following paragraphs.

Transport TLS/SSL encryption

The transport protocol is used for communication between nodes within an Elasticsearch cluster. Because each node in an Elasticsearch cluster is both a client and a server to other nodes in the cluster, all transport certificates must be both client and server certificates. If TLS/SSL certificates do not have Extended Key Usage defined, then they are already defacto client and server certificates. If transport certificates do have an Extended Key Usagesection, which is usually the case for CA-signed certificates used in corporate environments, then they must explicitly enable both clientAuth and serverAuth.

Note that Elasticsearch comes with a utility called elasticsearch-certutilthat can be used for generating self-signed certificates that can be used for encrypting internal communications within an Elasticsearch cluster.

The following commands can be used for generating certificates that can be used for transport communications, as described in this page on Encrypting Communications in Elasticsearch:

bin/elasticsearch-certutil ca
ENTER ENTER
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
ENTER ENTER ENTER

Once the above commands have been executed, we will have TLS/ SSL certificates that can be used for encrypting communications.

The newly created certificates should be copied into a sub-directory called certs located within the config directory. The certificates will then be specified in the elasticsearch.yml file as follows:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Now restart all of the nodes in our Elasticsearch cluster for the above changes to take effect.

Define built-in user’s passwords

We must now define passwords for the built-in users as described in Setting built-in user passwords. Note that if we are running with a Gold or Platinum license, the previous steps to enable TLS/SSL for the transport communications must be executed before the cluster will start. Additionally, note that setting built-in user’s passwords should be completed before we enable TLS/SSL for http communications, as the command for setting passwords will communicate with the cluster via unsecured http.

Built-in users passwords can be setup with the following command:

bin/elasticsearch-setup-passwords interactive

Be sure to remember the passwords that we have assigned for each of the built-in users. We will make use of the elastic superuser to help configure PKI authentication later in this blog.

Http TLS/SSL encryption

For http communications, the Elasticsearch nodes will only act as servers and therefore can use Server certificates —  i.e. http TLS/SSL certificates do not need to enable Client authentication.

In many cases, certificates for http communications would be signed by a corporate CA. It is worth noting that the certificates used for encrypting http communications can be totally independent from the certificates that are used for transport communications.

To reduce the number of steps in this blog, we’ll use the same certificates for http communications as we have already used for the transport communications. These are specified in elasticsearch.yml file as follows:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

Enabling PKI authentication

As discussed inConfiguring a PKI Realm, the following must be added to the elasticsearch.yml file to allow PKI authentication.

xpack.security.authc.realms.pki1.type: pki

Combined changes to elasticsearch.yml

Once the above steps have been followed, we should have the following defined in our elasticsearch.yml configuration:

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional xpack.security.authc.realms.pki1.type: pki

Note that once the above changes have been made to our elasticsearch.ymlfile, we will have to restart all of the Elasticsearch nodes in our cluster in order for the changes to take effect.

Creating a client certificate

Certificates that will be used for PKI authentication must be signed by the same CA as the certificates that are used for encrypting http communications. Normally, these would be signed by an official CA within an organization. However, because we have already used a self signed CA, we also sign our http client certificates with that same self-signed CA which we previously saved as elastic-stack-ca.p12. We can create a certificate for client authentication as follows:

bin/elasticsearch-certutil cert --ca \
config/certs/elastic-stack-ca.p12 \
-name "CN=something,OU=Consulting Team,DC=mydomain,DC=com"
ENTER
client.p12 ENTER
ENTER

The above will create a file called client.p12, which contains all of the information required for PKI authentication to our Elasticsearch cluster. However, in order to use this certificate it is helpful to break it into its private key, public certificate, and CA certificate. This can be done with the following commands:

Private Key

openssl pkcs12 -in client.p12 -nocerts -nodes  > client.key

Public Certificate

openssl pkcs12 -in client.p12 -clcerts -nokeys  > client.cer

CA Certificate

openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.cer

Which should produce three files,

  1. client.key —  The private key
  2. client.cer —  The public certificate
  3. client-ca.cer — The CA that signed the public certificate

Create a directory called certs in Kibana’s config directory, and move all of the client certificates there.

Configure Kibana to authenticate to elasticsearch

Now that we have enabled security on the Elasticsearch cluster, communications to the cluster must be authenticated. Therefore, if we plan on using Kibana to interact with the cluster, then we must enable security and configure Kibana to authenticate to the cluster as the kibana user over https. As we have not yet fully setup PKI authentication from Kibana to the Elasticsearch cluster, authentication can initially be done with the following lines in the kibana.yml file:

elasticsearch.url: "https://localhost:9200" #ensure https not http
xpack.security.enabled: true
elasticsearch.username: "kibana"
elasticsearch.password: "our new kibana password here"
elasticsearch.ssl.certificateAuthorities: config/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

Ensure that we change localhost to the name of one of our Elasticsearch nodes, and that the certificates are available in the config/certs directory within the Kibana folder.

Note that the kibana user is like a service account that works behind the scenes to authenticate the Kibana application to the Elasticsearch cluster. We will generally never directly login to the Elasticsearch cluster or into the Kibana UI as the kibana user.

Restart Kibana in order for it to authenticate to the Elasticsearch cluster as the kibana user. We should be able to now login through the Kibana UI as the elastic built-in superuser.

PKI Authentication

We can use the three new client certificate files to test PKI authentication to the cluster with curl. Open a new terminal and cd to Kibana’s config/certs directory, and use curl to call the authenticate API as shown below.

curl https://localhost:9200/_xpack/security/_authenticate?pretty \
--key client.key --cert client.cer --cacert client-ca.cer -k -v

Be sure to replace localhost with the name of a node in our Elasticsearch cluster and be sure to use https (not http). Also note that the -k option is required as we did not create certificates with the hostnames specified, and therefore hostname verification must be turned off.

The above command should respond with something similar to the following:

{
 "username" : "something",
 "roles" : [ ],
 "full_name" : null,
 "email" : null,
 "metadata" : {
   "pki_dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com"
 },
 "enabled" : true
}

Notice that the roles is currently empty which means that although we have authenticated to Elasticsearch, we are not authorized to perform any actions. Authentication is allowed because the client certificate that we sent to the cluster was signed by the same CA as the http TLS/SSL certificates used by the Elasticsearch nodes. Now that we are authenticated, we need to authorize this user to be able to do something.

The pki_dn value returned from the authenticate API will be used to configure the roles that will be assigned to this certificate.

Open the Kibana UI and if we have not already done so, login as the elasticuser. As the elastic user has superuser privileges, this user can assign roles to the certificate. Execute the following command from Dev Tools in Kibana, ensuring that the previously returned pki_dn value is copied into the dn field as follows:

PUT _xpack/security/role_mapping/kibana_certificate_authorization
{
 "roles" : [ "kibana_system" ],
 "rules" : { "field" : { "dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com" } },
 "enabled": true
}

Now that we have assigned kibana_system role to this certificate, verify this is set correctly with another call to the authenticate API:

curl https://localhost:9200/_xpack/security/_authenticate?pretty \
--key client.key --cert client.cer --cacert client-ca.cer -k -v

And we should see the following response, which indicates that we now have the “kibana_system” role assigned to this certificate.

{
 "username" : "something",
 "roles" : [
   "kibana_system"  ],
 "full_name" : null,
 "email" : null,
 "metadata" : {
   "pki_dn" : "CN=something, OU=Consulting Team, DC=mydomain, DC=com"
 },
 "enabled" : true
}

Using PKI to authenticate Kibana to the Elasticsearch cluster

Now that we have tested our client-side certificate and assigned the kibana_system role to the certificate, we can use this certificate instead of a username and password, to authenticate Kibana to Elasticsearch.

Remove the following lines from our kibana.yml file:

elasticsearch.username: "kibana"
elasticsearch.password: "XXXXXX"

Ensure that all relevant certificates are copied to Kibana’s config/certs directory, and add the following lines to our kibana.yml file:

elasticsearch.url: "https://localhost:9200" #ensure https
xpack.security.enabled: true
elasticsearch.ssl.certificate: config/certs/client.cer
elasticsearch.ssl.key: config/certs/client.key
elasticsearch.ssl.certificateAuthorities: config/certs/client-ca.cer
elasticsearch.ssl.verificationMode: certificate

We can now restart Kibana, and it should authenticate to our Elasticsearch cluster, without any need for an embedded username and password!

Conclusion

In this blog post, I have demonstrated how to: enable security; configure TLS/SSL; set passwords for built-in users; use PKI for authentication; and finally, how to authenticate Kibana to an Elasticsearch cluster using PKI.

If you have any questions about PKI authentication with Elasticsearch, or any other Elasticsearch-related topics, have a look at our Discuss forumsfor valuable discussion,  insights, and information.

[不错]A step-by-step guide to enabling security, TLS/SSL, and PKI authentication in Elasticsearch的更多相关文章

  1. Tomcat Clustering - A Step By Step Guide --转载

    Tomcat Clustering - A Step By Step Guide Apache Tomcat is a great performer on its own, but if you'r ...

  2. Step by step guide to set up master and slave machines(转)

    Note: There is no need to install Jenkins on the slave machine. On your master machine go to Manage ...

  3. Step by step guide to set up master and slave machines on Windows

    Note: There is no need to install Jenkins on the slave machine. On your master machine go to Manage ...

  4. Step by Step use OBD2 Scanner Guide

    Learning to use a good automotive OBD2 code reader is one of the best ways you can continually inves ...

  5. Step by step Dynamics CRM 2011升级到Dynamics CRM 2013

    原创地址:http://www.cnblogs.com/jfzhu/p/4018153.html 转载请注明出处 (一)检查Customizations 从2011升级到2013有一些legacy f ...

  6. WPF Step By Step 自定义模板

    WPF Step By Step 自定义模板 回顾 上一篇,我们简单介绍了几个基本的控件,本节我们将讲解每个控件的样式的自定义和数据模板的自定义,我们会结合项目中的具体的要求和场景来分析,给出我们实现 ...

  7. e2e 自动化集成测试 架构 实例 WebStorm Node.js Mocha WebDriverIO Selenium Step by step (四) Q 反回调

    上一篇文章“e2e 自动化集成测试 架构 京东 商品搜索 实例 WebStorm Node.js Mocha WebDriverIO Selenium Step by step (三) SqlServ ...

  8. e2e 自动化集成测试 架构 实例 WebStorm Node.js Mocha WebDriverIO Selenium Step by step (二) 图片验证码的识别

    上一篇文章讲了“e2e 自动化集成测试 架构 京东 商品搜索 实例 WebStorm Node.js Mocha WebDriverIO Selenium Step by step 一 京东 商品搜索 ...

  9. enode框架step by step之Staged event-driven architecture思想的运用

    enode框架step by step之Staged event-driven architecture思想的运用 enode框架系列step by step文章系列索引: 分享一个基于DDD以及事件 ...

随机推荐

  1. 7.Go-用户信息和系统文件目录

    7.1.获取操作系统用户信息 (1)os包及子包功能 os/exec包,负责执行外部命令 os/singal对输入信息的访问 os/user通过名次或ID 查询用户账号 (2)在os/user中提供了 ...

  2. TCP/IP 协议栈4层结构及3次握手4次挥手

    TCP/IP 协议栈是一系列网络协议的总和,是构成网络通信的核心骨架,它定义了电子设备如何连入因特网,以及数据如何在它们之间进行传输.TCP/IP 协议采用4层结构,分别是应用层.传输层.网络层和链路 ...

  3. 【每日一包0008】arr-diff

    [github地址:https://github.com/ABCDdouyae...] arr-diff 多个数组比较,过滤出第一个数组独有的内容 用法:arr-diff(arr1, arr2, ar ...

  4. vue使用echart中国地图

    /* 引入 ECharts (按需加载) 文档:http://echarts.baidu.com/tutorial.html#%E5%9C%A8%20webpack%20%E4%B8%AD%E4%BD ...

  5. HDU 6444 Neko's loop ( 2018 CCPC 网络赛 && 裴蜀定理 && 线段树 )

    题目链接 题意 : 给出一个 n 个元素的环.可以任意选择起点.选完起点后.可以行走 m 步.每次前进 k 个单位.所走到的点将产生正或负贡献.问你一开始得准备多少才能使得初始资金加上在环上获取最大利 ...

  6. Controllers的使用

    代码 angularjs.html <!doctype html> <html> <head> <meta charset="UTF-8" ...

  7. pgadmin4 csrf错误导致docker-compose postgres服务下线

    docker-compse up 启动的前台服务, 过一会就自动停止 检查半天,发现是pgadmin4没安装正确不断报400 和 csrf error 然后pgadmin4为啥报这个, 因为pytho ...

  8. C语言博客作业04数组

    0.展示PTA总分 1.本章学习总结 1.1 学习内容总结 1.int a[10];为定义数组,表示数组有10个数 2.数组的下标都是从0开始,到n-1结束 3.数组里元素的个数不能大于数组的长度 4 ...

  9. Java 内部类、成员类、局部类、匿名类等

    Java各种称呼类详解 Java有各种各样类,内部类.嵌套类.成员类.局部类(本地类).静态类.匿名类.文件类以及这些组合起来的称呼类,成员内部类,成员匿名类,成员嵌套类,本地匿名类等,真是多的不行, ...

  10. Spring boot之MyBatis

    文章目录1. 环境依赖2. 数据源2.1. 方案一 使用 Spring Boot 默认配置2.2. 方案二 手动创建3. 脚本初始化4. MyBatis整合4.1. 方案一 通过注解的方式4.1.1. ...