在 K8s 中，kube-apiserver 使用 etcd 对 REST object 资源进行持久化存储，本文介绍如何配置生成自签 https 证书，搭建 etcd 集群给 apiserver 使用，并附相关坑点记录。

1. 安装 cfssl 工具

  1. cd /data/work
  3. wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O cfssl
  4. wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O cfssljson
  5. wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O cfssl-certinfo
  7. chmod +x cfssl*
  8. mv cfssl* /usr/local/bin/
  10. chmod +x cfssl*
  11. mv cfssl_linux-amd64 /usr/local/bin/cfssl
  12. mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
  13. mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

2. 创建 ca 证书

  1. cat > ca-csr.json <<EOF
  2. {
  3.   "CN": "etcd-ca",
  4.   "key": {
  5.       "algo": "rsa",
  6.       "size": 2048
  7.   },
  8.   "names": [
  9.     {
  10.       "C": "CN",
  11.       "ST": "Beijing",
  12.       "L": "Beijing",
  13.       "O": "etcd-ca",
  14.       "OU": "etcd-ca"
  15.     }
  16.   ],
  17.   "ca": {
  18.           "expiry": "87600h"
  19.   }
  20. }
  21. EOF
  23. cfssl gencert -initca ca-csr.json | cfssljson -bare ca
  25. => 会生成：ca-key.pem, ca.csr, ca.pem

3. 配置 ca 证书策略

  1. cat > ca-config.json <<EOF
  2. {
  3.   "signing": {
  4.       "default": {
  5.           "expiry": "87600h"
  6.         },
  7.       "profiles": {
  8.           "etcd-ca": {
  9.               "usages": [
  10.                   "signing",
  11.                   "key encipherment",
  12.                   "server auth",
  13.                   "client auth"
  14.               ],
  15.               "expiry": "87600h"
  16.           }
  17.       }
  18.   }
  19. }
  20. EOF

4. 配置 etcd 请求 csr

  1. cat > etcd-csr.json <<EOF
  2. {
  3.   "CN": "etcd",
  4.   "hosts": [
  5.     "127.0.0.1",
  6.     "etcd0-0.etcd",
  7.     "etcd1-0.etcd",
  8.     "etcd2-0.etcd"
  9.   ],
  10.   "key": {
  11.     "algo": "rsa",
  12.     "size": 2048
  13.   },
  14.   "names": [{
  15.     "C": "CN",
  16.     "ST": "Beijing",
  17.     "L": "Beijing",
  18.     "O": "etcd",
  19.     "OU": "etcd"
  20.   }]
  21. }
  22. EOF

5. 生成 etcd 证书

  1. cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca etcd-csr.json | cfssljson  -bare etcd
  3. => 会生成：etcd-key.pem, etcd.csr, etcd.pem
  5. mv etcd.pem etcd-server.pem
  6. mv etcd-key.pem etcd-server-key.pem

6. 创建 etcd cluster

yaml 文件：https://github.com/k8s-club/etcd-operator

  1. kubectl apply -f etcd-cluster.yaml

7. 查看 DNS 解析

dnsutils 安装：https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

  1. kubectl exec -it -n etcd dnsutils -- nslookup etcd
  3. Server:         9.165.x.x
  4. Address:        9.165.x.x#53
  6. Name:   etcd.etcd.svc.cluster.local
  7. Address: 9.165.x.x
  8. Name:   etcd.etcd.svc.cluster.local
  9. Address: 9.165.x.x
  10. Name:   etcd.etcd.svc.cluster.local
  11. Address: 9.165.x.x

8. 查看 etcd 集群状态

  1. kubectl exec -it -n etcd etcd0-0 -- sh
  3. /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint health
  5. +---------------------------+--------+-------------+-------+
  6. |         ENDPOINT          | HEALTH |    TOOK     | ERROR |
  7. +---------------------------+--------+-------------+-------+
  8. | https://etcd0-0.etcd:2379 |   true | 13.551982ms |       |
  9. | https://etcd1-0.etcd:2379 |   true | 13.540498ms |       |
  10. | https://etcd2-0.etcd:2379 |   true | 23.119639ms |       |
  11. +---------------------------+--------+-------------+-------+
  13. /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint status
  15. +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  16. |         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
  17. +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
  18. | http://etcd0-0.etcd:2379 | 4dde210279eea33a |  3.4.13 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
  19. | http://etcd1-0.etcd:2379 | 20669865d12a473b |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  20. | http://etcd2-0.etcd:2379 | 3f17922d1ed63113 |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
  21. +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

9. 验证 etcd 读写

  1. kubectl exec -it -n etcd etcd0-0 -- sh
  3. /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem put hello world
  4. OK
  6. /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem get hello
  7. hello
  8. world
  10. 查看所有 keys:
  11. /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem get "" --keys-only --prefix
  12. hello
  14. 查看所有 key-val:
  15. /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem get "" --prefix
  16. hello
  17. world

10. 配置 apiserver 请求 csr

  1. cat > apiserver-csr.json <<EOF
  2. {
  3.   "CN": "apiserver",
  4.   "hosts": [
  5.     "*.etcd"
  6.   ],
  7.   "key": {
  8.     "algo": "rsa",
  9.     "size": 2048
  10.   },
  11.   "names": [{
  12.     "C": "CN",
  13.     "ST": "Beijing",
  14.     "L": "Beijing",
  15.     "O": "apiserver",
  16.     "OU": "apiserver"
  17.   }]
  18. }
  19. EOF

11. 生成 apiserver 证书

  1. cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca apiserver-csr.json | cfssljson  -bare apiserver
  3. => 会生成：apiserver-key.pem, apiserver.csr, apiserver.pem
  5. mv apiserver.pem etcd-client-apiserver.pem
  6. mv apiserver-key.pem etcd-client-apiserver-key.pem

12. 创建 extension-apiserver

apiserver.yaml：通过 ConfigMap 将生成的 *.pem 证书挂载给 apiserver 使用

  1. ...
  2. containers:
  3.   - image: apiserver.xxxxx:latest
  4.     args:
  5.     - --etcd-servers=https://etcd0-0.etcd:2379
  6.     - --etcd-cafile=/etc/kubernetes/certs/ca.pem
  7.     - --etcd-certfile=/etc/kubernetes/certs/etcd-client-apiserver.pem
  8.     - --etcd-keyfile=/etc/kubernetes/certs/etcd-client-apiserver-key.pem
  9. ...
  1. kubectl apply -f apiserver.yaml

13. 坑点记录

13.1 证书 hosts 不对

  1. log:
  2. etcd0-0:
  3. {"level":"warn","ts":"2021-08-19T11:55:07.755Z","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:41226","server-name":"","error":"tls: first record does not look like a TLS handshake"}
  5. etcd1-0:
  6. {"level":"info","ts":"2021-08-19T11:54:16.830Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"[::]:2379"}
  7. {"level":"info","ts":"2021-08-19T11:54:16.838Z","caller":"etcdserver/server.go:716","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"30dd90df9a304e97","forward-ticks":18,"forward-duration":"4.5s","election-ticks":20,"election-timeout":"5s","active-remote-members":2}
  8. {"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"80c7f1f6c2848777","local-member-id":"30dd90df9a304e97","cluster-version":"3.4"}
  9. {"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
  11. etcd2-0:
  12. {"level":"warn","ts":"2021-08-19T11:54:17.782Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"9.165.x.x:40180","server-name":"etcd2-0.etcd","ip-addresses":["0.0.0.0","127.0.0.1"],"dns-names":["etcd0-0.etcd","etcd1-0.etcd","etcd2-0.etcd"],"error":"tls: \"9.165.x.x\" does not match any of DNSNames [\"etcd0-0.etcd\" \"etcd1-0.etcd\" \"etcd2-0.etcd\"] (lookup etcd1-0.etcd on 9.165.x.x:53: no such host)"}

解决：重新配置正确的 hosts 域名

13.2 证书 hosts 配置坑点

  1. "hosts": [
  2.     "127.0.0.1",
  3.     "etcd0-0.etcd",
  4.     "*.etcd" // 允许 * 泛域名，但不能为空 "" 或 *
  5.   ],

13.3 dns 设置参考

推荐设置 *.xxx.ns.svc，这样扩容后也不需要重签证书

参考：https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Go 代码参考如下：

  1. func genEtcdWildcardDnsName(namespace, serviceName string) []string {
  2. 	return []string{
  3. 		fmt.Sprintf("%s.%s.%s", serviceName, namespace, "svc"),
  4. 		fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, "svc"),
  5. 		fmt.Sprintf("%s.%s.%s", serviceName, namespace, DnsBase),
  6. 		fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, DnsBase),
  7. 	}
  8. }

13.4 leader/follower 已经建立成功了，但访问报错

  1. # /usr/local/bin/etcdctl put hello world
  2. {"level":"warn","ts":"2021-08-19T12:32:11.200Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-05ed1825-e70f-492a-af94-03c633d0affc/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
  3. Error: context deadline exceeded

解决：etcdctl 需要带证书访问

  1. /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd-server.pem --key=/etc/etcd/ssl/etcd-server-key.pem put hello world

13.5 http 与 https 之间不能切换

先通过 http 建立了 cluster，然后再用自签证书 https 来建立，这样就会报错：

  1. tls: first record does not look like a TLS handshake

经过验证：无论是从 http => https，还是从 https => http 的切换都会报这个错，因为一旦建立 cluster 成功，则把连接的协议(http/https) 写入到 etcd 存储里了，不能再更改连接协议。

解决：如果真正遇到需要切换协议，可尝试下面方式

  • 允许删除数据：删除后重新建立 cluster
  • 不允许删数据：可以尝试采用 snapshot & restore 进行快照与恢复操作

13.6 apiserver 可直接使用第 5 步生成的 etcd 证书吗？

经过验证，是可以直接使用 etcd 证书的，但生产上不建议这样使用。

生产上建议对 apiserver(或其他应用) 单独生成证书，可使用泛域名(*.xx.xx)、不同过期时间等方式灵活配置，也更有利于集群管控。

K8s 系列(三) - 如何配置 etcd https 证书？的更多相关文章

  1. Docker & k8s 系列三：在k8s中部署单个服务实例

    本章将会讲解: pod的概念,以及如何向k8s中部署一个单体应用实例. 在上面的篇幅中,我们了解了docker,并制作.运行了docker镜像,然后将镜像发布至中央仓库了.然后又搭建了本机的k8s环境 ...

  2. 给你的网站免费配置上 HTTPS 证书

    现在越来越多的网站或服务增加了 HTTPS 证书,苹果 AppStore.微信小程序等也已强制要求开发者需提供 HTTPS 的后端接口.在阿里云 / 腾讯云上有一年期的免费赛门铁克 SSL 证书可供尝 ...

  3. Nginx安装及配置免费HTTPS证书

    第一步:安装Nginx 安装Nginx 第二步:安装HTTPS证书( Let's Encrypt) 安装HTTPS证书 第三步骤:浏览器验证 Chrome浏览器打开开发者工具->Security ...

  4. startssl申请配置免费https证书

    之前给业务配置都是在沃通上申请免费证书,而后通过反向代理层的Nginx进行https认证. 今天来了个新需求,要求域名直接解析至阿里云SLB.https配置需要通过阿里云的控制台部署这倒无所谓,只是在 ...

  5. BizTalk开发系列(三十一)配置和使用HTTP适配器

    BizTalk的主机分别进程内主机和独立主机.但由于一直使用的是进程内主机,对于独立主机的认识比较模糊,前不久在做一个BizTalk的项目的时 候,个别系统使用HTTP的方式发布Txt之类的文本的.刚 ...

  6. [转帖]从零开始入门 K8s | 手把手带你理解 etcd

    从零开始入门 K8s | 手把手带你理解 etcd https://zhuanlan.zhihu.com/p/96721097 导读:etcd 是用于共享配置和服务发现的分布式.一致性的 KV 存储系 ...

  7. Docker & k8s 系列二：本机k8s环境搭建

    本篇将会讲解k8s是什么?本机k8s环境搭建,部署一个pod并演示几个kubectl命令,k8s dashboard安装. k8s是什么 k8s是kubernetes的简写,它是一个全新的基于容器技术 ...

  8. Docker & k8s 系列一：快速上手docker

    Docker & k8s 系列一:快速上手docker 本篇文章将会讲解:docker是什么?docker的安装,创建一个docker镜像,运行我们创建的docker镜像,发布自己的docke ...

  9. 教你快速撸一个免费HTTPS证书

    摘要: 免费 HTTPS 证书,了解一下? HTTPS 已成为业界标准,这篇博客将教你申请Let's Encrypt的免费 HTTPS 证书. 本文的操作是在 Ubuntu 16.04 下进行,使用 ...

随机推荐

  1. Hadoop 3.1.1 - 概述 - 集群安装

    Hadoop 集群安装 目标 本文描述了如何从少数节点到包含上千节点的大规模集群上安装和配置 Hadoop 集群.如果只是为了尝试,你可以先从单台机器上安装开始(参阅单节点安装). 本文并不包含诸如安 ...

  2. FastAPI实战：简易MockServe

    Mock 我个人理解就是为了伪造返回结果的东西,MockServe通常说的就是接口未开放完成但是现在需要调用,所以无论是通过转发还是直接访问这个url,其结果基本就是自己定义的 当然做仔细点就是 给个 ...

  3. 记一次mysql事务未提交导致锁未释放的问题

    记一次mysql事务未提交导致锁未释放的问题 ## 查看未提交的事务(3秒内未操作的事务) SELECT p.ID AS conn_id, P.USER AS login_user, P.HOST A ...

  4. Linux远程访问控制

    目录 一.SSH远程管理 1.1.定义 1.2.配置OpenSSH服务端 二.sshd服务支持两种验证方式 2.1.密码验证 2.2.使用SSH客户端程序 ssh远程登录 scp远程复制 sftp安全 ...

  5. docker-03

    Docker构建私有registry(仓库) #1 启动registry [root@docker ~]# docker run -d -p 5000:5000 --restart=always -- ...

  6. STM32—重定向printf和getchar函数到串口

    在STM32测试串口的时候经常需要在开发板和上位机之间传输数据,我们可以用c语言中的printf()函数和getchar()函数来简化传输. 以printf()为例: printf()函数实际上是一个 ...

  7. 计算文件的MD5值和sha256值

    1.计算文件的MD5值. 1)linux系统计算 MD5值:md5sum+文件名 sha256值:sha256su+文件名 2)windows系统计算 MD5值:利用Notepad++工具计算 sha ...

  8. NOIP 模拟 $16\; \rm Lost My Music$

    题解 \(by\;zj\varphi\) 一道凸包的题 设 \(\rm dep_u\) 表示节点 \(u\) 的深度,那么原式就可化为 \(-\frac{c_v-c_u}{dep_v-dep_u}\) ...

  9. 题解 party?

    传送门 挺遗憾的一个题 考场上想到的思路是题解的退化版,可以有71pts(赛时以为只有20pts),但因为这一场的策略原因没有打-- 首先发现颜色种类数很少,可以直接bitset上树剖维护,炸不了空间 ...

  10. vim宏录制

    宏录制 当你要重复某一个操作时,录制的宏可以很快地帮你完成任务. 准备文本 <!DOCTYPE html> <html lang="en"> <hea ...