kubernetes1.5.2集群部署过程--安全模式
使用https安全模式部署kubernetes集群,能保证集群通讯安全、有效限制非授权用户访问。但部署比非安全模式复杂的多。
本文为etcd、kubernetes集群中各个组件配置证书认证,所有组件通讯之间使用https通讯。
运行环境
宿主机:CentOS7 7.3.1611
关闭selinux
etcd 3.1.9
flunnel 0.7.1
docker 1.12.6
kubernetes 1.5.2
安装软件
yum install etcd kubernetes kubernetes-client kubernetes-master kubernetes-node flannel docker docker-devel docker-client docker-common -y
证书部署
cfssl
CFSSL是开源的PKI工具箱,可以创建一个轻松获取和操作证书的内部CA。该工具具有运行一个CA所需的全部功能。
运行CA需要一个CA证书和相应的私钥。私钥是极其敏感的数据,任何知道私钥的人都可以充当CA颁发证书,私钥的保护至关重要。
安装cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod a+x cfssl*
mv cfssl-certinfo_linux-amd64 cfssl-certinfo
mv cfssl_linux-amd64 cfssl
mv cfssljson_linux-amd64 cfssljson
签发证书
创建CA证书
创建 CA 配置文件
mkdir /root/ssl
cd /root/ssl
cat << EOF > ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
字段说明
ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile;
signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
server auth:表示client可以用该 CA 对server提供的证书进行验证;
client auth:表示server可以用该CA对client提供的证书进行验证;
创建 CA 证书签名请求
cat << EOF > ca-csr.json
{
"CN": "lykops.net",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GD",
"L": "SZ",
"O": "lykops.net",
"OU": "lykops.net"
}
]
}
EOF
生成 CA 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
签发kube-master证书
cat << EOF > kube-master-csr.json
{
"CN": "kube-master",
"hosts": [
"127.0.0.1",
"192.168.20.128",
"192.168.20.131",
"192.168.20.132",
"172.16.0.1",
"172.17.0.1",
"localhost",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.lykops.net",
"kubernetes.kube-system",
"kubernetes.kube-system.svc",
"kubernetes.kube-system.svc.lykops.net"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "GD",
"O": "kube-master",
"OU": "lykops.net"
}
]
}
EOF
如果hosts字段不为空则需要指定授权使用该证书的IP或域名列表。哪些主机需要访问,在hosts中指定。
生成证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-master-csr.json | cfssljson -bare kube-master
或者直接在命令行上指定相关参数:
echo '{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname="127.0.0.1,kubernetes,kubernetes.default" - | cfssljson -bare kubernetes
签发kubelet证书
cat << EOF > kubelet-csr.json
{
"CN": "kubelet",
"hosts": [
"127.0.0.1",
"192.168.20.128",
"192.168.20.131",
"192.168.20.132",
"172.16.0.1",
"172.17.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.lykops.net",
"kubernetes.kube-system",
"kubernetes.kube-system.svc",
"kubernetes.kube-system.svc.lykops.net"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "GD",
"O": "kubelet",
"OU": "lykops.net"
}
]
}
EOF
生成证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubelet-csr.json | cfssljson -bare kubelet
签发etcd证书
客户端连接证书
cat << EOF > etcd-client-csr.json
{
"CN": "etcd-client",
"hosts": [
"127.0.0.1",
"192.168.20.128"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "GD",
"O": "etcd-client",
"OU": "lykops.net"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-client-csr.json | cfssljson -bare etcd-client
集群连接证书
cat << EOF > etcd-member-csr.json
{
"CN": "etcd-member",
"hosts": [
"127.0.0.1",
"192.168.20.128"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SZ",
"L": "GD",
"O": "etcd-member",
"OU": "etcd"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-member-csr.json | cfssljson -bare etcd-member
校验证书
以kube-master证书为例
使用Opsnssl命令
openssl x509 -noout -text -in kubernetes.pem
...
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes
Validity
Not Before: Apr 5 05:36:00 2017 GMT
Not After : Apr 5 05:36:00 2018 GMT
Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
...
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:172.20.0.112, IP Address:172.20.0.113, IP Address:172.20.0.114, IP Address:172.20.0.115, IP Address:10.254.0.1
...
确认Issuer字段的内容和ca-csr.json一致; 确认Subject字段的内容和kubernetes-csr.json一致; 确认X509v3 Subject Alternative Name字段的内容和kubernetes-csr.json一致; 确认X509v3 Key Usage、Extended Key Usage字段的内容和ca-config.json中 kubernetesprofile一致;
使用Cfssl-Certinfo命令
cfssl-certinfo -cert kubernetes.pem
...
{
"subject": {
"common_name": "kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"kubernetes"
]
},
"issuer": {
"common_name": "Kubernetes",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "BeiJing",
"province": "BeiJing",
"names": [
"CN",
"BeiJing",
"BeiJing",
"k8s",
"System",
"Kubernetes"
]
},
"serial_number": "174360492872423263473151971632292895707129022309",
"sans": [
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"127.0.0.1",
"10.64.3.7",
"10.254.0.1"
],
"not_before": "2017-04-05T05:36:00Z",
"not_after": "2018-04-05T05:36:00Z",
"sigalg": "SHA256WithRSA",
...
下发证书
把etcd、ca全部拷贝到etcd服务器下的/etc/ssl/etcd,设置权限:chown
etcd:etcd /etc/ssl/etcd/*
把kube-master和etcd-client、ca全部拷贝到master服务器下的/etc/ssl/kube下,设置权限:chown
kube:kube /etc/ssl/kube/
把kubelet、ca、etcd-client全部拷贝到node服务器上的/etc/ssl/kube下,设置权限:chown
kube:kube /etc/ssl/kube/
部署etcd
cat /etc/etcd/etcd.conf
# [member]
ETCD_NAME=kube-master
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.128:2379,http://localhost:2379,http://localhost:4001"
#[cluster]
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.128:2379"
#[security]
ETCD_CERT_FILE="/etc/ssl/etcd/etcd-client.pem"
ETCD_KEY_FILE="/etc/ssl/etcd/etcd-client-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/ssl/etcd/ca.pem"
启动服务service etcd start
flanneld网络
配置flanneld服务
cat /etc/sysconfig/flanneld
FLANNEL_ETCD_ENDPOINTS="https://192.168.20.128:2379 --etcd-cafile=/etc/ssl/kube/ca.pem --etcd-certfile=/etc/ssl/kube/etcd-client.pem --etcd-keyfile=/etc/ssl/kube/etcd-client-key.pem"
FLANNEL_ETCD_PREFIX="/coreos.com/network"
#FLANNEL_OPTIONS=""
启动flannel服务
创建flannel网络(在etcd服务器上执行)
etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd-client.pem --key-file=/etc/ssl/etcd/etcd-client-key.pem mk /coreos.com/network/config '{"Network":"172.16.0.0/16"}'
etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd-client.pem --key-file=/etc/ssl/etcd/etcd-client-key.pem get /coreos.com/network/config
kube-master
API Server、controller-manager、scheduler三个服务部署在同一台主机上,所以无需使用https通讯,故使用普通的http方式进行通讯。
controllermanager-config
该文件为kubernetes集群中的组件(比如controllermanager等)、addons(比如dashboard等)提供集群组件之间通讯的安全验证配置文件。
其中下面的password、username为访问Server API的认证用户和密码,保存在kube-master服务器上,路径请见API Server配置文件中的--basic-auth-file
cat << EOF > /etc/kubernetes/kube-controllermanager-config
apiVersion: v1
kind: Config
users:
- name: controllermanager
user:
client-certificate: /etc/ssl/kube/kube-master.pem
client-key: /etc/ssl/kube/kube-master-key.pem
password: 1qaz2wsx
username: lykops
clusters:
- name: local
cluster:
certificate-authority: /etc/ssl/kube/ca.pem
server: https://192.168.20.128:6443
contexts:
- context:
cluster: local
user: controllermanager
name: my-context
current-context: my-context
EOF
apiserver服务
cat /etc/kubernetes/apiserver
###
# kubernetes system config
# The following values are used to configure the kube-apiserver
# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1 --basic-auth-file=/etc/kubernetes/useraccount.csv"
# The port on the local server to listen on.
KUBE_API_PORT="--insecure-port=8080 --secure-port=6443"
# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.20.128:2379 --etcd-cafile=/etc/ssl/kube/ca.pem --etcd-certfile=/etc/ssl/kube/etcd-client.pem --etcd-keyfile=/etc/ssl/kube/etcd-client-key.pem"
# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=172.17.0.0/16"
# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
# Add your own!
KUBE_API_ARGS="--log-dir=/var/log/kubernetes --client-ca-file=/etc/ssl/kube/ca.pem --tls-private-key-file=/etc/ssl/kube/kube-master-key.pem --tls-cert-file=/etc/ssl/kube/kube-master.pem "
--insecure-bind-address=127.0.0.1表示http端口开放在localhost上
--basic-auth-file=/etc/kubernetes/useraccount.csv登陆账号和密码,必须要配置,否则在后面会出现很多认证失败导致无法通讯的问题。
使用https访问API Server有两种方式:
1、不对称方式:CA证书+用户密码
2、对称方式:CA证书+签发的证书和密钥
重启 kube-apiserver 服务:systemctl restart kube-apiserver
config文件
cat /etc/kubernetes/config
###
# kubernetes system config
# kubernetes services, including
# kube-apiserver.service
# kube-controller-manager.service
# kube-scheduler.service
# kubelet.service
# kube-proxy.service
KUBE_LOGTOSTDERR="--logtostderr=false --log-dir=/var/log/kubernetes"
KUBE_LOG_LEVEL="--v=2"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://192.168.20.128:6443 --kubeconfig=/etc/kubernetes/kube-controllermanager-config"
Controller Manager服务
/etc/kubernetes/controller-manager
# The following values are used to configure the kubernetes controller-manager
# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--service-account-private-key-file=/etc/ssl/kube/kube-master-key.pem --root-ca-file=/etc/ssl/kube/ca.pem --master=http://localhost:8080"
scheduler服务
cat /etc/kubernetes/scheduler ### # kubernetes scheduler config KUBESCHEDULERARGS="--master=http://localhost:8080"
proxy服务
cat /etc/kubernetes/proxy
# kubernetes proxy config
KUBE_PROXY_ARGS="--master=http://localhost:8080"
如果日志报:
kube-controller-manager: E0830 17:08:37.826561 1557 controllermanager.go:558] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such file or directory
请执行
mkdir /etc/kubernetes/ca/
cp -rpf /etc/ssl/kube/ca.pem /etc/kubernetes/ca/
node
kubelet-config
cat << EOF > /etc/kubernetes/kubelet-config
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
client-certificate: /etc/ssl/kube/kubelet.pem
client-key: /etc/ssl/kube/kubelet-key.pem
password: 1qaz2wsx
username: lykops
clusters:
- name: local
cluster:
certificate-authority: /etc/ssl/kube/ca.pem
server: https://192.168.20.128:6443
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
EOF
kubelet服务
cat /etc/kubernetes/kubelet
###
# kubernetes kubelet (minion) config
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"
# The port for the info server to serve on
KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=kube-node1"
# location of the api-server
KUBELET_API_SERVER="--api-servers=https://192.168.20.128:6443 --client-ca-file=/etc/ssl/kube/ca.pem --tls-private-key-file=/etc/ssl/kube/kubelet-key.pem --tls-cert-file=/etc/ssl/kube/kubelet.pem --kubeconfig=/etc/kubernetes/kubelet-config"
# pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_ARGS="--cluster-domain=lykops.net --cluster_dns=172.17.114.114"
config文件
cat /etc/kubernetes/config
###
# kubernetes system config
# The following values are used to configure various aspects of all
# kubernetes services, including
# kube-apiserver.service
# kube-controller-manager.service
# kube-scheduler.service
# kubelet.service
# kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=false --log-dir=/var/log/kubernetes"
# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=2"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"
# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://192.168.20.128:6443 --kubeconfig=/etc/kubernetes/kubelet-config"
proxy服务为默认
kubernetes1.5.2集群部署过程--安全模式的更多相关文章
- kubernetes1.5.2集群部署过程--非安全模式
运行环境 宿主机:CentOS7 7.3.1611 关闭selinux etcd 3.1.9 flunnel 0.7.1 docker 1.12.6 kubernetes 1.5.2 安装软件 yum ...
- 超详细从零记录Hadoop2.7.3完全分布式集群部署过程
超详细从零记录Ubuntu16.04.1 3台服务器上Hadoop2.7.3完全分布式集群部署过程.包含,Ubuntu服务器创建.远程工具连接配置.Ubuntu服务器配置.Hadoop文件配置.Had ...
- centos6下redis cluster集群部署过程
一般来说,redis主从和mysql主从目的差不多,但redis主从配置很简单,主要在从节点配置文件指定主节点ip和端口,比如:slaveof 192.168.10.10 6379,然后启动主从,主从 ...
- Centos6.9下RabbitMQ集群部署记录
之前简单介绍了CentOS下单机部署RabbltMQ环境的操作记录,下面详细说下RabbitMQ集群知识,RabbitMQ是用erlang开发的,集群非常方便,因为erlang天生就是一门分布式语言, ...
- Elasticsearch学习总结 (Centos7下Elasticsearch集群部署记录)
一. ElasticSearch简单介绍 ElasticSearch是一个基于Lucene的搜索服务器.它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口.Elasticse ...
- hbase高可用集群部署(cdh)
一.概要 本文记录hbase高可用集群部署过程,在部署hbase之前需要事先部署好hadoop集群,因为hbase的数据需要存放在hdfs上,hadoop集群的部署后续会有一篇文章记录,本文假设had ...
- 2.Ceph 基础篇 - 集群部署及故障排查
文章转载自:https://mp.weixin.qq.com/s?__biz=MzI1MDgwNzQ1MQ==&mid=2247485243&idx=1&sn=e425c31a ...
- 分布式实时日志系统(一)环境搭建之 Jstorm 集群搭建过程/Jstorm集群一键安装部署
最近公司业务数据量越来越大,以前的基于消息队列的日志系统越来越难以满足目前的业务量,表现为消息积压,日志延迟,日志存储日期过短,所以,我们开始着手要重新设计这块,业界已经有了比较成熟的流程,即基于流式 ...
- 来了,老弟!__二进制部署kubernetes1.11.7集群
Kubernetes容器集群管理 Kubernetes介绍 Kubernetes是Google在2014年6月开源的一个容器集群管理系统,使用Go语言开发,Kubernetes也叫K8S.K8S是Go ...
随机推荐
- 【bzoj2401】陶陶的难题I “高精度”+欧拉函数+线性筛
题目描述 求 输入 第一行包含一个正整数T,表示有T组测试数据.接下来T<=10^5行,每行给出一个正整数N,N<=10^6. 输出 包含T行,依次给出对应的答案. 样例输入 7 1 10 ...
- hdu 2616 Kill the monster (DFS)
Kill the monster Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others) ...
- [CF1036C]Classy Numbers
题目大意:多个询问,每个询问问$[l,r](1\leqslant l\leqslant r\leqslant10^{18})$内有多少个数满足非零数位小于等于$3$. 题解:数位$DP$,$f_{i, ...
- JS DOM对象与jQuery对象的转换
JS转jQuery // 直接用$()来包裹 如同$(this) $(document) var jsObj = document.getElementById('test'); var jquery ...
- 洛谷 P2312 解方程 解题报告
P2312 解方程 题目描述 已知多项式方程: \(a_0+a_1x+a_2x^2+\cdots+a_nx^n=0\)求这个方程在 \([1,m]\) 内的整数解(\(n\) 和 \(m\) 均为正整 ...
- spring in action 学习笔记五:@Autowired这个注解如何理解
@Autowired这个注解的意思就是自动装配.他把一个bean对象自动装配到另一个对象中.下面的案例证明了spring的自动装配. 定义一个Sixi类.代码如下: package com.qls.a ...
- linxu安装方式
安装Linux操作系统的5种方法以及心得这几天没有调别的东西,想起自己还不太会在没有安装光盘的时候安装Linux,于是试了一下Linux的五种安装方法,下面是我的一些 篇一:安装Linux操作系统的5 ...
- 汕头市队赛 SRM13 T2
这道题很容易想到是二分 但是因为可能会爆LL 所以要加一波特判 #include<cstdio> #include<cstring> #include<algorithm ...
- ef unitofwork 主从表更新
readonly UnitOfWork _u = new UnitOfWork(); public M Get(int id) { return _u.T_MtnContractRepository( ...
- git 克隆一个新仓库
1.登陆git网页版,点击右上角创建新项目 2.更改project path(如果需要),填写project name,其它选项默认. 3.到本地要创建存放项目的目录下,打开git命令框,输入git ...