kubernetes二进制部署k8s-master集群controller-manager服务unhealthy问题
一.问题现象
我们使用二进制部署k8s的高可用集群时,在部署多master时,kube-controller-manager服务提示Unhealthy
[root@ceph-01 system]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
这里我们查看得知kube-controller-manager的服务运行时提示有一些日志报错问题:
[root@ceph-01 system]# systemctl status kube-controller-manager -l
● kube-controller-manager.service - Kubernetes Controller Manager
Loaded: loaded (/etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2018-12-29 03:56:00 EST; 31min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 126295 (kube-controller)
Tasks: 8
Memory: 8.4M
CGroup: /system.slice/kube-controller-manager.service
└─126295 /usr/local/bin/kube-controller-manager --port=0 --secure-port=10252 --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig --authentication-kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem --experimental-cluster-signing-duration=8760h --root-ca-file=/etc/kubernetes/cert/ca.pem --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --controllers=*,bootstrapsigner,tokencleaner --horizontal-pod-autoscaler-use-rest-clients=true --horizontal-pod-autoscaler-sync-period=10s --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem --use-service-account-credentials=true --alsologtostderr=true --logtostderr=false --log-dir=/var/log/kubernetes --v=2
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: I1229 03:56:00.395082 126295 flags.go:33] FLAG: --version="false"
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: I1229 03:56:00.395093 126295 flags.go:33] FLAG: --vmodule=""
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: W1229 03:56:00.819583 126295 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: W1229 03:56:00.820210 126295 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: I1229 03:56:00.820252 126295 controllermanager.go:151] Version: v1.13.1
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: I1229 03:56:00.822080 126295 secure_serving.go:116] Serving securely on 127.0.0.1:10252
Dec 29 03:56:00 ceph-01 kube-controller-manager[126295]: I1229 03:56:00.822954 126295 leaderelection.go:205] attempting to acquire leader lease kube-system/kube-controller-manager...
Dec 29 03:57:44 ceph-01 kube-controller-manager[126295]: I1229 03:57:44.753997 126295 log.go:172] http: TLS handshake error from 127.0.0.1:40918: tls: first record does not look like a TLS handshake
Dec 29 03:57:46 ceph-01 kube-controller-manager[126295]: I1229 03:57:46.558093 126295 log.go:172] http: TLS handshake error from 127.0.0.1:40948: tls: first record does not look like a TLS handshake
Dec 29 04:08:35 ceph-01 kube-controller-manager[126295]: I1229 04:08:35.872211 126295 log.go:172] http: TLS handshake error from 127.0.0.1:43564: tls: first record does not look like a TLS handshake
二.问题解决
这里我们推测是kube-controller-manager服务的Service文件的配置问题:
[root@ceph-01 system]# cat kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--port=0 \
--secure-port=10252 \
--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig \
--authentication-kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
--experimental-cluster-signing-duration=8760h \
--root-ca-file=/etc/kubernetes/cert/ca.pem \
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
--leader-elect=true \
--feature-gates=RotateKubeletServerCertificate=true \
--controllers=*,bootstrapsigner,tokencleaner \
--horizontal-pod-autoscaler-use-rest-clients=true \
--horizontal-pod-autoscaler-sync-period=10s \
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
--use-service-account-credentials=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
我们在service文件中加了--port=0
和--secure-port=10252
和--bind-address=127.0.0.1
这三行配置的功能是:
- --port=0:关闭监听 http /metrics 的请求,同时 --address 参数无效,--bind-address 参数有效
- --secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求
这里我们去掉这三行配置:
[root@ceph-01 system]# cat kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig \
--authentication-kubeconfig=/etc/kubernetes/cert/kube-controller-manager.kubeconfig \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \
--experimental-cluster-signing-duration=8760h \
--root-ca-file=/etc/kubernetes/cert/ca.pem \
--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \
--leader-elect=true \
--feature-gates=RotateKubeletServerCertificate=true \
--controllers=*,bootstrapsigner,tokencleaner \
--horizontal-pod-autoscaler-use-rest-clients=true \
--horizontal-pod-autoscaler-sync-period=10s \
--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \
--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \
--use-service-account-credentials=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
重启相关服务:
[root@ceph-01 system]# systemctl daemon-reload
[root@ceph-01 system]# systemctl restart kube-controller-manager
三.查看集群服务是否正常
[root@ceph-01 system]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
kubernetes二进制部署k8s-master集群controller-manager服务unhealthy问题的更多相关文章
- kubernetes kubeadm部署高可用集群
k8s kubeadm部署高可用集群 kubeadm是官方推出的部署工具,旨在降低kubernetes使用门槛与提高集群部署的便捷性. 同时越来越多的官方文档,围绕kubernetes容器化部署为环境 ...
- kubeadm 线上集群部署(二) K8S Master集群安装以及工作节点的部署
PS:所有机器主机名请提前设置好 在上一篇,ETCD集群我们已经搭建成功了,下面我们需要搭建master相关组件,apiverser需要与etcd通信并操作 1.配置证书 将etcd证书上传到mast ...
- 在kubernetes上部署zookeeper,kafka集群
本文采用网上镜像:mirrorgooglecontainers/kubernetes-zookeeper:1.0-3.4.10 准备共享存储:nfs,glusterfs,seaweed或其他,并在no ...
- 内网环境上部署k8s+docker集群:集群ftp的yum源配置
接触docker已经有一年了,想把做的时候的一些知识分享给大家. 因为公司机房是内网环境无法连接外网,所以这里所有的部署都是基于内网环境进行的. 首先,需要通过ftp服务制作本地的yum源,可以从ht ...
- kubernetes haproxy+keepalive实现master集群高可用
前言 master的HA,实际是apiserver的HA.Master的其他组件controller-manager.scheduler都是可以通过etcd做选举(--leader-elect),而A ...
- 二进制搭建kubernetes多master集群【三、配置k8s master及高可用】
前面两篇文章已经配置好了etcd和flannel的网络,现在开始配置k8s master集群. etcd集群配置参考:二进制搭建kubernetes多master集群[一.使用TLS证书搭建etcd集 ...
- k8s 组件介绍__单Master集群部署
参考链接:https://github.com/opsnull/follow-me-install-kubernetes-cluster kubernetes 概述 1.kubernetes 是什么 ...
- K8S部署Redis Cluster集群
kubernetes部署单节点redis: https://www.cnblogs.com/zisefeizhu/p/14282299.html Redis 介绍 • Redis代表REmote DI ...
- K8S部署Redis Cluster集群(三主三从模式) - 部署笔记
一.Redis 介绍 Redis代表REmote DIctionary Server是一种开源的内存中数据存储,通常用作数据库,缓存或消息代理.它可以存储和操作高级数据类型,例如列表,地图,集合和排序 ...
- docker swarm快速部署redis分布式集群
环境准备 四台虚拟机 192.168.2.38(管理节点) 192.168.2.81(工作节点) 192.168.2.100(工作节点) 192.168.2.102(工作节点) 时间同步 每台机器都执 ...
随机推荐
- Python+MapReduce实现矩阵相乘
算法原理 map阶段 在map阶段,需要做的是进行数据准备.把来自矩阵A的元素aij,标识成p条<key, value>的形式,key="i,k",(其中k=1,2,. ...
- DataTable2JSON 和 DataTable2Class 性能比较
DataTable 用 5000行和50000行数据做测试,得出转class效率貌似高一点点,不过优化并不大,还是 sql ,网络请求方面做优化比较显著. jobject 2019-03-07 06: ...
- 搭建openstack环境时出现的问题
penstack环境搭建程度(安装完keystone) 然后运行 openstack domain create --description "An Example Domain" ...
- 《Spring技术内幕》读书笔记
简介: 1.spring 与unix.window这些操作在计算机系统中起到的作用是类似的 2.两大核心模块:IOC\AOP 3.为应用开发提供了许多现成的系统组件:事务处理.Web MV.JDBC. ...
- CART回归树
决策树算法原理(ID3,C4.5) 决策树算法原理(CART分类树) 决策树的剪枝 CART回归树模型表达式: 其中,数据空间被划分为R1~Rm单元,每个单元有一个固定的输出值Cm.这样可以计算模型输 ...
- 雷林鹏分享:解决CI框架的Disallowed Key Characters错误提示
用CI框架时,有时候会遇到这么一个问题,打开网页,只显示 Disallowed Key Characters 错误提示.有人说 url 里有非法字符.但是确定 url 是纯英文的,问题还是出来了.但清 ...
- 提升 Hive Query 执行效率 - Hive LLAP
从 Hive 刚推出到现在,得益于社区对它的不断贡献,使得 Hive执行 query 效率显著提升.其中比较有代表性的功能如 Tez (将多个 job整合为一个DAG job)以及 CBO(Cost- ...
- WARING
每一道题都先手玩样例! 认真读一下每一档数据,仔细计算每一档可以拿的分数! 读完题目后,把所有能想到的思路写在纸上. 最优化题目考虑dp和贪心两种方法 字符串题目前缀考虑trie树,后缀考虑fail树 ...
- GIS开发 图形常见算法
摘录:OSGeo中国中心 http://www.osgeo.cn/post/ae457 计算机的出现使得很多原本十分繁琐的工作得以大幅度简化,但是也有一些在人们直观看来很容易的问题却需要拿出一套并不简 ...
- IDEA(添加类注释以及方法注释)
添加类注释: File---Setting----Editor----Code Style-----File and Code Templates--------Class #if (${PA ...