ceph kubernetes中使用
1.在管理节点上，进入刚创建的放置配置文件的目录，用 ceph-deploy 执行如下步骤
mkdir /opt/cluster-ceph cd /opt/cluster-ceph ceph-deploy new master1 master2 master3
2.添加epel源
yum install -y yum-utils && yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && yum install --nogpgcheck -y epel-release && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && rm -f /etc/yum.repos.d/dl.fedoraproject.org*
直接进行ceph的安装，会报如下的错误：
--> Finished Dependency Resolution
Error: Package: :ceph-common-10.2.-.el7.x86_64 (ceph)
Requires: libbabeltrace-ctf.so.()(64bit)
Error: Package: :ceph-osd-10.2.-.el7.x86_64 (ceph)
Requires: libleveldb.so.()(64bit)
Error: Package: :ceph-mon-10.2.-.el7.x86_64 (ceph)
Requires: libleveldb.so.()(64bit)
Error: Package: :librbd1-10.2.-.el7.x86_64 (ceph)
Requires: liblttng-ust.so.()(64bit)
Error: Package: :ceph-base-10.2.-.el7.x86_64 (ceph)
Requires: liblttng-ust.so.()(64bit)
Error: Package: :librgw2-10.2.-.el7.x86_64 (ceph)
Requires: libfcgi.so.()(64bit)
Error: Package: :ceph-common-10.2.-.el7.x86_64 (ceph)
Requires: libbabeltrace.so.()(64bit)
Error: Package: :librados2-10.2.-.el7.x86_64 (ceph)
Requires: liblttng-ust.so.()(64bit)
2. 安装 Ceph
[root@localhost ~]# yum install --downloadonly --downloaddir=/tmp/ceph ceph 在每台主机上安装ceph
[root@localhost ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph/*.rpm
配置初始 monitor(s)、并收集所有密钥
# 请务必在 ceph-cluster 目录下 [root@admin ceph-cluster]# ceph-deploy mon create-initial
初始化 ceph.osd 节点
创建存储空间
[root@osd1 ~]# mkdir -p /data/ceph-osd
[root@osd1 ~]# chown ceph.ceph /data/ceph-osd/ -R [root@osd2 ~]# mkdir -p /data/ceph-osd
[root@osd2 ~]# chown ceph.ceph /data/ceph-osd/ -R
[root@osd3 ~]# mkdir -p /data/ceph-osd
[root@osd3 ~]# chown ceph.ceph /data/ceph-osd/ -R
[root@osd4 ~]# mkdir -p /data/ceph-osd
[root@osd4 ~]# chown ceph.ceph /data/ceph-osd/ -R
创建OSD：
[root@admin ceph-cluster]# ceph-deploy osd prepare node1:/data/ceph-osd node2:/data/ceph-osd node3:/data/ceph-osd node4:/data/ceph-osd 激活 OSD
[root@admin ceph-cluster]# ceph-deploy osd activate node1:/data/ceph-osd node2:/data/ceph-osd node3:/data/ceph-osd node4:/data/ceph-osd
用 ceph-deploy 把配置文件和 admin 密钥拷贝到管理节点和 Ceph 节点，这样你每次执行 Ceph 命令行时就无需指定 monitor 地址和 ceph.client.admin.keyring 了 [root@admin ceph-cluster]# ceph-deploy admin master1 master2 master3 node1 node2 node3 node4 确保你对 ceph.client.admin.keyring 有正确的操作权限。 chmod +r /etc/ceph/ceph.client.admin.keyring （所有机器） 如果配置文件更改，需要同步配置文件到所有节点 [root@admin ceph-cluster]# ceph-deploy --overwrite-conf admin master1 master2 master3 node1 node2 node3 node4
官网的zookeeper yaml，去掉了亲和性
apiVersion: v1
kind: Service
metadata:
namespace: testsubject
name: zk-hs
labels:
app: zk
spec:
ports:
- port:
name: server
- port:
name: leader-election
clusterIP: None
selector:
app: zk
---
apiVersion: v1
kind: Service
metadata:
namespace: testsubject
name: zk-cs
labels:
app: zk
spec:
ports:
- port:
name: client
selector:
app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
namespace: testsubject
name: zk-pdb
spec:
selector:
matchLabels:
app: zk
maxUnavailable:
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
namespace: testsubject
name: zk
spec:
selector:
matchLabels:
app: zk
serviceName: zk-hs
replicas:
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: zk
spec:
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: "192.168.200.10/senyint/kubernetes-zookeeper:1.0-3.4.10"
resources:
requests:
memory: "4Gi"
cpu: ""
ports:
- containerPort:
name: client
- containerPort:
name: server
- containerPort:
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers= \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port= \
--election_port= \
--server_port= \
--tick_time= \
--init_limit= \
--sync_limit= \
--heap=512M \
--max_client_cnxns= \
--snap_retain_count= \
--purge_interval= \
--max_session_timeout= \
--min_session_timeout= \
--log_level=OFF"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds:
timeoutSeconds:
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds:
timeoutSeconds:
volumeMounts:
- name: datazk
mountPath: /var/lib/zookeeper
volumeClaimTemplates:
- metadata:
name: datazk
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: ceph-rbd-database
resources:
requests:
storage: 20Gi
ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-admin-testsubject
namespace: testsubject
type: "kubernetes.io/rbd"
data:
key: QVFERkcvQmF5ckFkSnhBQVVkM2VCdC82K3dOTnZIM3V0ZHpnTnc9PQo=
rbd-storage-data-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd-database
namespace: testsubject
provisioner: kubernetes.io/rbd
parameters:
monitors: 192.168.200.11:,192.168.200.12:,192.168.200.13:
adminId: admin
adminSecretName: ceph-secret-admin-testsubject
adminSecretNamespace: "testsubject"
pool: fengjian
userId: admin
userSecretName: ceph-secret-admin-testsubject
imageFormat: ""
imageFeatures: "layering"
不建立pv， 直接 使用storageclass，然后建立pvc， deployment 指定 claimName
ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret-admin
type: "kubernetes.io/rbd"
data:
key: QVFERkcvQmF5ckFkSnhBQVVkM2VCdC82K3dOTnZIM3V0ZHpnTnc9PQo=
rbd-storage-data-class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd-provisioner
provisioner: kubernetes.io/rbd
parameters:
monitors: 192.168.200.11:6789,192.168.200.12:6789,192.168.200.13:6789
adminId: admin
adminSecretName: ceph-secret-admin
adminSecretNamespace: default
pool: fengjian
userId: admin
userSecretName: ceph-secret-admin
imageFormat: "2"
imageFeatures: "layering"
redis.yaml
apiVersion: apps/v1 # for versions before 1.9. use apps/v1beta2
kind: Deployment
metadata:
name: redis-master
spec:
selector:
matchLabels:
app: redis
role: master
tier: backend
replicas:
template:
metadata:
labels:
app: redis
role: master
tier: backend
spec:
containers:
- name: master
image: 192.168.200.10/redis/redis:master
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort:
volumeMounts:
- name: datadir
mountPath: /data
volumes:
- name: datadir
persistentVolumeClaim:
claimName: redis-master-rbd-pvc --- apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
role: master
tier: backend
spec:
ports:
- port:
targetPort:
selector:
app: redis
role: master
tier: backend
kafka配置文件
参考 ： https://kow3ns.github.io/kubernetes-kafka/manifests/
[root@master1 ceph_rbd]# cat kafka.yaml
apiVersion: v1
kind: Service
metadata:
name: kafka-hs
labels:
app: kafka
spec:
ports:
- port:
name: server
clusterIP: None
selector:
app: kafka
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: kafka-pdb
spec:
selector:
matchLabels:
app: kafka
maxUnavailable:
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: kafka
spec:
serviceName: kafka-hs
replicas:
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: kafka
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- kafka
topologyKey: "kubernetes.io/hostname"
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight:
podAffinityTerm:
labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
terminationGracePeriodSeconds:
containers:
- name: k8skafka
imagePullPolicy: Always
image: 192.168.200.10/source/kubernetes-kafka:1.0-10.2.
resources:
requests:
memory: "12Gi"
cpu:
ports:
- containerPort:
name: server
command:
- sh
- -c
- "exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \
--override listeners=PLAINTEXT://:9093 \
--override zookeeper.connect=zk-cs.default.svc.cluster.local: \
--override log.dir=/var/lib/kafka \
--override auto.create.topics.enable=true \
--override auto.leader.rebalance.enable=true \
--override background.threads= \
--override compression.type=producer \
--override delete.topic.enable=false \
--override leader.imbalance.check.interval.seconds= \
--override leader.imbalance.per.broker.percentage= \
--override log.flush.interval.messages= \
--override log.flush.offset.checkpoint.interval.ms= \
--override log.flush.scheduler.interval.ms= \
--override log.retention.bytes=- \
--override log.retention.hours= \
--override log.roll.hours= \
--override log.roll.jitter.hours= \
--override log.segment.bytes= \
--override log.segment.delete.delay.ms= \
--override message.max.bytes= \
--override min.insync.replicas= \
--override num.io.threads= \
--override num.network.threads= \
--override num.recovery.threads.per.data.dir= \
--override num.replica.fetchers= \
--override offset.metadata.max.bytes= \
--override offsets.commit.required.acks=- \
--override offsets.commit.timeout.ms= \
--override offsets.load.buffer.size= \
--override offsets.retention.check.interval.ms= \
--override offsets.retention.minutes= \
--override offsets.topic.compression.codec= \
--override offsets.topic.num.partitions= \
--override offsets.topic.replication.factor= \
--override offsets.topic.segment.bytes= \
--override queued.max.requests= \
--override quota.consumer.default= \
--override quota.producer.default= \
--override replica.fetch.min.bytes= \
--override replica.fetch.wait.max.ms= \
--override replica.high.watermark.checkpoint.interval.ms= \
--override replica.lag.time.max.ms= \
--override replica.socket.receive.buffer.bytes= \
--override replica.socket.timeout.ms= \
--override request.timeout.ms= \
--override socket.receive.buffer.bytes= \
--override socket.request.max.bytes= \
--override socket.send.buffer.bytes= \
--override unclean.leader.election.enable=true \
--override zookeeper.session.timeout.ms= \
--override zookeeper.set.acl=false \
--override broker.id.generation.enable=true \
--override connections.max.idle.ms= \
--override controlled.shutdown.enable=true \
--override controlled.shutdown.max.retries= \
--override controlled.shutdown.retry.backoff.ms= \
--override controller.socket.timeout.ms= \
--override default.replication.factor= \
--override fetch.purgatory.purge.interval.requests= \
--override group.max.session.timeout.ms= \
--override group.min.session.timeout.ms= \
--override inter.broker.protocol.version=0.10.-IV0 \
--override log.cleaner.backoff.ms= \
--override log.cleaner.dedupe.buffer.size= \
--override log.cleaner.delete.retention.ms= \
--override log.cleaner.enable=true \
--override log.cleaner.io.buffer.load.factor=0.9 \
--override log.cleaner.io.buffer.size= \
--override log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 \
--override log.cleaner.min.cleanable.ratio=0.5 \
--override log.cleaner.min.compaction.lag.ms= \
--override log.cleaner.threads= \
--override log.cleanup.policy=delete \
--override log.index.interval.bytes= \
--override log.index.size.max.bytes= \
--override log.message.timestamp.difference.max.ms= \
--override log.message.timestamp.type=CreateTime \
--override log.preallocate=false \
--override log.retention.check.interval.ms= \
--override max.connections.per.ip= \
--override num.partitions= \
--override producer.purgatory.purge.interval.requests= \
--override replica.fetch.backoff.ms= \
--override replica.fetch.max.bytes= \
--override replica.fetch.response.max.bytes= \
--override reserved.broker.max.id= "
env:
- name: KAFKA_HEAP_OPTS
value : "-Xmx2G -Xms2G"
- name: KAFKA_OPTS
value: "-Dlogging.level=INFO"
volumeMounts:
- name: datadir
mountPath: /var/lib/kafka
readinessProbe:
tcpSocket:
port:
initialDelaySeconds:
periodSeconds:
securityContext:
runAsUser:
fsGroup:
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.beta.kubernetes.io/storage-class: "ceph-rbd-provisioner"
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100Gi
zookeeper.yaml
https://github.com/kow3ns apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
spec:
ports:
- port:
name: server
- port:
name: leader-election
clusterIP: None
selector:
app: zk
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
spec:
ports:
- port:
name: client
selector:
app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
selector:
matchLabels:
app: zk
maxUnavailable:
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: zk
spec:
serviceName: zk-hs
replicas:
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: zk
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: "gcr.io/google_containers/kubernetes-zookeeper:1.0-3.4.10"
resources:
requests:
memory: "4Gi"
cpu: ""
ports:
- containerPort:
name: client
- containerPort:
name: server
- containerPort:
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers= \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port= \
--election_port= \
--server_port= \
--tick_time= \
--init_limit= \
--sync_limit= \
--heap=3G \
--max_client_cnxns= \
--snap_retain_count= \
--purge_interval= \
--max_session_timeout= \
--min_session_timeout= \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds:
timeoutSeconds:
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds:
timeoutSeconds:
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
securityContext:
runAsUser:
fsGroup:
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 250Gi
ceph kubernetes中使用的更多相关文章
- 关于 Kubernetes 中的 Volume 与 GlusterFS 分布式存储
容器中持久化的文件生命周期是短暂的,如果容器中程序崩溃宕机,kubelet 就会重新启动,容器中的文件将会丢失,所以对于有状态的应用容器中持久化存储是至关重要的一个环节:另外很多时候一个 Pod 中可 ...
- Kubernetes 中的核心组件与基本对象概述
Kubernetes 是 Google 基于 Borg 开源的容器编排调度,用于管理容器集群自动化部署.扩容以及运维的开源平台.作为云原生计算基金会 CNCF(Cloud Native Computi ...
- Kubernetes 中的pv和pvc
原文地址:http://www.cnblogs.com/leidaxia/p/6485646.html 持久卷 PersistentVolumes 本文描述了 Kubernetes 中的 Persis ...
- Kubernetes中的Volume介绍
Kubernetes中支持的所有磁盘挂载卷简介发表于 2018年1月26日 Weihai Feb 10,2016 7400 字 | 阅读需要 15 分钟 容器磁盘上的文件的生命周期是短暂的,这就使得在 ...
- Kubernetes中分布式存储Rook-Ceph部署快速演练
最近在项目中有涉及到Kubernetes的分布式存储部分的内容,也抽空多了解了一些.项目主要基于Rook-Ceph运行,考虑到Rook-Ceph部署也不那么简单,官方文档的步骤起点也不算低,因此,在整 ...
- Kubernetes中分布式存储Rook-Ceph的使用：一个ASP.NET Core MVC的案例
在<Kubernetes中分布式存储Rook-Ceph部署快速演练>文章中,我快速介绍了Kubernetes中分布式存储Rook-Ceph的部署过程,这里介绍如何在部署于Kubernete ...
- docker对cpu使用及在kubernetes中的应用
docker对CPU的使用 docker对于CPU的可配置的主要几个参数如下: --cpu-shares CPU shares (relative weight) --cpu-period Limit ...
- 【转】干货，Kubernetes中的Source Ip机制。
准备工作 你必须拥有一个正常工作的 Kubernetes 1.5 集群,用来运行本文中的示例.该示例使用一个简单的 nginx webserver 回送它接收到的请求的 HTTP 头中的源 IP 地址 ...
- kubernetes中的Pause容器如何理解？
前几篇文章都是讲的Kubernetes集群和相关组件的部署,但是部署只是入门的第一步,得理解其中的一些知识才行.今天给大家分享下Kubernets的pause容器的作用. Pause容器 全称infr ...
随机推荐
- 【游记】CCHO TY国初划水记
没想到第一篇游记竟然是化学国初(其实是上次SXACM时候懒得写 DAY0 一下午做了5个小时的校车,服务区水真贵 肝了4个小时模拟题,颠到崩溃. 下榻在距离山大附不远的一个酒店,高三人好多哇,我们年级 ...
- Mac破解Sublime Text 3 3176
## Sublime Text 3 Serial key build is 3176 > * Added these lines into /etc/hosts 127.0.0.1 www.su ...
- 【ZeroMQ】2、高性能的通讯库-zeroMQ
首先,让我来介绍一下什么是ZMQ(全称:ZeroMQ): 官方: “ZMQ(以下ZeroMQ简称ZMQ)是一个简单好用的传输层,像框架一样的一个socket library,他使得Socket编程更加 ...
- java中递归实现复制多级文件夹
常见的流的用法 递归实现复制多级文件夹 FileInputStream & FileOutputStream String content = null;//用来储存解码后的byte数组 in ...
- 4+1视图与UML对应关系
4+1视图模型概况 Kruchten 提出了一个"4+1"视图模型,从5个不同的视角包括包括逻辑试图.进程视图.物理视图.开发视图.场景视图来描述软件体系结构.每一个视图只 ...
- java.lang.ExceptionInInitializerError异常
今天在开发的过程中,遇到java.lang.ExceptionInInitializerError异常,百度查了一下,顺便学习学习,做个笔记 静态初始化程序中发生意外异常的信号,抛出Exception ...
- Linux 安装mysql,mariadb,mysql主从同步
myariadb安装 centos7 mariadb的学习 在企业里面,多半不会使用阿里云的mariadb版本,因为版本太低,安全性太低,公司会配置myariadb官方的yum仓库 1.手动创建mar ...
- 分布式Session一致性解决方案有哪些？
1.使用cookie代替session(不安全,不推荐使用) 2.使用数据库存储session(效率低,不推荐使用) 3.使用nginx反向代理ip绑定方法,同一个ip只能在同一台服务器上进行访问(不 ...
- @PathVariable与@RequestParam的区别
@PathVariable与@RequestParam的区别首先呢这二个注解都是接收参数使用的,下面来看一下它们的区别.@PathVariable注解@RequestMapping(value ={“ ...
- windows下安装composer方法
composer是一个新崛起的PHP的依赖管理工具.官方安装方法见: 此处假定: (1)php安装目录为C:/php5.4 (2)php 安装目录已经加入PATH环境变量(这样就可以在命令行直接输入“ ...