helm3.1安装及结合ceph rbd 部署harbor

[root@bs-k8s-ceph ~]# ceph -s
  cluster:
    id:     11880418-1a9a-4b55-a353-4b141e2199d8
    health: HEALTH_WARN
            Long heartbeat ping times on back interface seen, longest is 3884.944 msec
            Long heartbeat ping times on front interface seen, longest is 3888.368 msec
            application not enabled on 1 pool(s)
            clock skew detected on mon.bs-hk-hk02, mon.bs-k8s-ceph
 
  services:
    mon: 3 daemons, quorum bs-hk-hk01,bs-hk-hk02,bs-k8s-ceph
    mgr: bs-hk-hk01(active), standbys: bs-hk-hk02, bs-k8s-ceph
    osd: 6 osds: 6 up, 6 in
 
  data:
    pools:   3 pools, 320 pgs
    objects: 416  objects, 978 MiB
    usage:   8.6 GiB used, 105 GiB / 114 GiB avail
    pgs:     320 active+clean
[root@bs-k8s-ceph ~]# ceph osd pool application enable harbor rbd
enabled application 'rbd' on pool 'harbor'
[root@bs-k8s-ceph ~]# ceph -s
  cluster:
    id:     11880418-1a9a-4b55-a353-4b141e2199d8
    health: HEALTH_WARN
            Long heartbeat ping times on back interface seen, longest is 3870.142 msec
            Long heartbeat ping times on front interface seen, longest is 3873.410 msec
            clock skew detected on mon.bs-hk-hk02
 
  services:
    mon: 3 daemons, quorum bs-hk-hk01,bs-hk-hk02,bs-k8s-ceph
    mgr: bs-hk-hk01(active), standbys: bs-hk-hk02, bs-k8s-ceph
    osd: 6 osds: 6 up, 6 in
 
  data:
    pools:   3 pools, 320 pgs
    objects: 416  objects, 978 MiB
    usage:   8.6 GiB used, 105 GiB / 114 GiB avail
    pgs:     320 active+clean
 
# systemctl restart ceph.target     //让时间停一会
[root@bs-k8s-ceph ~]# ceph -s
  cluster:
    id:     11880418-1a9a-4b55-a353-4b141e2199d8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum bs-hk-hk01,bs-hk-hk02,bs-k8s-ceph
    mgr: bs-hk-hk01(active), standbys: bs-hk-hk02, bs-k8s-ceph
    osd: 6 osds: 6 up, 6 in
 
  data:
    pools:   3 pools, 320 pgs
    objects: 416  objects, 978 MiB
    usage:   8.6 GiB used, 105 GiB / 114 GiB avail
    pgs:     320 active+clean
[root@bs-k8s-master01 ~]# kubectl get nodes
The connection to the server 20.0.0.250:8443 was refused - did you specify the right host or port?
[root@bs-hk-hk01 ~]# systemctl start haproxy
[root@bs-k8s-master01 k8s]# kubectl get nodes
NAME              STATUS     ROLES    AGE     VERSION
bs-k8s-master01   Ready      master   7d10h   v1.17.2
bs-k8s-master02   Ready      master   7d10h   v1.17.2
bs-k8s-master03   Ready      master   7d10h   v1.17.2
bs-k8s-node01     Ready      <none>   7d10h   v1.17.2
bs-k8s-node02     Ready      <none>   7d10h   v1.17.2
bs-k8s-node03     NotReady   <none>   7d9h    v1.17.2    //为了节省cpu而关掉
https://github.com/helm/helm/releases
[root@bs-k8s-master01 helm3]# pwd
/data/k8s/helm3
[root@bs-k8s-master01 helm3]# ll
总用量 11980
-rw-r--r-- 1 root root 12267464 2月  17 2020 helm-v3.1.0-linux-amd64.tar.gz
[root@bs-k8s-master01 helm3]# cp linux-amd64/helm /usr/local/bin/helm
[root@bs-k8s-master01 helm3]# helm  version
version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}
[root@bs-k8s-master01 helm3]# helm  --help
The Kubernetes package manager
 
Common actions for Helm:
 
- helm search:    search for charts
- helm pull:      download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts
 
Environment variables:
 
+------------------+-----------------------------------------------------------------------------+
| Name             | Description                                                                 |
+------------------+-----------------------------------------------------------------------------+
| $XDG_CACHE_HOME  | set an alternative location for storing cached files.                       |
| $XDG_CONFIG_HOME | set an alternative location for storing Helm configuration.                 |
| $XDG_DATA_HOME   | set an alternative location for storing Helm data.                          |
| $HELM_DRIVER     | set the backend storage driver. Values are: configmap, secret, memory       |
| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.                  |
| $KUBECONFIG      | set an alternative Kubernetes configuration file (default "~/.kube/config") |
+------------------+-----------------------------------------------------------------------------+
 
Helm stores configuration based on the XDG base directory specification, so
 
- cached files are stored in $XDG_CACHE_HOME/helm
- configuration is stored in $XDG_CONFIG_HOME/helm
- data is stored in $XDG_DATA_HOME/helm
 
By default, the default directories depend on the Operating System. The defaults are listed below:
 
+------------------+---------------------------+--------------------------------+-------------------------+
| Operating System | Cache Path                | Configuration Path             | Data Path               |
+------------------+---------------------------+--------------------------------+-------------------------+
| Linux            | $HOME/.cache/helm         | $HOME/.config/helm             | $HOME/.local/share/helm |
| macOS            | $HOME/Library/Caches/helm | $HOME/Library/Preferences/helm | $HOME/Library/helm      |
| Windows          | %TEMP%\helm               | %APPDATA%\helm                 | %APPDATA%\helm          |
+------------------+---------------------------+--------------------------------+-------------------------+
 
Usage:
  helm [command]
 
Available Commands:
  completion  Generate autocompletions script for the specified shell (bash or zsh)
  create      create a new chart with the given name
  dependency  manage a chart's dependencies
  env         Helm client environment information
  get         download extended information of a named release
  help        Help about any command
  history     fetch release history
  install     install a chart
  lint        examines a chart for possible issues
  list        list releases
  package     package a chart directory into a chart archive
  plugin      install, list, or uninstall Helm plugins
  pull        download a chart from a repository and (optionally) unpack it in local directory
  repo        add, list, remove, update, and index chart repositories
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  show        show information of a chart
  status      displays the status of the named release
  template    locally render templates
  test        run tests for a release
  uninstall   uninstall a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client version information
 
Flags:
      --add-dir-header                   If true, adds the file directory to the header
      --alsologtostderr                  log to standard error as well as files
      --debug                            enable verbose output
  -h, --help                             help for helm
      --kube-context string              name of the kubeconfig context to use
      --kubeconfig string                path to the kubeconfig file
      --log-backtrace-at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log-dir string                   If non-empty, write log files in this directory
      --log-file string                  If non-empty, use this log file
      --log-file-max-size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
  -n, --namespace string                 namespace scope for this request
      --registry-config string           path to the registry config file (default "/root/.config/helm/registry.json")
      --repository-cache string          path to the file containing cached repository indexes (default "/root/.cache/helm/repository")
      --repository-config string         path to the file containing repository names and URLs (default "/root/.config/helm/repositories.yaml")
      --skip-headers                     If true, avoid header prefixes in the log messages
      --skip-log-headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
  -v, --v Level                          number for the log level verbosity
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
 
Use "helm [command] --help" for more information about a command.
[root@bs-k8s-master01 helm3]# source <(helm completion bash)
[root@bs-k8s-master01 helm3]# echo "source <(helm completion bash)" >> ~/.bashrc
[root@bs-k8s-master01 rbd]# helm repo add aliyun  https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
"aliyun" has been added to your repositories
[root@bs-k8s-master01 helm3]#  helm repo add stable  https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
"stable" has been added to your repositories
[root@bs-k8s-master01 helm3]# helm repo add google  https://kubernetes-charts.storage.googleapis.com
"google" has been added to your repositories
[root@bs-k8s-master01 helm3]# helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
[root@bs-k8s-master01 helm3]# helm repo list
NAME        URL
aliyun      https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
stable      https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
google      https://kubernetes-charts.storage.googleapis.com
jetstack    https://charts.jetstack.io 
 
[root@bs-k8s-master01 helm3]# curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6794  100  6794    0     0    434      0  0:00:15  0:00:15 --:--:--   761
 
[root@bs-k8s-master01 helm3]# ./get_helm.sh
Downloading https://get.helm.sh/helm-v3.1.0-linux-amd64.tar.gz
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
[root@bs-k8s-master01 helm3]# helm version
version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}
 
[root@bs-k8s-master01 helm3]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aliyun" chart repository
Update Complete. ⎈ Happy Helming!⎈
[root@bs-k8s-master01 helm3]# helm search repo nginx
NAME                           CHART VERSION    APP VERSION    DESCRIPTION
aliyun/nginx-ingress           0.9.5            0.10.2         An nginx Ingress controller that uses ConfigMap...
aliyun/nginx-lego              0.3.1                           Chart for nginx-ingress-controller and kube-lego
google/nginx-ingress           1.30.3           0.28.0         An nginx Ingress controller that uses ConfigMap...
google/nginx-ldapauth-proxy    0.1.3            1.13.5         nginx proxy with ldapauth
google/nginx-lego              0.3.1                           Chart for nginx-ingress-controller and kube-lego
stable/nginx-ingress           0.9.5            0.10.2         An nginx Ingress controller that uses ConfigMap...
stable/nginx-lego              0.3.1                           Chart for nginx-ingress-controller and kube-lego
aliyun/gcloud-endpoints        0.1.0                           Develop, deploy, protect and monitor your APIs ...
google/gcloud-endpoints        0.1.2            1              DEPRECATED Develop, deploy, protect and monitor...
stable/gcloud-endpoints        0.1.0                           Develop, deploy, protect and monitor your APIs ...
[root@bs-k8s-master01 helm3]# helm repo remove stable
"stable" has been removed from your repositories
[root@bs-k8s-master01 helm3]# helm repo remove google
"google" has been removed from your repositories
[root@bs-k8s-master01 helm3]# helm repo remove jetstack
"jetstack" has been removed from your repositories
[root@bs-k8s-master01 helm3]# helm repo list
NAME      URL
aliyun    https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
[root@bs-k8s-master01 helm3]# helm repo add harbor https://helm.goharbor.io
"harbor" has been added to your repositories
[root@bs-k8s-master01 harbor]# pwd
/data/k8s/harbor
[root@bs-k8s-master01 harbor]# ll
总用量 48
-rw-r--r-- 1 root root   701 2月  16 19:26 ceph-harbor-pvc.yaml
-rw-r--r-- 1 root root   863 2月  16 19:18 ceph-harbor-secret.yaml
-rw-r--r-- 1 root root   994 2月  16 19:21 ceph-harbor-storageclass.yaml
-rw-r--r-- 1 root root 35504 2月  17 13:07 harbor-1.3.0.tgz
drwxr-xr-x 2 root root   134 2月  16 19:13 rbd
[root@bs-k8s-master01 harbor]# tar xf harbor-1.3.0.tgz
[root@bs-k8s-master01 harbor]# cd harbor/
[root@bs-k8s-master01 harbor]# ls
cert  Chart.yaml  conf  LICENSE  README.md  templates  values.yaml
[root@bs-k8s-master01 harbor]# cp values.yaml{,.bak}
[root@bs-k8s-master01 harbor]# diff values.yaml{,.bak}
26c26
<     commonName: "zisefeizhu.harbor.org"
---
>     commonName: ""
29c29
<       core: zisefeizhu.harbor.org
---
>       core: core.harbor.domain
101c101
< externalURL: https://zisefeizhu.harbor.org
---
> externalURL: https://core.harbor.domain
123c123
<       storageClass: "ceph-harbor"
---
>       storageClass: ""
129c129
<       storageClass: "ceph-harbor"
---
>       storageClass: ""
135c135
<       storageClass: "ceph-harbor"
---
>       storageClass: ""
143c143
<       storageClass: "ceph-harbor"
---
>       storageClass: ""
151c151
<       storageClass: "ceph-harbor"
---
>       storageClass: ""
253c253
< harborAdminPassword: "zisefeizhu"
---
> harborAdminPassword: "Harbor12345"
[root@bs-k8s-master01 nginx-ingress]# pwd
/data/k8s/nginx-ingress
[root@bs-k8s-master01 k8s]# cd nginx-ingress/
[root@bs-k8s-master01 nginx-ingress]# helm pull aliyun/nginx-ingress
[root@bs-k8s-master01 nginx-ingress]# tar xf nginx-ingress-0.9.5.tgz
[root@bs-k8s-master01 nginx-ingress]# pwd
/data/k8s/nginx-ingress/nginx-ingress
[root@bs-k8s-master01 nginx-ingress]# pwd
/data/k8s/nginx-ingress
[root@bs-k8s-master01 nginx-ingress]# helm install nginx-ingress nginx-ingress
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
[root@bs-k8s-master01 nginx-ingress]# grep -irl "extensions/v1beta1" nginx-ingress | grep deploy
nginx-ingress/templates/controller-deployment.yaml
nginx-ingress/templates/default-backend-deployment.yaml
[root@bs-k8s-master01 nginx-ingress]# grep -irl "extensions/v1beta1" nginx-ingress | grep deploy | xargs sed -i 's#extensions/v1beta1#apps/v1#g'
[root@bs-k8s-master01 nginx-ingress]# helm install nginx-ingress nginx-ingress
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec
由于k8s1.16版本升级，需要Deployment.spec中加selector,所以愉快地加上就行了。
 
[root@bs-k8s-master01 nginx]# helm install nginx-ingress nginx-ingress
NAME: nginx-ingress
LAST DEPLOYED: Mon Feb 17 14:12:27 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The nginx-ingress controller has been installed.
Get the application URL by running these commands:
  export HTTP_NODE_PORT=$(kubectl --namespace default get services -o jsonpath="{.spec.ports[0].nodePort}" nginx-ingress-controller)
  export HTTPS_NODE_PORT=$(kubectl --namespace default get services -o jsonpath="{.spec.ports[1].nodePort}" nginx-ingress-controller)
  export NODE_IP=$(kubectl --namespace default get nodes -o jsonpath="{.items[0].status.addresses[1].address}")
 
  echo "Visit http://$NODE_IP:$HTTP_NODE_PORT to access your application via HTTP."
  echo "Visit https://$NODE_IP:$HTTPS_NODE_PORT to access your application via HTTPS."
 
An example Ingress that makes use of the controller:
 
  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls
 
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
 
  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
[root@bs-k8s-master01 nginx]# kubectl get pods
NAME                                             READY   STATUS             RESTARTS   AGE
nginx-ingress-controller-8fbb5974-l7dsx          1/1     Running            0          115s
nginx-ingress-default-backend-744fdc79c4-xcvqp   1/1     Running            0          115s
[root@bs-k8s-master01 nginx]# pwd
/data/k8s/nginx
[root@bs-k8s-master01 nginx]# ll
总用量 12
drwxr-xr-x 3 root root   119 2月  17 13:32 nginx-ingress
-rw-r--r-- 1 root root 10830 2月  17 13:25 nginx-ingress-0.9.5.tgz
[root@bs-k8s-master01 harbor]# helm install harbor -n harbor harbor
NAME: harbor
LAST DEPLOYED: Mon Feb 17 14:16:05 2020
NAMESPACE: harbor
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://zisefeizhu.harbor.org.
For more details, please visit https://github.com/goharbor/harbor.
[root@bs-k8s-master01 harbor]# kubectl get pvc -n harbor
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-harbor-harbor-redis-0               Bound    pvc-4b2c0362-aca9-4fc2-b3e8-5fed5bf46b43   1Gi        RWO            ceph-harbor    66s
database-data-harbor-harbor-database-0   Bound    pvc-ce201f8c-0909-4f69-8eb9-aeaeb542de98   1Gi        RWO            ceph-harbor    66s
harbor-harbor-chartmuseum                Bound    pvc-1ec866fa-413a-463d-bb04-a0376577ae69   5Gi        RWO            ceph-harbor    6m38s
harbor-harbor-jobservice                 Bound    pvc-03dd5393-fad1-471b-8384-b0a5f5403d90   1Gi        RWO            ceph-harbor    6m38s
harbor-harbor-registry                   Bound    pvc-b7268d13-e92a-4ab3-846a-26d14672e56c   5Gi        RWO            ceph-harbor    6m38s
[root@bs-k8s-master01 harbor]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                           STORAGECLASS   REASON   AGE
pvc-03dd5393-fad1-471b-8384-b0a5f5403d90   1Gi        RWO            Retain           Bound    harbor/harbor-harbor-jobservice                 ceph-harbor             <invalid>
pvc-1ec866fa-413a-463d-bb04-a0376577ae69   5Gi        RWO            Retain           Bound    harbor/harbor-harbor-chartmuseum                ceph-harbor             <invalid>
pvc-494a130d-018c-4be3-9b31-e951cc4367a5   20Gi       RWO            Retain           Bound    default/wp-pv-claim                             ceph-rbd                27h
pvc-4b2c0362-aca9-4fc2-b3e8-5fed5bf46b43   1Gi        RWO            Retain           Bound    harbor/data-harbor-harbor-redis-0               ceph-harbor             <invalid>
pvc-8ffa3182-a2f6-47d9-a71d-ff8e8b379a16   1Gi        RWO            Retain           Bound    default/ceph-pvc                                ceph-rbd                29h
pvc-ac7d3a09-123e-4614-886c-cded8822a078   20Gi       RWO            Retain           Bound    default/mysql-pv-claim                          ceph-rbd                27h
pvc-b7268d13-e92a-4ab3-846a-26d14672e56c   5Gi        RWO            Retain           Bound    harbor/harbor-harbor-registry                   ceph-harbor             <invalid>
pvc-ce201f8c-0909-4f69-8eb9-aeaeb542de98   1Gi        RWO            Retain           Bound    harbor/database-data-harbor-harbor-database-0   ceph-harbor             <invalid>
[root@bs-k8s-master01 harbor]# kubectl get pods -n harbor -o wide
NAME                                          READY   STATUS             RESTARTS   AGE    IP              NODE            NOMINATED NODE   READINESS GATES
harbor-harbor-chartmuseum-dcc6f779f-68tvn     1/1     Running            0          32m    10.209.208.21   bs-k8s-node03   <none>           <none>
harbor-harbor-clair-69789f6695-5zrf8          1/2     CrashLoopBackOff   9          32m    10.209.145.26   bs-k8s-node02   <none>           <none>
harbor-harbor-core-5675f84d5f-ddhj2           0/1     CrashLoopBackOff   8          32m    10.209.145.27   bs-k8s-node02   <none>           <none>
harbor-harbor-database-0                      1/1     Running            1          32m    10.209.46.93    bs-k8s-node01   <none>           <none>
harbor-harbor-jobservice-74f469588d-m6w64     0/1     Running            3          32m    10.209.46.91    bs-k8s-node01   <none>           <none>
harbor-harbor-notary-server-fcbcfdf9c-zgjk8   0/1     CrashLoopBackOff   9          32m    10.209.208.19   bs-k8s-node03   <none>           <none>
harbor-harbor-notary-signer-9789894bd-8p67d   0/1     CrashLoopBackOff   9          32m    10.209.208.20   bs-k8s-node03   <none>           <none>
harbor-harbor-portal-56456988bb-6cb9j         1/1     Running            0          32m    10.209.208.18   bs-k8s-node03   <none>           <none>
harbor-harbor-redis-0                         1/1     Running            0          32m    10.209.46.92    bs-k8s-node01   <none>           <none>
harbor-harbor-registry-6946847b6f-qdgfp       2/2     Running            0          32m    10.209.145.28   bs-k8s-node02   <none>           <none>
rbd-provisioner-75b85f85bd-d4b8d              1/1     Running            0          136m   10.209.145.25   bs-k8s-node02   <none>           <none>
 
下面的操作相比就不需要记载了
注意点：
    1.  pvc创建完毕后不要执行。切记。
    2.  本地hosts里的Ip是 nginx-controller的节点ip