Dashboard:https://github.com/kubernetes/dashboard

一、Dashboard部署

由于需要用到k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0,这里有2种方式进行pull 镜像。docker search该镜像名称,直接pull,再重新进行tag;另外一种方式是通过谷歌容器镜像拉取。

[root@k8s-node01 ~]# docker pull siriuszg/kubernetes-dashboard-amd64
[root@k8s-node01 ~]# docker tag siriuszg/kubernetes-dashboard-amd64:latest k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0 或者是
[root@k8s-node01 ~]# docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:v1.10.0

再看其部署的过程:

[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created [root@k8s-master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-nmcmz / Running 54d
coredns-78fcdf6894-p5pfm / Running 54d
etcd-k8s-master / Running 54d
kube-apiserver-k8s-master / Running 54d
kube-controller-manager-k8s-master / Running 54d
kube-flannel-ds-n5c86 / Running 54d
kube-flannel-ds-nrcw2 / Running 52d
kube-flannel-ds-pgpr7 / Running 54d
kube-proxy-glzth / Running 52d
kube-proxy-rxlt7 / Running 54d
kube-proxy-vxckf / Running 54d
kube-scheduler-k8s-master / Running 54d
kubernetes-dashboard-767dc7d4d-n4clq 1/1 Running 0 3s

[root@k8s-master ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> /UDP,/TCP 54d
kubernetes-dashboard ClusterIP 10.105.204.4 <none> /TCP 30m [root@k8s-master ~]# kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kube-system #以打补丁方式修改dasboard的访问方式
service/kubernetes-dashboard patched
[root@k8s-master ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> /UDP,/TCP 54d
kubernetes-dashboard NodePort 10.105.204.4 <none> :/TCP 31m

浏览器访问:https://192.168.56.12:32645,如图:这里需要注意的是谷歌浏览器会禁止不安全证书访问,建议使用火狐浏览器,并且需要在高级选项中添加信任

在k8s中 dashboard可以有两种访问方式:kubeconfig(HTTPS)和token(http)

1、token认证

(1)创建dashboard专用证书

[root@k8s-master pki]# (umask ;openssl genrsa -out dashboard.key )
Generating RSA private key, bit long modulus
.....................................................................................................+++
..........................+++
e is (0x10001)

(2)证书签署请求

[root@k8s-master pki]# openssl req -new -key dashboard.key -out dashboard.csr -subj "/O=magedu/CN=dashboard"
[root@k8s-master pki]# openssl x509 -req -in dashboard.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out dashboard.crt -days
Signature ok
subject=/O=magedu/CN=dashboard
Getting CA Private Key

(3)定义令牌方式仅能访问default名称空间

[root@k8s-master pki]# kubectl create secret generic dashboard-cert -n kube-system --from-file=./dashboard.crt --from-file=dashboard.key=./dashboard.key   #secret创建
secret/dashboard-cert created [root@k8s-master pki]# kubectl get secret -n kube-system |grep dashboard
dashboard-cert Opaque 2 1m
kubernetes-dashboard-certs Opaque 3h
kubernetes-dashboard-key-holder Opaque 3h
kubernetes-dashboard-token-jpbgw kubernetes.io/service-account-token 3h [root@k8s-master pki]# kubectl create serviceaccount def-ns-admin -n default #创建serviceaccount
serviceaccount/def-ns-admin created [root@k8s-master pki]# kubectl create rolebinding def-ns-admin --clusterrole=admin --serviceaccount=default:def-ns-admin #service account账户绑定到集群角色admin
rolebinding.rbac.authorization.k8s.io/def-ns-admin created [root@k8s-master pki]# kubectl describe secret def-ns-admin-token-k9fz9 #查看def-ns-admin这个serviceaccount的token
Name: def-ns-admin-token-k9fz9
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=def-ns-admin
kubernetes.io/service-account.uid=56ed901c-d042-11e8-801a-000c2972dc1f Type: kubernetes.io/service-account-token Data
====
ca.crt: bytes
namespace: bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZi1ucy1hZG1pbi10b2tlbi1rOWZ6OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWYtbnMtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NmVkOTAxYy1kMDQyLTExZTgtODAxYS0wMDBjMjk3MmRjMWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWYtbnMtYWRtaW4ifQ.QfB5RR19nBv4-kFYyzW5-2n5Ksg-kON8lU18-COLBNfObQTDHs926m4k9f_5bto4YGncYi7sV_3oEec8ouW1FRjJWfY677L1IqIlwcuqc-g0DUo21zkjY_s3Lv3JSb_AfXUbZ7VTeWOhvwonqfK8uriGO1-XET-RBk1CE4Go1sL7X5qDgPjNO1g85D9IbIZG64VygplT6yZNc-b7tLNn_O49STthy6J0jdNk8lYxjy6UJohoTicy2XkZMHp8bNPBj9RqGqMSnnJxny5WO3vHxYAodKx7h6w-PtuON84lICnhiJ06RzsWjZfdeaQYg4gCZmd2J6Hdq0_K32n3l3kFLg

将该token复制后,填入验证,要知道的是,该token认证仅可以查看default名称空间的内容,如下图:

2、kubeconfig认证

(1)配置def-ns-admin的集群信息

[root@k8s-master pki]# kubectl config set-cluster kubernetes --certificate-authority=./ca.crt --server="https://192.168.56.11:6443" --embed-certs=true --kubeconfig=/root/def-ns-admin.conf
Cluster "kubernetes" set.

(2)使用token写入集群验证

[root@k8s-master ~]# kubectl config set-credentials -h  #认证的方式可以通过crt和key文件,也可以使用token进行配置,这里使用tonken

Usage:
kubectl config set-credentials NAME [--client-certificate=path/to/certfile] [--client-key=path/to/keyfile]
[--token=bearer_token] [--username=basic_user] [--password=basic_password] [--auth-provider=provider_name]
[--auth-provider-arg=key=value] [options] [root@k8s-master pki]# kubectl describe secret def-ns-admin-token-k9fz9
Name: def-ns-admin-token-k9fz9
Namespace: default
Labels: <none>
Annotations: kubernetes.io/service-account.name=def-ns-admin
kubernetes.io/service-account.uid=56ed901c-d042-11e8-801a-000c2972dc1f Type: kubernetes.io/service-account-token Data
====
ca.crt: bytes
namespace: bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZi1ucy1hZG1pbi10b2tlbi1rOWZ6OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWYtbnMtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NmVkOTAxYy1kMDQyLTExZTgtODAxYS0wMDBjMjk3MmRjMWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWYtbnMtYWRtaW4ifQ.QfB5RR19nBv4-kFYyzW5-2n5Ksg-kON8lU18-COLBNfObQTDHs926m4k9f_5bto4YGncYi7sV_3oEec8ouW1FRjJWfY677L1IqIlwcuqc-g0DUo21zkjY_s3Lv3JSb_AfXUbZ7VTeWOhvwonqfK8uriGO1-XET-RBk1CE4Go1sL7X5qDgPjNO1g85D9IbIZG64VygplT6yZNc-b7tLNn_O49STthy6J0jdNk8lYxjy6UJohoTicy2XkZMHp8bNPBj9RqGqMSnnJxny5WO3vHxYAodKx7h6w-PtuON84lICnhiJ06RzsWjZfdeaQYg4gCZmd2J6Hdq0_K32n3l3kFLg 这里的token是base64编码,此处需要进行解码操作
[root@k8s-master ~]# kubectl get secret def-ns-admin-token-k9fz9 -o jsonpath={.data.token} |base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZi1ucy1hZG1pbi10b2tlbi1rOWZ6OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWYtbnMtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NmVkOTAxYy1kMDQyLTExZTgtODAxYS0wMDBjMjk3MmRjMWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWYtbnMtYWRtaW4ifQ.QfB5RR19nBv4-kFYyzW5-2n5Ksg-kON8lU18-COLBNfObQTDHs926m4k9f_5bto4YGncYi7sV_3oEec8ouW1FRjJWfY677L1IqIlwcuqc-g0DUo21zkjY_s3Lv3JSb_AfXUbZ7VTeWOhvwonqfK8uriGO1-XET-RBk1CE4Go1sL7X5qDgPjNO1g85D9IbIZG64VygplT6yZNc-b7tLNn_O49STthy6J0jdNk8lYxjy6UJohoTicy2XkZMHp8bNPBj9RqGqMSnnJxny5WO3vHxYAodKx7h6w-PtuON84lICnhiJ06RzsWjZfdeaQYg4gCZmd2J6Hdq0_K32n3l3kFLg 配置token信息
[root@k8s-master ~]# kubectl config set-credentials def-ns-admin --token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZi1ucy1hZG1pbi10b2tlbi1rOWZ6OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWYtbnMtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NmVkOTAxYy1kMDQyLTExZTgtODAxYS0wMDBjMjk3MmRjMWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWYtbnMtYWRtaW4ifQ.QfB5RR19nBv4-kFYyzW5-2n5Ksg-kON8lU18-COLBNfObQTDHs926m4k9f_5bto4YGncYi7sV_3oEec8ouW1FRjJWfY677L1IqIlwcuqc-g0DUo21zkjY_s3Lv3JSb_AfXUbZ7VTeWOhvwonqfK8uriGO1-XET-RBk1CE4Go1sL7X5qDgPjNO1g85D9IbIZG64VygplT6yZNc-b7tLNn_O49STthy6J0jdNk8lYxjy6UJohoTicy2XkZMHp8bNPBj9RqGqMSnnJxny5WO3vHxYAodKx7h6w-PtuON84lICnhiJ06RzsWjZfdeaQYg4gCZmd2J6Hdq0_K32n3l3kFLg --kubeconfig=/root/def-ns-admin.conf
User "def-ns-admin" set.

(3)配置上下文和当前上下文

[root@k8s-master ~]# kubectl config set-context def-ns-admin@kubernetes --cluster=kubernetes --user=def-ns-admin --kubeconfig=/root/def-ns-admin.conf
Context "def-ns-admin@kubernetes" created.
[root@k8s-master ~]# kubectl config use-context def-ns-admin@kubernetes --kubeconfig=/root/def-ns-admin.conf
Switched to context "def-ns-admin@kubernetes". [root@k8s-master ~]# kubectl config view --kubeconfig=/root/def-ns-admin.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: REDACTED
server: https://192.168.56.11:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: def-ns-admin
name: def-ns-admin@kubernetes
current-context: def-ns-admin@kubernetes
kind: Config
preferences: {}
users:
- name: def-ns-admin
user:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZi1ucy1hZG1pbi10b2tlbi1rOWZ6OSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWYtbnMtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NmVkOTAxYy1kMDQyLTExZTgtODAxYS0wMDBjMjk3MmRjMWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkZWYtbnMtYWRtaW4ifQ.QfB5RR19nBv4-kFYyzW5-2n5Ksg-kON8lU18-COLBNfObQTDHs926m4k9f_5bto4YGncYi7sV_3oEec8ouW1FRjJWfY677L1IqIlwcuqc-g0DUo21zkjY_s3Lv3JSb_AfXUbZ7VTeWOhvwonqfK8uriGO1-XET-RBk1CE4Go1sL7X5qDgPjNO1g85D9IbIZG64VygplT6yZNc-b7tLNn_O49STthy6J0jdNk8lYxjy6UJohoTicy2XkZMHp8bNPBj9RqGqMSnnJxny5WO3vHxYAodKx7h6w-PtuON84lICnhiJ06RzsWjZfdeaQYg4gCZmd2J6Hdq0_K32n3l3kFLg

将/root/def-ns-admin.conf文件发送到宿主机,浏览器访问时选择Kubeconfig认证,载入该配置文件,点击登陆,即可实现访问,如图:

二、总结

1、部署dashboard:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

2、将Service改为Node Port方式进行访问:

kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kube-system

3、访问认证:

认证时的账号必须为ServiceAccount:其作用是被dashboard pod拿来由kubenetes进行认证;认证方式有2种:
token:

  • (1)创建ServiceAccount,根据其管理目标,使用rolebinding或clusterbinding绑定至合理的role或clusterrole;
  • (2)获取此ServiceAccount的secret,查看secret的详细信息,其中就有token;
  • (3)复制token到认证页面即可登录。

kubeconfig:把ServiceAccount的token封装为kubeconfig文件

  • (1)创建ServiceAccount,根据其管理目标,使用rolebinding或clusterbinding绑定至合理的role或clusterrole;
  • (2)kubectl get secret |awk '/^ServiceAccount/{print $1}'
  • KUBE_TOKEN=$(kubectl get secret SERVICEACCOUNT_SECRET_NAME -o jsonpath={.data.token} | base64 -d)
  • (3)生成kubeconfig文件
kubectl config set-cluster
kubectl config set-credentials NAME --token=$KUBE_TOKEN
kubectl config set-context
kubectl config use-context

Kubernetes学习之路(十九)之Kubernetes dashboard认证访问的更多相关文章

  1. Kubernetes学习之路(九)之kubernetes命令式快速创建应用

    1.使用命令kubectl run创建应用 语法: kubectl run NAME --image=image [--env="key=value"] [--port=port] ...

  2. 嵌入式Linux驱动学习之路(十九)触摸屏驱动、tslib测试

    触摸屏使用流程: 1. 按下产生中断. 2.在中断处理程序中启动AD转换XY坐标. 3.AD转换结束并产生AD中断. 4. 在AD的中断处理函数中上报信息,启动定时器. 5. 定时器时间到后进入中断, ...

  3. IOS学习之路十九(JSON与Arrays 或者 Dictionaries相互转换)

    今天写了个json与Arrays 或者 Dictionaries相互转换的例子很简单: 通过 NSJSONSerialization 这个类的 dataWithJSONObject: options: ...

  4. Kubernetes学习之路目录

    Kubernetes基础篇 环境说明 版本说明 系统环境 Centos 7.2 Kubernetes版本 v1.11.2 Docker版本 v18.09 Kubernetes学习之路(一)之概念和架构 ...

  5. Gradle 1.12用户指南翻译——第四十九章. Build Dashboard 插件

    本文由CSDN博客貌似掉线翻译,其他章节的翻译请参见: http://blog.csdn.net/column/details/gradle-translation.html 翻译项目请关注Githu ...

  6. Kubernetes学习之路(十五)之Ingress和Ingress Controller

    目录 一.什么是Ingress? 1.Pod 漂移问题 2.端口管理问题 3.域名分配及动态更新问题 二.如何创建Ingress资源 三.Ingress资源类型 1.单Service资源型Ingres ...

  7. Kubernetes学习之路(十)之资源清单定义

    一.Kubernetes常用资源 以下列举的内容都是 kubernetes 中的 Object,这些对象都可以在 yaml 文件中作为一种 API 类型来配置. 类别 名称 工作负载型资源对象 Pod ...

  8. Kubernetes学习之路(四)之Node节点二进制部署

    K8S Node节点部署 1.部署kubelet (1)二进制包准备 [root@linux-node1 ~]# cd /usr/local/src/kubernetes/server/bin/ [r ...

  9. kubernetes学习笔记之十:RBAC

    第一章.RBAC介绍 在Kubernetes中,授权有ABAC(基于属性的访问控制).RBAC(基于角色的访问控制).Webhook.Node.AlwaysDeny(一直拒绝)和AlwaysAllow ...

  10. Kubernetes学习之路(二十一)之网络模型和网络策略

    目录 Kubernetes的网络模型和网络策略 1.Kubernetes网络模型和CNI插件 1.1.Docker网络模型 1.2.Kubernetes网络模型 1.3.Flannel网络插件 1.4 ...

随机推荐

  1. 6.1 函数的返回值、匿名函数lambda、filter函数、map函数、reduce函数

      函数的返回值: 函数一旦执行到   return,函数就会结束,并会返回return 后面的值,如果不使用显式使用return返回,会默认返回None . return None可以简写为   r ...

  2. SQL Server 高并发Insert数据解析,实践

    在现实的生产环境中,有可能遇到高并发insert的应用.在此应用时由于堆表(Heap)和聚集表的结构不同导致在高并发的情形下insert效率不尽相同.接下来我会简单的以测试用例来简要说明.并举例说明如 ...

  3. innodb compressed 表碰到的问题,BUG么?

    innodb compressed 表碰到的问题 (摘自 http://papaisadba.puyu.me/?p=239) compressed innodb 表是MySQL 5.5开始提供的功能, ...

  4. 如何在 Azure 中标记 Windows 虚拟机

    本文介绍在 Azure 中通过 Azure 资源管理器标记 Windows 虚拟机的不同方式.标记是用户定义的键/值对,可直接放置在资源或资源组中.针对每个资源和资源组,Azure 当前支持最多 15 ...

  5. 大于2t的磁盘分区,并格式化ext4挂载

    1:MBR分区表:(MBR含义:主引导记录) 所支持的最大卷:2T (T; terabytes,1TB=1024GB) 对分区的设限:最多4个主分区或3个主分区加一个扩展分区. 2:GPT分区表:(G ...

  6. MySQL MTS复制: hitting slave_pending_jobs_size_max

    测试步骤: 从库停止复制:stop slave; 主库创建大表400万条记录. 开启从库复制:start slave; 监测从库error log持续输出: 2018-12-06T10:40:52.6 ...

  7. C# 引用的程序集没有强名称

    首先查一下什么是强名称程序集,见百度百科帖子:http://baike.baidu.com/view/1145682.htm简单来说,就是为了解决Windows Dll Hell问题的,即不同公司开发 ...

  8. 阿里八八Alpha阶段Scrum(8/12)

    今日进度 叶文滔: 已经成功解决兼容性问题,目前正在嵌入多级按钮API,预计明天可以完成 王国超: 今天终于debug了,被卡了几天的fragment嵌套listview终于成功了 俞鋆: 研究了一下 ...

  9. 小程序报错数据传输长度为 xxx 已经超过最大长度 xxx

    这个错误通常在通过base64给images赋值时发生,setData()微信设置的一次最大传输长度为1M,所有如果编码后的base64字符串长度超过了1M就会报这个错误. 如何解决? 我们可以采取曲 ...

  10. ES6标准入门之变量的解构赋值简单解说

    首先我们来看一看解构的概念,在ES6标准下,允许按照一定模式从数组和对象中提取值,然后对变量进行赋值,这被称作解构,简而言之粗糙的理解就是变相赋值. 解构赋值的规则是,只要等号右边的值不是对象或者数组 ...