aiflow部署文档

一、简介

本文档用来进行aiflow项目的完全部署

二、安装环境

2.1 安装系统镜像版本

使用Centos镜像文件：CentOS-7-x86_64-DVD-1908.iso

cat /etc/centos-release

CentOS Linux release 7.7.1908 (Core)

2.2 环境准备

准备不低于2台虚拟机。 1台master，其余的做node。本文档只准备了3台虚拟机。均在阿里云

主机名	内网	配置
master	172.31.121.126	1处理器4核心 8G
node1	172.31.121.127	1处理器4核心 8G
node2	172.31.121.128	1处理器4核心 8G

2.3 配置ssh免密登录

首先每个虚拟机用root身份登录。

关闭所有节点的selixux以及firewalld

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

setenforce 0

systemctl disable firewalld

systemctl stop firewalld

禁用交换分区。为了保证 kubelet 正常工作，你必须禁用交换分区。

swapoff -a

free -m  #查看当前swap情况，如果swap不都为0，重启 reboot

vi /etc/fstab #在swap分区这行前加 '#' 禁用掉，:wq保存退出

然后在每个执行一遍生成密钥

ssh-keygen -t rsa

#连续按三下回车，生成的密钥保存在/root/.ssh/下

然后把每台机器的id_rsa.pub公钥记录下来，每台生成的都不一样

#node2的

cat /root/.ssh/id_rsa.pub

#底下这么长一串是一整行

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCe3noZeJokxnJe2wA6gh/xINExd7+zcG2BWvs2Uu2sN0QKVtRcqp7N4NCdKNM3nYDl4FpX/bl4+ZOD5UWpPFLZsmiDZbE1PX3baGnmpgL5gzyWQwUzyLfqyLIA/MaJCSpQHszuHlSHpJF6eFxqttHietM2gKLAYzUHAYWX+UNs2XeEHPqMlnDvtA02tRVc0UT8W3uM31FwjHOTDgoDw6tWhZTQm9Lft87HJ1iYYBn root@node2

#node1的

cat /root/.ssh/id_rsa.pub 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBN7ZksfWKHln8/eTdOGVxOZqQOV+b0eZfp27Kq/leMQWdqD8DEiPh7LUBQZ9xsghH0FkdwVw6oRL8/JDFARMIlaW1Ml3nB72ZLQB0QUfzp6OJWl7vW2vUbPqceBA+Sm7UQ/by2HkwqNqJdNx+RLupZUu780gIBJ3aeP0fjAr19U46Iksgs+c+jV7EAnWTfM05vTODUVwgt2PoCZhHPO0hLfrGheBIHbuK+tAzQWMkWErw/xljKM2v root@node1

#master的

cat /root/.ssh/id_rsa.pub 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxHp/nZ1h6r/9lRkvf3F1a+Ger0jVXXJivL6vLXS0bx/gV4Q1M6LLKW2FcU3EDIbyGe8eNxApNq7KW4IgYD/+vVxBpWbDNLHXKtPls9GxgSLMX5siDIV08QGfk5Wz0e6U9Fsb3JmoKozGryDuXyY9V/KwaU+LJwo7WTMb9Q4VIR5bnGLyuGu/ygeK201W5qnjVfm2o4/JnHEU82DbOnL71NYbkg+8QakvpizHZYKKUaT92ocOCGIYA+K6a6wDX root@master

只在master上操作，在/root/.ssh/下新增一个authorized_keys文件

vim /root/.ssh/authorized_keys

#把前面三个密钥都复制进来，一个是一整行

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCe3noZeJokxnJe2wA6gh/xINExd7+zcG2BWvs2Uu2sN0QKVtRcqp7N4NCdKNM3nYDl4FpX/bl4+ZOD5UWpPFLZsmiDZbE1PX3baGnmpgL5gzyWQwUzyLfqyLIA/MaJCSpQHszuHlSHpJF6eFxqttHietM2gKLAYzUHAYWX+UNs2XeEHPqMlnDvtA02tRVc0UT8W3uM31FwjHOTDgoDw6tWhZTQm9Lft87HJ1iYYBn root@node2

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBN7ZksfWKHln8/eTdOGVxOZqQOV+b0eZfp27Kq/leMQWdqD8DEiPh7LUBQZ9xsghH0FkdwVw6oRL8/JDFARMIlaW1Ml3nB72ZLQB0QUfzp6OJWl7vW2vUbPqceBA+Sm7UQ/by2HkwqNqJdNx+RLupZUu780gIBJ3aeP0fjAr19U46Iksgs+c+jV7EAnWTfM05vTODUVwgt2PoCZhHPO0hLfrGheBIHbuK+tAzQWMkWErw/xljKM2v root@node1

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxHp/nZ1h6r/9lRkvf3F1a+Ger0jVXXJivL6vLXS0bx/gV4Q1M6LLKW2FcU3EDIbyGe8eNxApNq7KW4IgYD/+vVxBpWbDNLHXKtPls9GxgSLMX5siDIV08QGfk5Wz0e6U9Fsb3JmoKozGryDuXyY9V/KwaU+LJwo7WTMb9Q4VIR5bnGLyuGu/ygeK201W5qnjVfm2o4/JnHEU82DbOnL71NYbkg+8QakvpizHZYKKUaT92ocOCGIYA+K6a6wDX root@master

只在master上操作，发送到每个node节点，scp和第一次ssh时需要输入密码。发完了ssh测试一下

scp /root/.ssh/authorized_keys root@node1:/root/.ssh/

scp /root/.ssh/authorized_keys root@node2:/root/.ssh/

#ssh测试一下

ssh node1

ssh node2

2.4 yum配置阿里源

master和node都要用

进入目录，备份原源，获取新源

#进入目录

cd /etc/yum.repos.d/

#备份原源

cp ./CentOS-Base.repo ./CentOS-Base.repo.bak

#获取阿里源

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

#清理缓存

yum clean all

#生成新的缓存

yum makecache

# !!!!千万千万不要yum update，看到任何教程让这样做都不要做，轻则系统重启重则完全重装

#清理epel源

rpm -e epel-release

#获取epel源

curl -o /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo

#清理缓存

yum clean all

#生成新的缓存

yum makecache

2.5 配置Java环境

master和node都要做

删除自带的openJDK

cd

rpm -qa | grep java

java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64

java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64

#显示前缀为pyhton、tzdata和javapackages的三个可以不删

rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64

rpm -e --nodeps java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64

传入jdk-8u202-linux-x64.tar.gz，解压，记录当前路径

tar -zxvf jdk-8u202-linux-x64.tar.gz

cd jdk1.8.0_202

pwd  #记录路径

/root/jdk1.8.0_202

修改 /etc/profile

vim /etc/profile

#在文件的export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE HISTCONTROL这一行下面写入这几行

#注意看JAVA_HOME=后面是记录的路径

export JAVA_HOME=/root/jdk1.8.0_202

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

应用修改，查看java安装是否成功

source /etc/profile

java -version

#显示如下内容即可，

java version "1.8.0_202"

Java(TM) SE Runtime Environment (build 1.8.0_202-b08)

Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

2.6 安装Python环境

master和node都要做

安装编译环境

cd

yum -y groupinstall "Development tools"

#这是一行

yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

yum install libffi-devel -y

传入Python-3.8.10.tgz，解压，记录当前路径

tar -zxvf Python-3.8.10.tgz 

cd Python-3.8.10/

执行配置，安装

#如果有会提示file exist，不用管

mkdir /usr/local/python3

#prefix是预安装路径，不要改

./configure --prefix=/usr/local/python3

make && make install

创建软连接

ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3

ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3

#删除原来的python2

rm -rf /usr/bin/python

ln -s /usr/local/python3/bin/python3 /usr/bin/python

查看安装是否成功

python3 -V

Python 3.8.10

pip3 -V

pip 21.1.1 from /usr/local/python3/lib/python3.8/site-packages/pip (python 3.8)

导入所需的包，每个节点都要

pip3 install --upgrade pip

pip3 install kfp

pip3 install kubernetes

2.7 前期准备

完成kubeflow环境搭建.md的内容

三、安装MySQL

只在master进行

将mysql-5.6.10-linux-glibc2.5-x86_64.tar.gz包上传到master的/usr/local目录下

删除原本的mysql

yum list installed | grep mysql

yum -y remove 文件名（此处为上一步查到的结果）

rpm -qa|grep mariadb # 查询出来已安装的mariadb

rpm -e --nodeps 文件名  // 卸载mariadb，文件名为上述命令查询出来的文件

删除etc目录下的my.cnf
```
rm /etc/my.cnf
```

解压

tar -zxvf mysql-5.6.10-linux-glibc2.5-x86_64.tar.gz

#将解压好的文件夹命名为mysql

mv 解压出来的文件夹名 mysql

#进入mysql目录

cd mysql

执行以下命令来创建mysql用户组

groupadd mysql

#执行以下命令来创建一个用户名为mysql的用户并加入mysql用户组

useradd -g mysql mysql

创建配置文件my.cnf

#复制一份my.cnf到/etc目录下

cp /usr/local/mysql/my-default.cnf /etc/my.cnf

#修改配置,只需将端口一行放开且改为3307

vim /etc/my.cnf

[root@bdilab001 mysql]# cat /etc/my.cnf

# For advice on how to change settings please see

# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html

# *** DO NOT EDIT THIS FILE. It's a template which will be copied to the

# *** default location during install, and will be replaced if you

# *** upgrade to a newer version of MySQL.

[mysqld]

# Remove leading # and set to the amount of RAM for the most important data

# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.

# innodb_buffer_pool_size = 128M

# Remove leading # to turn on a very important data integrity option: logging

# changes to the binary log between backups.

# log_bin

# These are commonly set, remove the # and set as required.

# basedir = .....

#datadir = /var/lib/mysql

port = 3307

# server_id = .....

# socket = .....

# Remove leading # to set options mainly useful for reporting servers.

# The server defaults are faster for transactions and fast SELECTs.

# Adjust sizes as needed, experiment to find the optimal values.

# join_buffer_size = 128M

# sort_buffer_size = 2M

# read_rnd_buffer_size = 2M 

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

修改当前目录拥有着为mysql用户

cd /usr/local/mysql

chown -R mysql:mysql ./

安装

./scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql/ --datadir=/usr/local/mysql/data/ #安装数据库

#若报错如下：

FATAL ERROR: please install the following Perl modules before executing

./scripts/mysql_install_db:Data::Dumper

#解决：安装autoconf库（之后若出现相似错误，则按提示安装即可）

yum -y install autoconf

启动，重启，关闭

[root@bdilab001 mysql]# service mysql restart

Shutting down MySQL..                                      [  OK  ]

Starting MySQL.                                            [  OK  ]

#service mysql stop

#service mysql restart

修改密码

#添加软连接

ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql

#进入mysql命令行

mysql –u root –p

#若报错如下

mysql  Ver 14.14 Distrib 5.6.10, for linux-glibc2.5 (x86_64) using  EditLine wrapper

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Usage: mysql [OPTIONS] [database]

  -?, --help          Display this help and exit.

  -I, --help          Synonym for -?

  --auto-rehash       Enable automatic rehashing.

  。。。

#输入

[root@bdilab001 mysql]# mysql

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 1

Server version: 5.6.10 MySQL Community Server (GPL)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use mysql

#如果第一次安装直接第三步，如果忘了密码从第一步开始

#1.进入/etc/my.cnf 在[mysql]下添加skip-grant-tables 启动安全模式

vi /etc/my.cnf

#2.重启服务

service mysql restart

#3.登录

mysql -u root -p

#或者

mysql

#4.使用mysql数据库

mysql> use mysql

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

#5.改变密码

mysql> update user set password= passworD ("bdilab@1308") where user='root';

Query OK, 4 rows affected (0.00 sec)

Rows matched: 4  Changed: 4  Warnings: 0

#6.刷新权限

mysql>  flush privileges;

Query OK, 0 rows affected (0.00 sec)

#7.退出MySql编辑模式

mysql> exit

设置用户访问权限

#进入mysql命令行

[root@bdilab001 mysql]# mysql

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 1

Server version: 5.6.10 MySQL Community Server (GPL)

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

#执行权限赋予命令

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; 

#刷新权限

mysql>  flush privileges;

Query OK, 0 rows affected (0.00 sec)

#退出MySql编辑模式

mysql> exit

四、安装Hadoop

只在node2节点操作，安装单机版Hadoop和Hbase

导入hadoop-2.10.1.tar.gz和hbase-2.3.5-bin.tar.gz，进入hadoop目录

tar -zxvf hadoop-2.10.1.tar.gz

tar -zxvf hbase-2.3.5-bin.tar.gz

cd hadoop-2.10.1

#现在的绝对路径是 /root/hadoop-2.10.1

修改/root/hadoop-2.10.1/etc/hadoop/hadoop-env.sh

#第25行，值填你的java路径，和etc/profile里面的一样，前面不要有 # (井号)

export JAVA_HOME=/root/jdk1.8.0_202

修改/root/hadoop-2.10.1/etc/hadoop/yarn-env.sh

#第23行，值填你的java路径，和etc/profile里面的一样，前面不要有 # (井号)

export JAVA_HOME=/root/jdk1.8.0_202

修改/root/hadoop-2.10.1/etc/hadoop/core-site.xml

<configuration>

	<property>

		<name>fs.defaultFS</name>

		<value>hdfs://localhost:9000</value>

	</property>

	<property>

		<name>io.file.buffer.size</name>

		<value>131072</value>

	</property>

	<property>

		<name>hadoop.tmp.dir</name>

		<value>file:/home/cloud/temp/hadoop-2.10.1</value>

		<description>Abasefor other temporary directories.</description>

	</property>

</configuration>

修改/root/hadoop-2.10.1/etc/hadoop/hdfs-site.xml

<configuration>

	<property>

		<name>dfs.namenode.secondary.http-address</name>

		<value>localhost:9001</value>

	</property>

	<property>

		<name>dfs.namenode.name.dir</name>

		<value>file:/root/hadoop-2.10.1/dfs/name</value>

	</property>

	<property>

		<name>dfs.datanode.data.dir</name>

		<value>file:/root/hadoop-2.10.1/dfs/data</value>

	</property>

	<property>

		<name>dfs.replication</name>

		<value>1</value>

	</property>

	<property>

		<name>dfs.webhdfs.enabled</name>

		<value>true</value>

	</property>

</configuration>

修改/root/hadoop-2.10.1/etc/hadoop/mapred-site.xml

#先创建文件

cp /root/hadoop-2.10.1//etc/hadoop/mapred-site.xml.template  /root/hadoop-2.10.1//etc/hadoop/mapred-site.xml

再改

<configuration>

	<property>

		<name>mapreduce.framework.name</name>

		<value>yarn</value>

	</property>

	<property>

		<name>mapreduce.jobhistory.address</name>

		<value>localhost:10020</value>

	</property>

	<property>

		<name>mapreduce.jobhistory.webapp.address</name>

		<value>localhost:50030</value>

	</property>

	<property>

		<name>mapreduce.reduce.memory.mb</name>

		<value>4096</value>

	</property>

</configuration>

修改/root/hadoop-2.10.1/etc/hadoop/slaves，如果里面是localhost就不用改了

localhost

修改/root/hadoop-2.10.1/etc/hadoop/yarn-site.xml

<configuration>

	<property>

		<name>yarn.nodemanager.aux-services</name>

		<value>mapreduce_shuffle</value>

	</property>

	<property>

		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

		<value>org.apache.hadoop.mapred.ShuffleHandler</value>

	</property>

	<property>

		<name>yarn.resourcemanager.address</name>

		<value>localhost:8032</value>

	</property>

	<property>

		<name>yarn.resourcemanager.scheduler.address</name>

		<value>localhost:8030</value>

	</property>

	<property>

		<name>yarn.resourcemanager.resource-tracker.address</name>

		<value>localhost:8031</value>

	</property>

	<property>

		<name>yarn.resourcemanager.admin.address</name>

		<value>localhost:8033</value>

	</property>

	<property>

		<name>yarn.resourcemanager.webapp.address</name>

		<value>localhost:8088</value>

	</property>

</configuration>

初始化主节点

/root/hadoop-2.10.1/bin/hadoop namenode -format

启动hadoop

/root/hadoop-2.10.1/sbin/start-dfs.sh

#要输入yes就yes

/root/hadoop-2.10.1/sbin/start-yarn.sh

查看进程

jps

20200 Jps

17106 SecondaryNameNode

18483 ResourceManager

16811 DataNode

16559 NameNode

18639 NodeManager

#一共五个进程

然后在浏览器输入http://node2:50070查看详情，node2换成ip，注意是http没有s

关闭Hadoop命令。注意如果开启了HBase，在关闭Hadoop之前必须关闭HBase

/root/hadoop-2.10.1/sbin/stop-yarn.sh

/root/hadoop-2.10.1/sbin/stop-dfs.sh

五、安装Hbase

只在node2进行

回到上层目录，本文是/root/。然后进入hbase目录

cd ..

cd hbase-2.3.5

修改/root/hbase-2.3.5/conf/hbase-env.sh

#第28行，值填你的java路径，和etc/profile里面的一样，前面不要有 # (井号)

export JAVA_HOME=/root/jdk1.8.0_202

#第126行，把前面#去掉

export HBASE_MANAGES_ZK=true

修改/root/hbase-2.3.5/conf/hbase-site.xml

<configuration>

  <property>

    <name>hbase.rootdir</name>

    <value>hdfs://localhost:9000/hbase</value>

  </property>

  <property>

    <name>hbase.cluster.distributed</name>

    <value>true</value>

  </property>

  <property>

    <name>hbase.zookeeper.quorum</name>

    <value>localhost</value>

  </property>

  <property>

    <name>hbase.tmp.dir</name>

    <value>file:/home/cloud/hbase/tmp</value>

  </property>

  <property>

    <name>hbase.master.port</name>

    <value>60000</value>

  </property>

  <property>

    <name>hbase.master.info.port</name>

    <value>60020</value>

  </property>

  <property>

    <name>dfs.replication</name>

    <value>1</value>

  </property>

  <property>

    <name>hbase.unsafe.stream.capability.enforce</name>

    <value>false</value>

  </property>

</configuration>

修改/root/hbase-2.3.5/conf/regionservers，里面是localhost就不用改

localhost

在启动hbase之前，一定要确保hadoop启动了(通过jps和50070确定)，没启动的话启动hadoop

启动hadoop

/root/hadoop-2.10.1/sbin/start-dfs.sh

/root/hadoop-2.10.1/sbin/start-yarn.sh

启动hbase

/root/hbase-2.3.5/bin/start-hbase.sh

jps查看

jps

19388 Jps

19090 HRegionServer

18772 HQuorumPeer

18940 HMaster

15138 NameNode

15390 DataNode

15759 SecondaryNameNode

16277 ResourceManager

16434 NodeManager

#hadoop5个，hbase三个HRegionServer、HQuorumPeer、HMaster

然后在浏览器输入http://node2:60020查看详情，node2换成ip，注意是http没有s

备注，关闭hbase命令。

#一定要先关HBase再关Hadoop!

/root/hbase-2.3.5/bin/stop-hbase.sh

六、安装Redis

只在master进行

首先导入redis-6.2.4.tar.gz，进入目录

tar -zxvf redis-6.2.4.tar.gz

cd redis-6.2.4

安装

make

#等待执行结束

make install

修改配置文件，本机位置是/root/redis-6.2.4/redis.conf

#第75行有个bind，注释掉

# bind 127.0.0.1 -::1

#第94行

protected-mode no

#第257行

daemonize yes

#第901行

requirepass bdilab@1308

保存后，启动redis-server

#目录是/root/redis-6.2.4/，使用redis.conf启动

/root/redis-6.2.4/src/redis-server /root/redis-6.2.4/redis.conf

启动一个redis-cli测试

#启动

/root/redis-6.2.4/src/redis-cli

#输入set name 1，意为创建一个变量name，值为1

127.0.0.1:6379> set name 1

(error) NOAUTH Authentication required.

#出现这个 NOAUTH Authentication required说明密码配置生效了

127.0.0.1:6379> AUTH bdilab@1308

OK

127.0.0.1:6379> set testname 1

OK

127.0.0.1:6379> exit

七、NFS搭建

7.1服务器端安装

在每个节点上安装nfs服务（master，node）

yum -y install nfs-utils rpcbind

#查看是否安装正常

[root@bdilab001 nfs]# rpm -qa nfs-utils

nfs-utils-1.3.0-0.68.el7.x86_64

#创建/nfs/aiflow目录

mkdir /nfs

mkdir /nfs/aiflow

#更改目录权限（*很重要）

chown -R nfsnobody.nfsnobody /nfs/aiflow/

nfs服务端配置目录(master)

NFS服务器端的主要配置文件为/etc/exports，通过此配置文件可以设置服务端的共享文件目录。这个目录文件的修改需要非常小心，若填错会导致系统不可启动

echo "/nfs/aiflow *(rw,async,no_root_squash)" >> /etc/exports

#每条配置记录由NFS共享目录、NFS客户端地址和参数这3部分组成，格式如下[NFS共享目录] [NFS客户端地址1(参数1,参数2,参数3……)] [客户端地址2(参数1,参数2,参数3……)]NFS共享目录：nfs服务端上共享出去的文件目录；

	#NFS客户端地址：允许其访问的NFS服务端的客户端地址,*表示所有客户端IP都可以访问；

	#访问参数：括号中逗号分隔项，主要是一些权限选项,rw表示读写权限

#输入以下命令使共享目录生效

exportfs -r

启动客户端服务(master)

[root@bdilab001 nfs]# systemctl start rpcbind

[root@bdilab001 nfs]# systemctl enable rpcbind

[root@bdilab001 nfs]# systemctl start nfs

[root@bdilab001 nfs]# systemctl enable nfs

Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

#查看启动成功

[root@bdilab001 nfs]# showmount -e localhost

Export list for localhost:

/nfs/aiflow *

#需要放行TCP和UDP的端口111,20048和2048（这个只需要TCP）

7.2 客户端安装

启动服务（node）

[root@bdilab003 /]# systemctl start rpcbind

[root@bdilab003 /]# systemctl enable rpcbind

[root@bdilab003 /]# systemctl start nfs

[root@bdilab003 /]# systemctl enable nfs

Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.

将客户端目录挂载到服务端的共享目录（node）

#显示nfs挂载信息

[root@bdilab003 /]# showmount -e 120.27.69.55

Export list for 120.27.69.55:

/nfs/aiflow *

#若是没有开放111，20048端口会报如下错误（*必须要tcp和udp都开放）

[root@bdilab002 /]# showmount -e 120.27.69.55

clnt_create: RPC: Port mapper failure - Timed out

#挂载

mount -t nfs 120.27.69.55:/nfs/aiflow /nfs/aiflow

#120.27.69.55是服务器地址，将node节点的/nfs/aiflow目录挂载到服务器上/nfs/aiflow目录上。

#如果报错如下，则是没有设置目录权限或者没有开放2049端口(TCP)

mount.nfs: Connection timed out

测试

[root@bdilab002 /]# df -h

#可以找到下面一行

120.27.69.55:/nfs/aiflow   99G   31G   64G  33% /nfs/aiflow

八、（可选）安装harbor(docker的镜像仓库，防止镜像丢失)

只在master进行

先安装彻底、y

导入docker-compose-Linux-x86_64

cp ./docker-compose-Linux-x86_64 /usr/local/bin/

chmod +x /usr/local/bin/docker-compose-Linux-x86_64

chmod 777 /usr/local/bin/docker-compose-Linux-x86_64

ln -s /usr/local/bin/docker-compose-Linux-x86_64 /usr/bin/docker-compose

#查看版本

docker-compose --version

docker-compose version 1.29.2, build 5becea4c

进入目录

#回到根目录

cd

#创建目录

mkdir harbor

cd harbor

导入harbor-offline-installer-v2.2.3.tgz

tar -zxvf harbor-offline-installer-v2.2.3.tgz

安装Harbor

回到harbor目录修改配置文件

#此时的绝对路径是/root/harbor/harbor

cp harbor.yml.tmpl harbor.yml

#修改harbor.yml

vim harbor.yml

#以下为要修改的内容

#第5行，监听地址，可以是域名，这里填的是master的 内网ip

hostname: 137.27.69.33

#第10行，http端口

  port: 8081

#注释12-18行

#注释就是在这一行的最前面加 #  (井号)

#第34行，密码

harbor_admin_password: bdilab@1308

修改完毕按esc，输入:wq (前面有冒号)保存退出

运行./install.sh

./install.sh

#输出以下内容即安装成功

[Step 5]: starting Harbor ...

Creating network "harbor_harbor" with the default driver

Creating harbor-log ... done

Creating harbor-portal ... done

Creating redis         ... done

Creating registry      ... done

Creating harbor-db     ... done

Creating registryctl   ... done

Creating harbor-core   ... done

Creating nginx             ... done

Creating harbor-jobservice ... done

 ----Harbor has been installed and started successfully.----

浏览器访问http://120.27.69.33:8081，用户名admin，密码bdilab@1308，注意是http，且访问公网ip(120开头这个是外网)和内网ip都行

备注：启动和停止操作

docker-compose down #停止

docker-compose up -d #启动

修改所有节点的docker配置文件以访问harbor

#先停止harbor（在目录/root/harbor/harbor下）

cd /root/harbor/harbor

docker-compose down   #停止

#修改docker配置文件，在/etc/docker/daemon.json

vim /etc/docker/daemon.json

#修改为以下内容

{

  "insecure-registries": ["172.31.121.126:8081"]

}

重启所有节点的docker

systemctl daemon-reload

systemctl restart docker

等几分钟后在master启动

docker-compose up -d  #启动

docker登录

docker login 172.31.121.126:8081

#然后根据提示输入用户名和密码

使用方法

#首先查看本地镜像

docker images

#给一个镜像打tag，最后面为新的tag地址，格式为    仓库地址  +  要上传到的仓库项目名  +  镜像名:tag

docker tag SOURCE_IMAGE[:TAG] 137.27.69.33:8081/library/REPOSITORY[:TAG]

#推送到harbor仓库，然后可以在在线地址看到

docker push 137.27.69.33:8081/library/REPOSITORY[:TAG]

#从harbor下载到本地docker

docker push 137.27.69.33:8081/library/REPOSITORY[:TAG]

然后可以在harbor中pull很多gcr.io等等这些不能在公网下载到的镜像保存下来，防止丢失

创建secret使得kubernetes可以从私有仓库拉取镜像

[root@bdilab001 harbor]# kubectl create secret docker-registry aiflow --docker-server=aiflow --docker-username=admin --docker-password=bdilab@1308

secret/aiflow created

#kubectl create secret docker-registry secret名称（在创建yaml的时候需要使用） --docker-server=Harbor私有镜像仓库的域名 --docker-username=用户名 --docker-password=密码

#查看已创建的secret

[root@bdilab001 harbor]# kubectl get secrets

NAME                  TYPE                                  DATA   AGE

aiflow                kubernetes.io/dockerconfigjson        1      3m38s

default-token-gbntq   kubernetes.io/service-account-token   3      8d

istio.default         istio.io/key-and-cert                 3      6d14h

九、部署AIflow项目