k8s tensorflow

Online learning

github source

Kubeflow实战系列

Prepare


了解ksonnet初探Google之Kubeflow （采用的是0.8.0）
install depinstall ksKubeflow 安利：在 Kubernetes 上进行机器学习
ksonnet功能类似helm kubernetes应用的包管理工具。 可以对比存裸部署sparkKubeflow 入门Jupyter notebook入门教程（上） - CSDN博客 Jupyter notebook教程 （more)
Docker 入门教程

最新的安装方式：

KSURL="https://github.com/kubeflow/kubeflow/releases/latest"
LASTURL=""
LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
    LASTURL=$(wget -O /dev/null --content-disposition $KSURL  >& |awk '/^Location: /{print $2}')
    if [ $? -ne  ]; then
        echo "Error: Can not find the latest ksonnet version by wget."
    fi
fi

LATES=${LASTURL##*/}
echo $LATES

mkdir kubeflow_$LATES
cd kubeflow_$LATES
export KUBEFLOW_TAG=$LATES

curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash

echo "Please run:"
echo "export KUBEFLOW_SRC=`pwd`"
echo "export KFAPP=~/kfapp"

cat > `pwd`/kubeflow.env <<EOL
export KUBEFLOW_TAG=$LATES
export KUBEFLOW_SRC=`pwd`
export KFAPP=~/kfapp
EOL

echo "Please run the commnad before doing any kubeflow setting up:"
echo "source `pwd`/kubeflow.env"

------------------------------------------------
以下是陈旧的安装方式源代码安装ksonnet脚本（ksonnet的变化很大， 经常出问题。 不建议源码安装）

go get -u github.com/golang/dep/cmd/dep
go get github.com/ksonnet/ksonnet

# Build and install binary under shortname `ks` into $GOPATH/bin
cd $GOPATH/src/github.com/ksonnet/ksonnet
sudo -E -H make install
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig

安装最最新发布版的ksonnet

KSURL="https://github.com/ksonnet/ksonnet/releases/latest"
LASTURL=""
LASTURL=$(curl $KSURL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
    LASTURL=$(wget -O /dev/null --content-disposition $KSURL  >& |awk '/^Location: /{print $2}')
    if [ $? -ne  ]; then
        echo "Error: Can not find the latest ksonnet version by wget."
    fi
fi
LATES=""
if [ "$LASTURL" = "$KSURL" ];  then
  echo "Error: Not find the latest ksonnet version by curl."
else
  if test -n "$LASTURL"; then
    echo "OK: Find the latest ksonnet version."
  else
    echo "Error: Not find the latest ksonnet version by wget."
  fi
fi
LATES=${LASTURL##*/}
if [[ $LATES =~ ^v[-.]{,}$ ]]; then
    LATES=${LATES#v}
fi
wget https://github.com/ksonnet/ksonnet/releases/download/v$LATES/ks_${LATES}_linux_amd64.tar.gz

GOROOT=/usr/local

tar  -C $GOROOT/bin -xzf ks_${LATES}_linux_amd64.tar.gz --wildcards --no-anchored 'ks'
ln -s $GOROOT/bin/ks_${LATES}_linux_amd64/ks $GOROOT/bin/ks
# tar --extract --file=ks_0..0_linux_amd64.tar.gz ks_0..0_linux_amd64/ks

Using Kubeflow （翻译） (中文) （latest EN)

(中文简要介绍)

ks init my-kubeflow
sudo chown $USER:$USER ~/my-kubeflow -R
cd my-kubeflow
ks registry add kubeflow \
github.com/google/kubeflow/tree/master/kubeflow
ks pkg install kubeflow/core
ks pkg install kubeflow/tf-serving
ks pkg install kubeflow/tf-job
ks generate core kubeflow-core --name=kubeflow-core

check  kubectl contexts:

kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://localhost:6443/
  name: local-up-cluster
contexts:
- context:
    cluster: local-up-cluster
    user: local-up-cluster
  name: local-up-cluster
current-context: local-up-cluster
kind: Config
preferences: {}
users:
- name: local-up-cluster
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Jupyter启动之后运行tf的example

Troubleshooting

1.

$ sudo -E -H ks pkg install kubeflow/core
ERROR GET https://api.github.com/repos/google/kubeflow/commits/master: 403 API rate limit exceeded for 134.134.139.74. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 26m21s]

 403 API rate limit of 60 still exceeded

issue 1254

issues 195

   generate one   GITHUB_TOKEN

GTOKEN=801c4f41553571107afa06d459c0910900aeb5fb
GTOKEN=5d67c71ea35d85a5459e6e05a3075363d438b8c7
export GITHUB_TOKEN=$GTOKEN
echo "GITHUB_TOKEN=$GTOKEN" >> ~/.profile

2.  Can not ks apply default -c kubeflow-core with ks 0.9.5

rollback to v0.8.0

@seekyiyi I have the same problem using v0.9.1(or dev build) version of ksonnet, but I use v0.8.0 can be solved.

centos 下安装go

git clone https://github.com/jamesarems/gocentos.git
cd gocentos
sh InstallGO.sh
source /etc/profile
go get -u github.com/golang/dep/cmd/dep
echo 'GOROOT=`go env |grep "GOROOT" |cut -d "=" -f2`
GOROOT=${GOROOT#\"}
GOROOT=${GOROOT%\"}
echo $GOROOT

GOPATH=`go env |grep GOPATH |cut -d "=" -f `
GOPATH=${GOPATH%\"}
GOPATH=${GOPATH#\"}
echo $GOPATH
declare -x GOROOT
declare -x GOPATH
export PATH=${PATH}:${GOROOT}/bin:${GOPATH}/bin' >> ~/.bashrc

source ~/.bashrc
echo $GOPATH
go get github.com/ksonnet/ksonnet
cd $GOPATH/src/github.com/ksonnet/ksonnet
make install
export KUBECONFIG=/root/config
ks init my-kubeflow

tensorflow 学习

get redirect URL

wget -O /dev/null --content-disposition https://github.com/ksonnet/ksonnet/releases/latest --server-response 2>&1 |   awk '(NR==1){SRC=$3;} /^  Location: /{DEST=$2} END{ print SRC, DEST}'

https://github.com/ksonnet/ksonnet/releases/latest https://github.com/ksonnet/ksonnet/releases/tag/v0.11.0

setup local registry （git hub)

Sharing a local registry with minikube(推荐)

在Kubernetes集群上部署高可用Harbor镜像仓库

docker 私有仓库Harbor

11.部署私有 docker registry

Kubernetes镜像仓库-Harbor的Helm部署

在kubernetes 集群上搭建docker 私有仓库Harbor

Get latest Github app version:

# cat github-latest.sh

#!/bin/bash
URL=$
APPURL=${URL%/releases/latest*}
APP=${APPURL##*/}
LASTURL=""
LASTURL=$(curl $URL -s -L -I -o /dev/null -w '%{url_effective}')
if (test $? -ne ); then
    LASTURL=$(wget -O /dev/null --content-disposition $URL  >& |awk '/^Location: /{print $2}')
    if [ $? -ne  ]; then
        echo "Error: Can not find the latest $APP version by wget."
    fi
fi
LATES=""
if [ "$LASTURL" = "$URL" ];  then
  echo "Error: Not find the latest $APP version by curl."
else
  if test -n "$LASTURL"; then
    echo "OK: Find the latest $APP version."
  else
    echo "Error: Not find the latest $APP version by wget."
  fi
fi
LATES=${LASTURL##*/}
if [[ $LATES =~ ^v[-.]{,}$ ]]; then
    LATES=${LATES#v}
fi
echo "Version:$LATES"
exit 

URL=https://github.com/goharbor/harbor/releases/latest
V=`./github-latest.sh $URL|grep Version: |cut -d ":" -f2`

BIN=https://storage.googleapis.com/harbor-releases/harbor-offline-installer-v$V.tgz

wget $BIN

harbor 官网使用， 使用k8s部署

doker official setup registry