Deepo
Deepo is a series of Docker images that
- allows you to quickly set up your deep learning research environment
- supports almost all commonly used deep learning frameworks
- supports GPU acceleration (CUDA and cuDNN included), also works in CPU-only mode
- works on Linux (CPU version/GPU version), Windows (CPU version) and OS X (CPU version)
and their Dockerfile generator that
- allows you to customize your own environment with Lego-like modules
- automatically resolves the dependencies for you
GPU Version
Installation
Step 1. Install Docker and nvidia-docker.
Step 2. Obtain the all-in-one image from Docker Hub
docker pull ufoym/deepo
For users in China who may suffer from slow speeds when pulling the image from the public Docker registry, you can pull deepo
images from the China registry mirror by specifying the full path, including the registry, in your docker pull command, for example:
docker pull registry.docker-cn.com/ufoym/deepo
or
docker pull hub-mirror.c.163.com/ufoym/deepo
or
docker pull docker.mirrors.ustc.edu.cn/ufoym/deepo
Usage
Now you can try this command:
docker run --runtime=nvidia --rm ufoym/deepo nvidia-smi
This should work and enables Deepo to use the GPU from inside a docker container. If this does not work, search the issues section on the nvidia-docker GitHub – many solutions are already documented. To get an interactive shell to a container that will not be automatically deleted after you exit do
docker run --runtime=nvidia -it ufoym/deepo bash
If you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run --runtime=nvidia -it -v /host/data:/data -v /host/config:/config ufoym/deepo bash
This will make /host/data
from the host visible as /data
in the container, and /host/config
as /config
. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host
or --shm-size
command line options to docker run
.
docker run --runtime=nvidia -it --ipc=host ufoym/deepo bash
CPU Version
Installation
Step 1. Install Docker.
Step 2. Obtain the all-in-one image from Docker Hub
docker pull ufoym/deepo:cpu
Usage
Now you can try this command:
docker run -it ufoym/deepo:cpu bash
If you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run -it -v /host/data:/data -v /host/config:/config ufoym/deepo:cpu bash
This will make /host/data
from the host visible as /data
in the container, and /host/config
as /config
. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host
or --shm-size
command line options to docker run
.
docker run -it --ipc=host ufoym/deepo:cpu bash
You are now ready to begin your journey.
$ python
>>> import tensorflow
>>> import sonnet
>>> import torch
>>> import keras
>>> import mxnet
>>> import cntk
>>> import chainer
>>> import theano
>>> import lasagne
>>> import caffe
>>> import caffe2
$ caffe --version
caffe version 1.0.0
$ darknet
usage: darknet <function>
$ th
│ ______ __ | Torch7
│ /_ __/__ ________/ / | Scientific computing for Lua.
│ / / / _ \/ __/ __/ _ \ | Type ? for help
│ /_/ \___/_/ \__/_//_/ | https://github.com/torch
│ | http://torch.ch
│
│th>
Customization
Note that docker pull ufoym/deepo
mentioned in Quick Start will give you a standard image containing all available deep learning frameworks. You can customize your own environment as well.
Unhappy with all-in-one solution?
If you prefer a specific framework rather than an all-in-one image, just append a tag with the name of the framework. Take tensorflow for example:
docker pull ufoym/deepo:tensorflow
Jupyter support
Step 1. pull the image with jupyter support
docker pull ufoym/deepo:all-jupyter
Step 2. run the image
docker run --runtime=nvidia -it -p 8888:8888 --ipc=host ufoym/deepo:all-jupyter jupyter notebook --no-browser --ip=0.0.0.0 --allow-root --NotebookApp.token= --notebook-dir='/root'
Build your own customized image with Lego-like modules
Step 1. prepare generator
git clone https://github.com/ufoym/deepo.git
cd deepo/generator
Step 2. generate your customized Dockerfile
For example, if you like pytorch
and lasagne
, then
python generate.py Dockerfile pytorch lasagne
This should generate a Dockerfile that contains everything for building pytorch
and lasagne
. Note that the generator can handle automatic dependency processing and topologically sort the lists. So you don’t need to worry about missing dependencies and the list order.
You can also specify the version of Python:
python generate.py Dockerfile pytorch lasagne python==3.6
Step 3. build your Dockerfile
docker build -t my/deepo .
This may take several minutes as it compiles a few libraries from scratch.
Comparison to alternatives
. | modern-deep-learning | dl-docker | jupyter-deeplearning | Deepo |
---|---|---|---|---|
ubuntu | 16.04 | 14.04 | 14.04 | 18.04 |
cuda | X | 8.0 | 6.5-8.0 | 8.0-10.0/None |
cudnn | X | v5 | v2-5 | v7 |
onnx | X | X | X | O |
theano | X | O | O | O |
tensorflow | O | O | O | O |
sonnet | X | X | X | O |
pytorch | X | X | X | O |
keras | O | O | O | O |
lasagne | X | O | O | O |
mxnet | X | X | X | O |
cntk | X | X | X | O |
chainer | X | X | X | O |
caffe | O | O | O | O |
caffe2 | X | X | X | O |
torch | X | O | O | O |
darknet | X | X | X | O |
Tags
Available Tags
. | CUDA 10.0 / Python 3.6 | CPU-only / Python 3.6 |
---|---|---|
all-in-one | latest all all-py36 py36-cu100 all-py36-cu100 |
all-py36-cpu all-cpu py36-cpu cpu |
all-in-one with jupyter | all-jupyter-py36-cu100 all-jupyter-py36 all-jupyter |
all-py36-jupyter-cpu py36-jupyter-cpu |
Theano | theano-py36-cu100 theano-py36 theano |
theano-py36-cpu theano-cpu |
TensorFlow | tensorflow-py36-cu100 tensorflow-py36 tensorflow |
tensorflow-py36-cpu tensorflow-cpu |
Sonnet | sonnet-py36-cu100 sonnet-py36 sonnet |
sonnet-py36-cpu sonnet-cpu |
PyTorch / Caffe2 | pytorch-py36-cu100 pytorch-py36 pytorch |
pytorch-py36-cpu pytorch-cpu |
Keras | keras-py36-cu100 keras-py36 keras |
keras-py36-cpu keras-cpu |
Lasagne | lasagne-py36-cu100 lasagne-py36 lasagne |
lasagne-py36-cpu lasagne-cpu |
MXNet | mxnet-py36-cu100 mxnet-py36 mxnet |
mxnet-py36-cpu mxnet-cpu |
CNTK | cntk-py36-cu100 cntk-py36 cntk |
cntk-py36-cpu cntk-cpu |
Chainer | chainer-py36-cu100 chainer-py36 chainer |
chainer-py36-cpu chainer-cpu |
Caffe | caffe-py36-cu100 caffe-py36 caffe |
caffe-py36-cpu caffe-cpu |
Torch | torch-cu100 torch |
torch-cpu |
Darknet | darknet-cu100 darknet |
darknet-cpu |
Deprecated Tags
. | CUDA 9.0 / Python 3.6 | CUDA 9.0 / Python 2.7 | CPU-only / Python 3.6 | CPU-only / Python 2.7 |
---|---|---|---|---|
all-in-one | py36-cu90 all-py36-cu90 |
all-py27-cu90 all-py27 py27-cu90 |
all-py27-cpu py27-cpu |
|
all-in-one with jupyter | all-jupyter-py36-cu90 |
all-py27-jupyter py27-jupyter |
all-py27-jupyter-cpu py27-jupyter-cpu |
|
Theano | theano-py36-cu90 |
theano-py27-cu90 theano-py27 |
theano-py27-cpu |
|
TensorFlow | tensorflow-py36-cu90 |
tensorflow-py27-cu90 tensorflow-py27 |
tensorflow-py27-cpu |
|
Sonnet | sonnet-py36-cu90 |
sonnet-py27-cu90 sonnet-py27 |
sonnet-py27-cpu |
|
PyTorch | pytorch-py36-cu90 |
pytorch-py27-cu90 pytorch-py27 |
pytorch-py27-cpu |
|
Keras | keras-py36-cu90 |
keras-py27-cu90 keras-py27 |
keras-py27-cpu |
|
Lasagne | lasagne-py36-cu90 |
lasagne-py27-cu90 lasagne-py27 |
lasagne-py27-cpu |
|
MXNet | mxnet-py36-cu90 |
mxnet-py27-cu90 mxnet-py27 |
mxnet-py27-cpu |
|
CNTK | cntk-py36-cu90 |
cntk-py27-cu90 cntk-py27 |
cntk-py27-cpu |
|
Chainer | chainer-py36-cu90 |
chainer-py27-cu90 chainer-py27 |
chainer-py27-cpu |
|
Caffe | caffe-py36-cu90 |
caffe-py27-cu90 caffe-py27 |
caffe-py27-cpu |
|
Caffe2 | caffe2-py36-cu90 caffe2-py36 caffe2 |
caffe2-py27-cu90 caffe2-py27 |
caffe2-py36-cpu caffe2-cpu |
caffe2-py27-cpu |
Torch | torch-cu90 |
torch-cu90 torch |
torch-cpu |
|
Darknet | darknet-cu90 |
darknet-cu90 darknet |
darknet-cpu |
Citation
@misc{ming2017deepo,
author = {Ming Yang},
title = {Deepo: set up deep learning environment in a single command line.},
year = {2017},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/ufoym/deepo}}
}
Contributing
We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.
Licensing
Deepo is MIT licensed.
Deepo的更多相关文章
- ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(一)
ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(一) 本教程适用于想要在远程服务器上配置docker图形界面用于深度学习的用户. (一)ubuntu18.04配置n ...
- ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(三)
ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(三) 本教程适用于想要在远程服务器上配置docker图形界面用于深度学习的用户. (三)配置远程桌面连接访问dock ...
- ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(二)
ubuntu18.04配置nvidia docker和远程连接ssh+远程桌面连接(二) 本教程适用于想要在远程服务器上配置docker图形界面用于深度学习的用户. (二)nvidia docker配 ...
- Vmvare + Ubuntu 16.04环境搭建 + 相关软件安装配置笔记【深度学习】
前言 由于学习与工作的需要,加上之前配置好的vmmachines都损坏了,我就重新弄一个ubuntu虚拟机,配置一下环境,给自己留个记录 1.文件 2.配置过程 1.在Vmware中新建虚拟机,自定义 ...
- 教你如何用Docker快速搭建深度学习环境
本教程搭建集 Tensorflow.Keras.Coffe.PyTorch 等深度学习框架于一身的环境,及jupyter. 本教程使用nvidia-docker启动实例,通过本教程可以从一个全新的Ub ...
- [AI] 切换cuda版本的万金油
1. 环境 ubuntu16.04 GTX1080Ti x 4 nvidia-418 cuda-10.1 pytorch1.0.0 目标:在最新的显卡驱动下,使用不同版本的cuda和深度学习框架来执行 ...
- docker出现如下错误:Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
在docker中配置deepo时出现了错误: 在出现这个错误之前,我是先用如下命令查看NVIDIA-docker是否安装成功. docker run --runtime=nvidia --rm nvi ...
- 服务器搭建远程docker深度学习环境
服务器搭建远程docker深度学习环境 本文大部分内容参考知乎文章 Docker+PyCharm快速搭建机器学习开发环境 搭建过程中出现ssh连接问题可以查看最后的注意事项 Docker Docker ...
随机推荐
- Nginx 反向代理功能-实现http反向代理
Nginx 反向代理功能-实现http反向代理 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任.
- 测试cnblog
this prelkdfkdjfkljasdlkfjlkjlkjsdf this this code this this pre codeljkjkjkdjkdjkdjkjdkjdkdlajlkdjf ...
- LCD编程_简单测试
首先,需要编写一个led_test.c的文件,依据代码框架,在led_test.c中我们能够看到的只是led.c.我们是看不到led_controller.c的.比如说,在led_test.c中,需要 ...
- jenkins pipeline之stash/unstash
在使用jenkins时,经常会问到文件怎么存储? 正好jenkins有一个stash的功能,下面简单介绍下: 先看看官方是怎么解释的: stash: Stash some files to be us ...
- CentOS7.5 系统最小化安装与初始化配置
CentOS7.5 系统最小化安装与初始化配置 1.安装标准化的系统 1.1.系统安装期间的语言 选择:中文-简体中文,安装完成也会默认支持中文输出,便于管理 1.2.时区选择 亚洲上海,CST时区( ...
- CF1081C-Colorful Bricks-(dp||组合数)
http://codeforces.com/problemset/problem/1081/C 题意:有n个排成一行板块,有m种颜色,要让这些板块有k对相邻板块不同颜色,有多少种涂色方法? 比如样例2 ...
- sublime text3中Package Control的安装
手动安装Package Control,亲测有效成功 1.点击https://github.com/wbond/package_control去github下载Package Control安装包下载 ...
- 洛谷P2704 [NOI2001]炮兵阵地题解
题目描述 司令部的将军们打算在\(N * M\)的网格地图上部署他们的炮兵部队.一个\(N * M\)的地图由N行M列组成,地图的每一格可能是山地(用\("H"\) 表示),也可能 ...
- 【cf比赛记录】Codeforces Round #604 (Div. 2)
比赛传送门 感觉这场是最近以来做过的最顺手的一场,持续上分,开心w A了 前三题,然后第四题其实还有半个多小时,但怕身体撑不住,就先退了,其实第四题也很简单 自己认为的算法标签: A.暴力模拟.字 ...
- prometheus、node_exporter、cAdvisor常用参数
本节将介绍一下我在使用过程中用到的promethues.node_exporter.cAdvisor的常用参数,做一个总结 一.prometheus prometheus分为容器安装和二进制文件安装, ...