【Tools】Anaconda Operaction

专为数据科学和机器学习工作流程而设计，是一个开源包管理器，环境管理器，以及Python和R编程语言的分发。它通常用于大规模数据处理，科学计算和预测分析。pip install xxx ,在特定环境里使用pip，下载的包会存在特定环境的目录里面，例如:D:\Anaconda3\envs\nlp\Lib\site-packages\fasttext； conda install xxx ，不管在什么环境下载的包，都统一放在一个目录里面：D:\Anaconda3\pkgs\fasttext， 如果另一个环境也需要这个包，不需要重新下载；

Anaconda:

• 添加conda 镜像源:

conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/
conda config --set show_channel_urls yes
#-------
conda config --add channels conda-forge   #添加conda 源
#查看所有的 conda .condarc 配置文件内容
conda config --get

• 要充分利用conda 包的好处，遇到需要安装什么包的优选conda：https://anaconda.org/wdroz/openpose-gpu
• 通过源码编译的是在其他已经都编译的没有的状况下，或者需要修改源码
• 设置不进入conda环境，新版的anaconda（python3.7）安装后自动进入虚拟环境base，执行命令：

​ conda config --set auto_activate_base false

conda package

• A conda package is a compressed tarball file (.tar.bz2) or .conda file that contains:

  • system-level libraries.
  • Python or other modules.
  • executable programs and other components.
  • metadata under the info/ directory.
  • a collection of files that are installed directly into an install prefix.

• package structure:

  ├── bin            #relevant binaries for package
  │   └── pyflakes
  ├── info     # package metadata
  │   ├── LICENSE.txt
  │   ├── files
  │   ├── index.json
  │   ├── paths.json
  │   └── recipe
  └── lib       # relevant library files
      └── python3.5
  

  • metadata contain dependencies to system core, low-level libraries and can contain links to software files that are automatically downloaded when executed.

  • index.json

    • metadata about the package including platform, version, dependencies, and build info

  {
    "arch": "x86_64",
    "build": "py37hfa4b5c9_1",
    "build_number": 1,
    "depends": [
      "depend > 1.1.1"
    ],
    "license": "BSD 3-Clause",
    "name": "fun-packge",
    "platform": "linux",
    "subdir": "linux-64",
    "timestamp": 1535416612069,
    "version": "0.0.0"
  }
  

  • paths.json
    • a list of files in the package, along with their associated SHA-256, size in bytes, and the type of path (eg. hardlink vs. softlink)

  {
    "paths": [
      {
        "_path": "lib/python3.7/site-packages/fun-packge/__init__.py",
        "path_type": "hardlink",
        "sha256": "76f3b6e34feeb651aff33ca59e0279c4eadce5a50c6ad93b961c846f7ba717e9",
        "size_in_bytes": 2067
      },
      {
        "_path": "lib/python3.7/site-packages/fun-packge/__config__.py",
        "path_type": "hardlink",
        "sha256": "348e3602616c1fe4c84502b1d8cf97c740d886002c78edab176759610d287f06",
        "size_in_bytes": 87519
      },
      ...
  }
  

Conda Dir

• root_dir:
• pkgs: contains decompressed packages, ready to be linked in conda environments. Each package resides in a subdirectory corresponding to its canonical name.
• envs: conda environments
  • bin
  • include
  • share

.condarc

• Where conda looks for packages.
• how conda uses a proxy server.
• Where conda lists known environments.
• Whether to update the Bash prompt with the currently activated environment name.
• Whether user-built packages should be uploaded to Anaconda.org.
• What default packages or features to include in new environments.

Conda 下载流程

Conda follows these steps when installing a package:

1. Downloading and processing index metadata.
2. Reducing the index.
3. Expressing the package data and constraints as a SAT problem.
4. Running the solver.
5. Downloading and extracting packages.
6. Verifying package contents.
7. Linking packages from package cache into environments.

包首先现在到这个目录下

环境打包

#  1.创建环境的快照或备份
conda create --name snapshot --clone myenv

#  2. 复制环境  相同操作系统
# 生成 spec list 文件
conda list --explicit > spec-list.txt
pip freeze > requirements.txt
# 创建环境
conda create --name myname --file spec-list.txt

#  3. 生成environment.yml 跨平台操作系统复现环境
# 导出environment.yml 文件， 包括 pip 安装软件
conda env export > environment.yml
#创建环境
conda env create -f environment.yml
pip install -r requirements.txt

#  4.Pack 打包，包括该环境中安装的软件包的所有二进制文件， conda-pack 指定平台和操作系统，目标计算机必须集有与源计算机相同的平台和操作系统
# 安装
conda install -c conda-forge conda-pack
# 打包一个环境
# pack environment my_env into my_env.tar.gz
conda pack -n my_env
# pack environment my_env into out_name.tar.gz
conda pack -n my_env -o out_name.tar.gz
# pack environment located at an explicit path into my_env.tar.gz
conda pack -p /explicit/path/to/my_env
# 创建环境
mkdir -p my_env
tar -xzvf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
./my_env/bin/python

# Activate the environment. This adds `my_env/bin` to your path
source my_env/bin/activate

# Run Python from in the environment
(my_env) $ python

# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack

包管理

#方式一
#	文件夹那放到对应python 环境目录下lib->sitepackages然后import 就行。例如 winguiauto.py 文件放在Anaconda3\envs\pytorch-openpose\Lib\site-packages， 直接import winguiauto ，可能需要重启下
#Tips  python 目录(pytorch-openpose) >where python
	#C:\Users\dell\Anaconda3\envs\pytorch-openpose\python.exe
	#C:\Users\dell\Anaconda3\python.exe	#C:\Users\dell\AppData\Local\Microsoft\WindowsApps\python.exe
#方式二： 批量包下载
# 新建 .txt 文件 内容（h5requipy，numpy，tensorflow==2.0.0-alpha0）单独成行，
pip install -r requirement.txt #进行下载  -i https://pypi.tuna.tsinghua.edu.cn/simple/

#方式三： 直接 conda install / pip install 进行下载

#方式四： 离线下载
#	直接下载的whl文件，通过pip install （路径+whl文件名）
#	可以下载到本地 anaconda\install\Lib\site-packages路径下，或者在线下载安装

#查看版本
conda --version
conda update conda
#环境管理
conda create --name pysot python=3.7
pip install -r requirements.txt
conda info --envs        #conda env list
conda activate pysot
conda deactivate pysot
#根据environment.yml 创建环境
conda env create -f environment.yml  #yml 第一行是环境名称
conda remove --name ENVNAME --all
#环境迁移
#包管理
conda search package
conda install package
conda uninstall package --name env
conda install scipy=0.15.0  #下载指定版本
conda list  # list package
conda list -n env  #查看一个环境中所有的包
#删除包
conda remove -n myenv scipy
#install with channel
conda install scipy --channel conda-forge
#本地下载   本地下载不会解决包依赖问题
conda install /packages-path/packages-filename.tar
conda config --set env_prompt '({name})'

查看包依赖

• The tarball is unarchived in the pkgs directory
• Files are hard-linked to the install path
• Shebang lines and other instance of a place-holder prefix are replaced with the install prefix
• The metadata is updated, so that conda knows that it is installed
• the process of command conda create -n py3k python=3.7
• Environments are simple: just link the package to a different directory
• Hand-links are very cheap, and very fast
• Conda environments are completely independent installations of everything
  • No fiddling with pythonPath or symlinking site-package
• Activating an environment just means changing your path so that its bin/ or scripts/ comes first

To find what packages are depending on a specific package in your environment, there is not one specific conda command. It requires a series of steps:

1. List the dependencies that a specific package requires to run: conda info package_name
2. Find your installation’s package cache directory: conda info
3. Find package dependencies. By default, Anaconda/Miniconda stores packages in ~/anaconda/pkgs/ (or ~/opt/pkgs/ on macOS Catalina). Each package has an index.json file which lists the package’s dependencies. This file resides in ~anaconda/pkgs/package_name/info/index.json.
4. Now you can find what packages depend on a specific package. Use grep to search all index.json files as follows: grep package_name ~/anaconda/pkgs/*/info/index.json

包导入原理:

• 怎么加载？——加载方式和用法

  1. import package 读这个包的__init__.py，也就是说导入包的本质是执行包下面的__init__.py文件，执行结束后会包名的目录下生成一个"pycache / init.cpython-36.pyc" 文件
  2. import module 读整个模块的内容
  3. import package1.package2.package3.module4　package读取__init__.py，module读取整个模块内容，按顺序读注意：因为package是读取__init__.py，所以调用的时候必须在__init__.py有引用的东西才能调用，否则会报错。
  4. 从指定路径导入:

  import sys
  sys.path.append('/path/to/folder/containing/my_package')  #注意是要导入包的上级目录
  import my_package
  

• 案例

├── sub_ui.py├── main.py├── test/│   └── test1.py└── model/    └── model1.py#-----  同级目录导入 main.py导入sub_ui.pyimport sub_ui  或from sub_ui import *#-----  子目录，main.py导入model/model1.py需要在model/下建立__init__.py空文件，让编译器认为这是一个模块。    import model.model1或    from model.model1 import *#----   跨目录导入 跨目录，model1.py导入test/test1.py需要在test1/下建立__init__.py空文件import sys sys.path.append("..") import test.test1 或者 import sys sys.path.append("..") import test.test1 import *

• 　从哪里导入:python中，每个.py文件被称之为模块，每个具有__init__.py文件的目录被称为包。只要模块或者包所在的目录在sys.path中，就可以使用import 模块或import 包来使用.对于python来说，所有被加载到内存的模块都是放在sys.modules里面，所以执行import时会首先去该列表中查询是否已添加。如果已经在sys.modules中，那就简单了，只需要将该module的name添加到我们正在调用该module的本地空间中。如果还没有放在sys.modules中，就需要在sys.path所有路径的目录中去按顺序查找该模块的文件，这些文件一般后缀为".py"、".pyo"、".pyc"、".pyd"、".dll"，找到这些模块后就可以将这些模块添加到sys.modules中，再将module name导入到本地。

>>> import sys>>> sys.modules{'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'>, 'sre_compile': <module 'sre_compile' from '/usr/lib/python2.7/sre_compile.pyc'>, '_sre': <module '_sre' (built-in)>, 'encodings': <module 'encodings' from '/usr/lib/python2.7/encodings/__init__.pyc'>, 'site': <module 'site' from '/usr/lib/python2.7/site.pyc'>, '__builtin__': <module '__builtin__' (built-in)>, 'sysconfig': <module 'sysconfig' from '/usr/lib/python2.7/sysconfig.pyc'>, '__main__': <module '__main__' (built-in)>, 'encodings.encodings': None, 'abc': <module 'abc' from '/usr/lib/python2.7/abc.pyc'>, 'posixpath': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_weakrefset': <module '_weakrefset' from '/usr/lib/python2.7/_weakrefset.pyc'>, 'errno': <module 'errno' (built-in)>, 'encodings.codecs': None, 'sre_constants': <module 'sre_constants' from '/usr/lib/python2.7/sre_constants.pyc'>, 're': <module 're' from '/usr/lib/python2.7/re.pyc'>, '_abcoll': <module '_abcoll' from '/usr/lib/python2.7/_abcoll.pyc'>, 'types': <module 'types' from '/usr/lib/python2.7/types.pyc'>, '_codecs': <module '_codecs' (built-in)>, 'encodings.__builtin__': None, '_warnings': <module '_warnings' (built-in)>, 'genericpath': <module 'genericpath' from '/usr/lib/python2.7/genericpath.pyc'>, 'stat': <module 'stat' from '/usr/lib/python2.7/stat.pyc'>, 'zipimport': <module 'zipimport' (built-in)>, '_sysconfigdata': <module '_sysconfigdata' from '/usr/lib/python2.7/_sysconfigdata.pyc'>, 'mpl_toolkits': <module 'mpl_toolkits' (built-in)>, 'warnings': <module 'warnings' from '/usr/lib/python2.7/warnings.pyc'>, 'UserDict': <module 'UserDict' from '/usr/lib/python2.7/UserDict.pyc'>, 'encodings.utf_8': <module 'encodings.utf_8' from '/usr/lib/python2.7/encodings/utf_8.pyc'>, 'sys': <module 'sys' (built-in)>, 'codecs': <module 'codecs' from '/usr/lib/python2.7/codecs.pyc'>, 'readline': <module 'readline' from '/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so'>, '_sysconfigdata_nd': <module '_sysconfigdata_nd' from '/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc'>, 'os.path': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_locale': <module '_locale' (built-in)>, 'sitecustomize': <module 'sitecustomize' from '/usr/lib/python2.7/sitecustomize.pyc'>, 'signal': <module 'signal' (built-in)>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'linecache': <module 'linecache' from '/usr/lib/python2.7/linecache.pyc'>, 'posix': <module 'posix' (built-in)>, 'encodings.aliases': <module 'encodings.aliases' from '/usr/lib/python2.7/encodings/aliases.pyc'>, 'exceptions': <module 'exceptions' (built-in)>, 'sre_parse': <module 'sre_parse' from '/usr/lib/python2.7/sre_parse.pyc'>, 'os': <module 'os' from '/usr/lib/python2.7/os.pyc'>, '_weakref': <module '_weakref' (built-in)>}>>> sys.path['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/ldd/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']

Error record:

• Missing dependencies for SOCKS support. 解决方法： unset all_proxy

linux 安装Anaconda

---

#本地安装pip install ~/Dowloads/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl#conda 安装本地包conda install --use-local pytorch-0.4.0-py35_cuda8.0.61_cudnn7.1.2_1.tar.bz2#change the channelconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --set show_channel_urls yes#下载安装curl -O https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh#通过SHA-256校验和通过加密哈希验证来验证安装程序的数据完整性sha256sum Anaconda3-5.1.0-Linux-x86_64.sh#安装bash Anaconda3-5.1.0-Linux-x86_64.sh#激活环境，否则无法找到conda命令source ~/.bashrcconda info#用来查看conda 内部安装环境版本等信息。#验证安装是否正确 罗列conda 安装所有软件conda list#创建 一个环境my_env  Python版本为3conda create --name my_env python=3   #使用 sudo 命令创建，否则无法在环境中注册#在环境my_env35中安装numpy包 conda install --name my_env35 numpy#激活环境 ，这样会有个前缀#更新 conda使用程序conda update conda#更新 或卸载 anaconda 安装包conda remove -n envname --all   #delete all the package and environmentconda update anaconda conda install anaconda-cleanrm -rf anacondasudo gedit ~/.bashrc #删除环境变量source ~/.bashrc #使环境变量生效

Pycharm ：

#PyCharm：https://www.jetbrains.com/pycharm/download/#section=linuxcd downloads/pycharm-2018.1/binbash pycharm.sh#进入file setting 设置选择编译器

Win环境配置

F:\Anaconda3F:\Anaconda3\ScriptsF:\Anaconda3\Library\bin#shell 中加入这三个目录，才可以在cmd 中使用conda

Jupyter使用

若想要关闭已经打开的终端和“ipynb”格式的笔记本，仅仅关闭其页面是无法彻底退出程序的，需要在Running页面点击其对应的“Shutdown”。

• 安装： conda install jupyter notebook

• 修改默认的主目录：jupyter notebook --generate-config

• 启动目录： jupyter notebook， http://localhost:8888； 可以对端口进行映射处理

  • 指定端口号来启动： jupyter notebook --port <port_number>
  • 启动服务但不打开网页： jupyter notebook --no-browser
  • 在conda 虚拟环境中运行启动 jupyter 命令，kernel 会自动关联所有的conda环境；

• 帮助命令: jupyter notebook --help

• 将你conda创建的环境与Jupyter Notebook相关联： conda install nb_conda

• 加载本地python文件：%load Python文件的绝对路径

• 直接运行本地python文件：%run Python文件的绝对路径； !python3 Python文件的绝对路径； !shell命令；

• 学习链接： https://www.jianshu.com/p/91365f343585#conda

• ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'

  • pip install jupyter_nbextensions_configurator
    

• Error executing Jupyter command 'notebook': [Errno 2] No such file or directory

  • jupyter notebook --port=11220 --allow-root #不适用sudo 权限就可以执行了

• Jupyter 常用快捷键

  ctrl + / #进行注释

  cell操作
  
  Ctrl+enter 执行本cell

shift+enter 执行本cell且 向下建立一个新cell

  A 向上建立一个cell

B 向下建立一个cell

d d 删除cell

O 收起output 或者打开output

  在cell里面的快捷键
  
  Esc+m m 把cell切换至markdown模式
  
  Esc+y y 把cell切换至code模式
  
  Esc+l l 显示行数
  
  Esc + F 在代码中查找、替换，忽略输出。
  
  Esc + O 在 cell 和输出结果间切换。

  复合快捷键
  
  shift 选中多个cell
  
  然后执行 shift+M 合并选中cell

Error Record

.1. 代理问题

#Anaconda 使用conda指令出现ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/ 解决办法printenv | grep all_proxy  #查看代理unset all_proxy && unset ALL_PROXY   #export all_proxy=""  export all_proxy="socks5://127.0.0.1:1080"export http_proxyexport https_proxyunset http_proxyunset https_proxyconda install pysocks     #下次创建环境是同时下载 pip和pysocks 包source ~/.bashrc

每次在虚拟环境内部使用pip的时候都需要获取虚拟环境内部的 pip命令路径：/Users/stefanannihilater/anaconda3/envs/setests3/bin/pip，显然这样非常的繁琐。

如果想在虚拟环境内部使用简单的pip install package调用虚拟环境内部的pip命令的话，只需要我们在创建虚拟环境的时候指定pip只对虚拟环境生效，而不影响全局库：

conda create -n 虚拟环境名 pip pysocks

• pip：是为了指定pip命令只对当前虚拟环境生效
• pysocks：是pip命令的依赖，如果不写，在虚拟环境内使用 pip命令的时候会出现Missing dependencies for SOCKS support.的报错。

下次可以把包下载到本地，然后局域网上传服务器，减少服务器流量开销；

• 错误记录SSL

  • 方式一：环境变量是否配好：

  D:\Anaconda3; D:\Anaconda3\Scripts; D:\Anaconda3\Library\bin
  

  • 方式二： 镜像网址是否可以访问
  • 方式三： 安装openssl 工具：地址

.2. cudatoolkit=11.1. 下载失败

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yesconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/conda install pytorch torchvision torchaudio cudatoolkit=11.1

.3. torchvision 库无法使用

Download error with CIFAR10/Mnist based on torchvision.datasets

• uninstall and reinstall torchvision

conda unstall torchvisionconda install pytorch torchvision torchaudio cudatoolkit=11.1

.4. pip 代理

.5. GPU资源问题

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED；

• GPU被沾满了，没有分配资源；