hdfs基本操作-python接口
安装hdfs包
pip install hdfs
查看hdfs目录
[root@hadoop hadoop]# hdfs dfs -ls -R /
drwxr-xr-x - root supergroup 0 2017-05-18 23:57 /Demo
-rw-r--r-- 1 root supergroup 3494 2017-05-18 23:57 /Demo/hadoop-env.sh
drwxr-xr-x - root supergroup 0 2017-05-18 19:01 /logs
-rw-r--r-- 1 root supergroup 2223 2017-05-18 19:01 /logs/anaconda-ks.cfg
-rw-r--r-- 1 root supergroup 57162 2017-05-18 18:32 /logs/install.log
创建hdfs连接实例
#!/usr/bin/env python
# -*- coding:utf-8 -*-
__Author__ = 'kongZhaGen' import hdfs
client = hdfs.Client("http://172.10.236.21:50070")
list:返回远程文件夹包含的文件或目录名称,如果路径不存在则抛出错误。
hdfs_path:远程文件夹的路径
status:同时返回每个文件的状态信息
def list(self, hdfs_path, status=False):
"""Return names of files contained in a remote folder. :param hdfs_path: Remote path to a directory. If `hdfs_path` doesn't exist
or points to a normal file, an :class:`HdfsError` will be raised.
:param status: Also return each file's corresponding FileStatus_. """
示例:
print client.list("/",status=False)
结果:
[u'Demo', u'logs']
status:获取hdfs系统上文件或文件夹的状态信息
hdfs_path:路径名称
strict:
False:如果远程路径不存在返回None
True:如果远程路径不存在抛出异常
def status(self, hdfs_path, strict=True):
"""Get FileStatus_ for a file or folder on HDFS. :param hdfs_path: Remote path.
:param strict: If `False`, return `None` rather than raise an exception if
the path doesn't exist. .. _FileStatus: FS_
.. _FS: http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#FileStatus """
示例:
print client.status(hdfs_path="/Demoo",strict=False)
结果:
None
makedirs:在hdfs上创建目录,可实现递归创建目录
hdfs_path:远程目录名称
permission:为新创建的目录设置权限
def makedirs(self, hdfs_path, permission=None):
"""Create a remote directory, recursively if necessary. :param hdfs_path: Remote path. Intermediate directories will be created
appropriately.
:param permission: Octal permission to set on the newly created directory.
These permissions will only be set on directories that do not already
exist. This function currently has no return value as WebHDFS doesn't return a
meaningful flag. """
示例:
如果想在远程客户端通过脚本给hdfs创建目录,需要修改hdfs-site.xml
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
重启hdfs
stop-dfs.sh
start-dfs.sh
递归创建目录
client.makedirs("/data/rar/tmp",permission=755)
rename:移动一个文件或文件夹
hdfs_src_path:源路径
hdfs_dst_path:目标路径,如果路径存在且是个目录,则源目录移动到此目录中。如果路径存在且是个文件,则会抛出异常
def rename(self, hdfs_src_path, hdfs_dst_path):
"""Move a file or folder. :param hdfs_src_path: Source path.
:param hdfs_dst_path: Destination path. If the path already exists and is
a directory, the source will be moved into it. If the path exists and is
a file, or if a parent destination directory is missing, this method will
raise an :class:`HdfsError`. """
示例:
client.rename("/SRC_DATA","/dest_data")
delete:从hdfs删除一个文件或目录
hdfs_path:hdfs系统上的路径
recursive:如果目录非空,True:可递归删除.False:抛出异常。
def delete(self, hdfs_path, recursive=False):
"""Remove a file or directory from HDFS. :param hdfs_path: HDFS path.
:param recursive: Recursively delete files and directories. By default,
this method will raise an :class:`HdfsError` if trying to delete a
non-empty directory. This function returns `True` if the deletion was successful and `False` if
no file or directory previously existed at `hdfs_path`. """
示例:
client.delete("/dest_data",recursive=True)
upload:上传文件或目录到hdfs文件系统,如果目标目录已经存在,则将文件或目录上传到此目录中,否则新建目录。
def upload(self, hdfs_path, local_path, overwrite=False, n_threads=1,
temp_dir=None, chunk_size=2 ** 16, progress=None, cleanup=True, **kwargs):
"""Upload a file or directory to HDFS. :param hdfs_path: Target HDFS path. If it already exists and is a
directory, files will be uploaded inside.
:param local_path: Local path to file or folder. If a folder, all the files
inside of it will be uploaded (note that this implies that folders empty
of files will not be created remotely).
:param overwrite: Overwrite any existing file or directory.
:param n_threads: Number of threads to use for parallelization. A value of
`0` (or negative) uses as many threads as there are files.
:param temp_dir: Directory under which the files will first be uploaded
when `overwrite=True` and the final remote path already exists. Once the
upload successfully completes, it will be swapped in.
:param chunk_size: Interval in bytes by which the files will be uploaded.
:param progress: Callback function to track progress, called every
`chunk_size` bytes. It will be passed two arguments, the path to the
file being uploaded and the number of bytes transferred so far. On
completion, it will be called once with `-1` as second argument.
:param cleanup: Delete any uploaded files if an error occurs during the
upload.
:param \*\*kwargs: Keyword arguments forwarded to :meth:`write`. On success, this method returns the remote upload path. """
示例:
>>> import hdfs
>>> client=hdfs.Client("http://172.10.236.21:50070")
>>> client.upload("/logs","/root/training/jdk-7u75-linux-i586.tar.gz")
'/logs/jdk-7u75-linux-i586.tar.gz'
>>> client.list("/logs")
[u'anaconda-ks.cfg', u'install.log', u'jdk-7u75-linux-i586.tar.gz']
content:获取hdfs系统上文件或目录的概要信息
print client.content("/logs/install.log")
结果:
{u'spaceConsumed': 57162, u'quota': -1, u'spaceQuota': -1, u'length': 57162, u'directoryCount': 0, u'fileCount': 1}
write:在hdfs文件系统上创建文件,可以是字符串,生成器或文件对象
def write(self, hdfs_path, data=None, overwrite=False, permission=None,
blocksize=None, replication=None, buffersize=None, append=False,
encoding=None):
"""Create a file on HDFS. :param hdfs_path: Path where to create file. The necessary directories will
be created appropriately.
:param data: Contents of file to write. Can be a string, a generator or a
file object. The last two options will allow streaming upload (i.e.
without having to load the entire contents into memory). If `None`, this
method will return a file-like object and should be called using a `with`
block (see below for examples).
:param overwrite: Overwrite any existing file or directory.
:param permission: Octal permission to set on the newly created file.
Leading zeros may be omitted.
:param blocksize: Block size of the file.
:param replication: Number of replications of the file.
:param buffersize: Size of upload buffer.
:param append: Append to a file rather than create a new one.
:param encoding: Encoding used to serialize data written.
"""
hdfs基本操作-python接口的更多相关文章
- caffe的python接口学习(7):绘制loss和accuracy曲线
使用python接口来运行caffe程序,主要的原因是python非常容易可视化.所以不推荐大家在命令行下面运行python程序.如果非要在命令行下面运行,还不如直接用 c++算了. 推荐使用jupy ...
- Windows+Caffe+VS2013+python接口配置过程
前段时间在笔记本上配置了Caffe框架,中间过程曲曲折折,但由于懒没有将详细过程总结下来,这两天又在一台配置较高的台式机上配置了Caffe,配置时便非常后悔当初没有写到博客中去,现已配置好Caffe, ...
- 机器学习caffe环境搭建——redhat7.1和caffe的python接口编译
相信看这篇文章的都知道caffe是干嘛的了,无非就是深度学习.神经网络.计算机视觉.人工智能这些,这个我就不多介绍了,下面说说我的安装过程即遇到的问题,当然还有解决方法. 说下我的环境:1>虚拟 ...
- Caffe + Ubuntu 14.04 64bit + 无CUDA(linux下安装caffe(无cuda)以及python接口)
安装Caffe指导书 环境: Linux 64位 显卡为Intel + AMD,非英伟达显卡 无GPU 一. 安装准备工作 1. 以管理员身份登录 在左上角点击图标,搜索terminal(即终端),以 ...
- caffe的python接口学习(1):生成配置文件
caffe是C++语言写的,可能很多人不太熟悉,因此想用更简单的脚本语言来实现.caffe提供matlab接口和python接口,这两种语言就非常简单,而且非常容易进行可视化,使得学习更加快速,理解更 ...
- Caffe学习系列(11):数据可视化环境(python接口)配置
参考:http://www.cnblogs.com/denny402/p/5088399.html 这节配置python接口遇到了不少坑. 1.我是利用anaconda来配置python环境,在将ca ...
- caffe中python接口的使用
下面是基于我自己的接口,我是用来分类一维数据的,可能不具通用性: (前提,你已经编译了caffe的python的接口) 添加 caffe塻块的搜索路径,当我们import caffe时,可以找到. 对 ...
- Caffe学习系列(13):数据可视化环境(python接口)配置
caffe程序是由c++语言写的,本身是不带数据可视化功能的.只能借助其它的库或接口,如opencv, python或matlab.大部分人使用python接口来进行可视化,因为python出了个比较 ...
- Python接口自动化——soap协议传参的类型是ns0类型的要创建工厂方法纪要
1:在Python接口自动化中,对于soap协议的xml的请求我们可以使用Suds Client来实现,其soap协议传参的类型基本上是有2种: 第一种是传参,不需要再创建啥, 第二种就是ns0类型的 ...
随机推荐
- 【jQuery源码】DOM Ready
一直以来,各种JS最佳实践都会告诉我们,将JS放在HTML的最后,即</body>之前,理由就是:JS会阻塞下载,而且,在JS中很有可能有对DOM的操作,放在HTML的最后,可以尽可能的保 ...
- 虚拟网络编辑器的知识和出现的一些问题(没有VMnet0或VMnet8)
不多说,直接上干货! 安装好的虚拟机,没有VMnet0!!! 解决的办法: 就可以了. 那么,问题现在又来了? 但是,每次都要这样,以右键,管理员身份运行,再关闭之后,以普通双击方式,即又没有VMne ...
- Go 提高性能的特性
1.值的高效处理和存储,允许创建紧凑的数据结构,避免不必要的填充字节.紧凑的数据结构能更好地利用缓存.更好的缓存利用率可带来更好的性能. 2.函数的调用有开销,减少函数调用开销的解决方案是内联.简单的 ...
- ruby文件操作
Ruby代码 1.#读文件 2.f = File.open("myfile.txt", "r") 3.f.each_line do|line| 4.puts & ...
- ruby中nil?, empty? and blank?的选择
In Ruby, you check with nil? if an object is nil: article = nil article.nil? # => true empty? che ...
- java.util.Collection源码分析和深度讲解
写在开头 java.util.Collection 作为Java开发最常用的接口之一,我们经常使用,今天我带大家一起研究一下Collection接口,希望对大家以后的编程以及系统设计能有所帮助,本文所 ...
- Kafka安装kafka-manager
1 .下载kafka-manager 想要查看和管理Kafka,完全使用命令并不方便,我们可以使用雅虎开源的Kafka-manager,GitHub地址如下: https://github.com/y ...
- HA_Snapshots 数据库快照
1. 创建测试数据库HA_Snapshot 2. 创建快照 create database HA_Snapshot_20 on ( name = HA_Snapshot, filename = '.. ...
- Jade——变体的HTML
什么是jade? jade是一个模板引擎,是变体的HTML. 模板引擎就是一个库,用来解释素具渲染视图的框架,也可以叫做HTML的预处理语言. jade是Node.js的一个模板引擎,他的语法借鉴了H ...
- Bootstrap中的datetimepicker插件用法总结(转载)
datetimepicker用法总结 目录 datetimepicker用法总结 目录 简述 官方文档 选项属性 1 format 格式 2 weekStart 一周从哪一天开始 3 startD ...