hdfs基本操作-python接口
安装hdfs包
pip install hdfs
查看hdfs目录
[root@hadoop hadoop]# hdfs dfs -ls -R /
drwxr-xr-x - root supergroup 0 2017-05-18 23:57 /Demo
-rw-r--r-- 1 root supergroup 3494 2017-05-18 23:57 /Demo/hadoop-env.sh
drwxr-xr-x - root supergroup 0 2017-05-18 19:01 /logs
-rw-r--r-- 1 root supergroup 2223 2017-05-18 19:01 /logs/anaconda-ks.cfg
-rw-r--r-- 1 root supergroup 57162 2017-05-18 18:32 /logs/install.log
创建hdfs连接实例
#!/usr/bin/env python
# -*- coding:utf-8 -*-
__Author__ = 'kongZhaGen' import hdfs
client = hdfs.Client("http://172.10.236.21:50070")
list:返回远程文件夹包含的文件或目录名称,如果路径不存在则抛出错误。
hdfs_path:远程文件夹的路径
status:同时返回每个文件的状态信息
def list(self, hdfs_path, status=False):
"""Return names of files contained in a remote folder. :param hdfs_path: Remote path to a directory. If `hdfs_path` doesn't exist
or points to a normal file, an :class:`HdfsError` will be raised.
:param status: Also return each file's corresponding FileStatus_. """
示例:
print client.list("/",status=False)
结果:
[u'Demo', u'logs']
status:获取hdfs系统上文件或文件夹的状态信息
hdfs_path:路径名称
strict:
False:如果远程路径不存在返回None
True:如果远程路径不存在抛出异常
def status(self, hdfs_path, strict=True):
"""Get FileStatus_ for a file or folder on HDFS. :param hdfs_path: Remote path.
:param strict: If `False`, return `None` rather than raise an exception if
the path doesn't exist. .. _FileStatus: FS_
.. _FS: http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#FileStatus """
示例:
print client.status(hdfs_path="/Demoo",strict=False)
结果:
None
makedirs:在hdfs上创建目录,可实现递归创建目录
hdfs_path:远程目录名称
permission:为新创建的目录设置权限
def makedirs(self, hdfs_path, permission=None):
"""Create a remote directory, recursively if necessary. :param hdfs_path: Remote path. Intermediate directories will be created
appropriately.
:param permission: Octal permission to set on the newly created directory.
These permissions will only be set on directories that do not already
exist. This function currently has no return value as WebHDFS doesn't return a
meaningful flag. """
示例:
如果想在远程客户端通过脚本给hdfs创建目录,需要修改hdfs-site.xml
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
重启hdfs
stop-dfs.sh
start-dfs.sh
递归创建目录
client.makedirs("/data/rar/tmp",permission=755)
rename:移动一个文件或文件夹
hdfs_src_path:源路径
hdfs_dst_path:目标路径,如果路径存在且是个目录,则源目录移动到此目录中。如果路径存在且是个文件,则会抛出异常
def rename(self, hdfs_src_path, hdfs_dst_path):
"""Move a file or folder. :param hdfs_src_path: Source path.
:param hdfs_dst_path: Destination path. If the path already exists and is
a directory, the source will be moved into it. If the path exists and is
a file, or if a parent destination directory is missing, this method will
raise an :class:`HdfsError`. """
示例:
client.rename("/SRC_DATA","/dest_data")
delete:从hdfs删除一个文件或目录
hdfs_path:hdfs系统上的路径
recursive:如果目录非空,True:可递归删除.False:抛出异常。
def delete(self, hdfs_path, recursive=False):
"""Remove a file or directory from HDFS. :param hdfs_path: HDFS path.
:param recursive: Recursively delete files and directories. By default,
this method will raise an :class:`HdfsError` if trying to delete a
non-empty directory. This function returns `True` if the deletion was successful and `False` if
no file or directory previously existed at `hdfs_path`. """
示例:
client.delete("/dest_data",recursive=True)
upload:上传文件或目录到hdfs文件系统,如果目标目录已经存在,则将文件或目录上传到此目录中,否则新建目录。
def upload(self, hdfs_path, local_path, overwrite=False, n_threads=1,
temp_dir=None, chunk_size=2 ** 16, progress=None, cleanup=True, **kwargs):
"""Upload a file or directory to HDFS. :param hdfs_path: Target HDFS path. If it already exists and is a
directory, files will be uploaded inside.
:param local_path: Local path to file or folder. If a folder, all the files
inside of it will be uploaded (note that this implies that folders empty
of files will not be created remotely).
:param overwrite: Overwrite any existing file or directory.
:param n_threads: Number of threads to use for parallelization. A value of
`0` (or negative) uses as many threads as there are files.
:param temp_dir: Directory under which the files will first be uploaded
when `overwrite=True` and the final remote path already exists. Once the
upload successfully completes, it will be swapped in.
:param chunk_size: Interval in bytes by which the files will be uploaded.
:param progress: Callback function to track progress, called every
`chunk_size` bytes. It will be passed two arguments, the path to the
file being uploaded and the number of bytes transferred so far. On
completion, it will be called once with `-1` as second argument.
:param cleanup: Delete any uploaded files if an error occurs during the
upload.
:param \*\*kwargs: Keyword arguments forwarded to :meth:`write`. On success, this method returns the remote upload path. """
示例:
>>> import hdfs
>>> client=hdfs.Client("http://172.10.236.21:50070")
>>> client.upload("/logs","/root/training/jdk-7u75-linux-i586.tar.gz")
'/logs/jdk-7u75-linux-i586.tar.gz'
>>> client.list("/logs")
[u'anaconda-ks.cfg', u'install.log', u'jdk-7u75-linux-i586.tar.gz']
content:获取hdfs系统上文件或目录的概要信息
print client.content("/logs/install.log")
结果:
{u'spaceConsumed': 57162, u'quota': -1, u'spaceQuota': -1, u'length': 57162, u'directoryCount': 0, u'fileCount': 1}
write:在hdfs文件系统上创建文件,可以是字符串,生成器或文件对象
def write(self, hdfs_path, data=None, overwrite=False, permission=None,
blocksize=None, replication=None, buffersize=None, append=False,
encoding=None):
"""Create a file on HDFS. :param hdfs_path: Path where to create file. The necessary directories will
be created appropriately.
:param data: Contents of file to write. Can be a string, a generator or a
file object. The last two options will allow streaming upload (i.e.
without having to load the entire contents into memory). If `None`, this
method will return a file-like object and should be called using a `with`
block (see below for examples).
:param overwrite: Overwrite any existing file or directory.
:param permission: Octal permission to set on the newly created file.
Leading zeros may be omitted.
:param blocksize: Block size of the file.
:param replication: Number of replications of the file.
:param buffersize: Size of upload buffer.
:param append: Append to a file rather than create a new one.
:param encoding: Encoding used to serialize data written.
"""
hdfs基本操作-python接口的更多相关文章
- caffe的python接口学习(7):绘制loss和accuracy曲线
使用python接口来运行caffe程序,主要的原因是python非常容易可视化.所以不推荐大家在命令行下面运行python程序.如果非要在命令行下面运行,还不如直接用 c++算了. 推荐使用jupy ...
- Windows+Caffe+VS2013+python接口配置过程
前段时间在笔记本上配置了Caffe框架,中间过程曲曲折折,但由于懒没有将详细过程总结下来,这两天又在一台配置较高的台式机上配置了Caffe,配置时便非常后悔当初没有写到博客中去,现已配置好Caffe, ...
- 机器学习caffe环境搭建——redhat7.1和caffe的python接口编译
相信看这篇文章的都知道caffe是干嘛的了,无非就是深度学习.神经网络.计算机视觉.人工智能这些,这个我就不多介绍了,下面说说我的安装过程即遇到的问题,当然还有解决方法. 说下我的环境:1>虚拟 ...
- Caffe + Ubuntu 14.04 64bit + 无CUDA(linux下安装caffe(无cuda)以及python接口)
安装Caffe指导书 环境: Linux 64位 显卡为Intel + AMD,非英伟达显卡 无GPU 一. 安装准备工作 1. 以管理员身份登录 在左上角点击图标,搜索terminal(即终端),以 ...
- caffe的python接口学习(1):生成配置文件
caffe是C++语言写的,可能很多人不太熟悉,因此想用更简单的脚本语言来实现.caffe提供matlab接口和python接口,这两种语言就非常简单,而且非常容易进行可视化,使得学习更加快速,理解更 ...
- Caffe学习系列(11):数据可视化环境(python接口)配置
参考:http://www.cnblogs.com/denny402/p/5088399.html 这节配置python接口遇到了不少坑. 1.我是利用anaconda来配置python环境,在将ca ...
- caffe中python接口的使用
下面是基于我自己的接口,我是用来分类一维数据的,可能不具通用性: (前提,你已经编译了caffe的python的接口) 添加 caffe塻块的搜索路径,当我们import caffe时,可以找到. 对 ...
- Caffe学习系列(13):数据可视化环境(python接口)配置
caffe程序是由c++语言写的,本身是不带数据可视化功能的.只能借助其它的库或接口,如opencv, python或matlab.大部分人使用python接口来进行可视化,因为python出了个比较 ...
- Python接口自动化——soap协议传参的类型是ns0类型的要创建工厂方法纪要
1:在Python接口自动化中,对于soap协议的xml的请求我们可以使用Suds Client来实现,其soap协议传参的类型基本上是有2种: 第一种是传参,不需要再创建啥, 第二种就是ns0类型的 ...
随机推荐
- apache URL重写 标志表 以及 错误解决方法
Apache mod_rewrite规则重写的标志一览 1) R[=code](force redirect) 强制外部重定向 强制在替代字符串加上http://thishost[:thisport] ...
- 在CentOS上装Redis
Redis官网 $ wget http://download.redis.io/releases/redis-3.2.5.tar.gz $ tar xzf redis-.tar.gz $ cd red ...
- css 实现元素长宽等比缩放
实现思路(长宽比2:1): 以父级元素为基准, 子级 width:100%; (也就是父级宽度的 100%), padding-top:50% (也就是父级宽度的 50%, 根据 css 规范, pa ...
- C/C++ -- Gui编程 -- Qt库的使用 -- 信号与槽的关联
Qt信号与槽的三种关联方法:1.设计界面关联,编辑信号/槽,自动关联 2.手动关联(1).头文件中定义槽 -----mywidget.h----- #ifndef MYWIDGET_H #define ...
- golang笔记(1)-数据库查询结果映射至结构体
通用的映射模式 query:="select id,name from user where id=?" //单个结构体ret:=&Activity{} DbClient( ...
- elasticsearch(一) 之 elasticsearch初识
目录 一 .elasticsearch 二 . elasticsearch 名词解释 集群(cluster) 节点(node) 索引(index) type(类型) Document(文档) shar ...
- 发布Framework 4.0到iis时,出现HTTP 错误 403.14 - Forbidden
新发布MVC到服务器的时候,经常碰到403.14错误,绝大部分的时候都是因为Framework 4.0需要重新注册下,在运行里输入:C:\Windows\Microsoft.NET\Framework ...
- 单点登录(SSO)
单点登录SSO(SingleSign-On)是身份管理中的一部分.SSO的一种较为通俗的定义是:SSO是指访问同一服务器不同应用中的受保护资源的同一用户,只需要登录一次,即通过一个应用中的安全验证后, ...
- Fiddler——PC上实现手机的抓包(转载 http://www.jianshu.com/p/13f8a81d7c7c)
Fiddler是15年初,在千牛中做超级促销插件时,发现没有root的Android机和没有越狱的iPhone无法修改host,因此没办法测试.为了让我这个磨人的PD也能看到,开发推荐了Fiddler ...
- SpringBoot 初接触之 404
1. 确认输入项目路径是否正确,如:http://localhost:8080/index,这里需要注意的是端口号的查看 2. 确认注解是否用对 在 Controller 层类上面使用的注解是 @Re ...