python chunk 方式读取大文件——本质上还是file read自身支持
参考:https://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python
最优雅方式:
file.readlines() takes in an optional size argument which approximates the number of lines read in the lines returned.
bigfile = open('bigfilename','r')
tmp_lines = bigfile.readlines(BUF_SIZE)
while tmp_lines:
process([line for line in tmp_lines])
tmp_lines = bigfile.readlines(BUF_SIZE)
或者:
To write a lazy function, just use yield
:
def read_in_chunks(file_object, chunk_size=1024):
"""Lazy function (generator) to read a file piece by piece.
Default chunk size: 1k."""
while True:
data = file_object.read(chunk_size)
if not data:
break
yield data
f = open('really_big_file.dat')
for piece in read_in_chunks(f):
process_data(piece)
Read a file in chunks in Python
This article is just to demonstrate how to read a file in chunks rather than all at once.
This is useful for a number of cases, such as chunked uploading or encryption purposes, or perhaps where the file you want to interact with is larger than your machine memory capacity.
# chunked file reading
from __future__ import division
import os def get_chunks(file_size):
chunk_start = 0
chunk_size = 0x20000 # 131072 bytes, default max ssl buffer size
while chunk_start + chunk_size < file_size:
yield(chunk_start, chunk_size)
chunk_start += chunk_size final_chunk_size = file_size - chunk_start
yield(chunk_start, final_chunk_size) def read_file_chunked(file_path):
with open(file_path) as file_:
file_size = os.path.getsize(file_path) print('File size: {}'.format(file_size)) progress = 0 for chunk_start, chunk_size in get_chunks(file_size): file_chunk = file_.read(chunk_size) # do something with the chunk, encrypt it, write to another file... progress += len(file_chunk)
print('{0} of {1} bytes read ({2}%)'.format(
progress, file_size, int(progress / file_size * 100))
) if __name__ == '__main__':
read_file_chunked('some-file.gif')
Also available as a Gist (https://gist.github.com/richardasaurus/21d4b970a202d2fffa9c)
The above will output:
File size: 698837
131072 of 698837 bytes read (18%)
262144 of 698837 bytes read (37%)
393216 of 698837 bytes read (56%)
524288 of 698837 bytes read (75%)
655360 of 698837 bytes read (93%)
698837 of 698837 bytes read (100%)
Hopefully handy to someone. This of course isn’t the only way, you could also use `file.seek` in the standard library to target chunks.
Processing large files using python
In the last year or so, and with my increased focus on ribo-seq data, I have come to fully appreciate what the term big data means. The ribo-seq studies in their raw forms can easily reach into hundreds of GBs, which means that processing them in both a timely and efficient manner requires some thought. In this blog post, and hopefully those following, I want to detail some of the methods I have come up (read: pieced together from multiple stack exchange posts), that help me take on data of this magnitude. Specifically I will be detailing methods for python and R, though some of the methods are transferrable to other languages.
My first big data tip for python is learning how to break your files into smaller units (or chunks) in a manner that you can make use of multiple processors. Let’s start with the simplest way to read a file in python.
with open("input.txt") as f:
data = f.readlines()
for line in data:
process(line)
This mistake made above, with regards to big data, is that it reads all the data into RAM before attempting to process it line by line. This is likely the simplest way to cause the memory to overflow and an error raised. Let’s fix this by reading the data in line by line, so that only a single line is stored in the RAM at any given time.
with open("input.txt") as f:
for line in f:
process(line)
This is a big improvement, namely it doesn’t crash when fed a big file (though also it’s shorter!). Next we should attempt to speed this up a bit by making use of all these otherwise idle cores.
import multiprocessing as mp
#init objects
pool = mp.Pool(cores)
jobs = []
#create jobs
with open("input.txt") as f:
for line in f:
jobs.append( pool.apply_async(process,(line)) )
#wait for all jobs to finish
for job in jobs:
job.get()
#clean up
pool.close()
Provided the order of which you process the lines don’t matter, the above generates a set (pool) of workers, ideally one for each core, before creating a bunch of tasks (jobs), one for each line, for the workers to do. I tend to use the Pool object provided by the multiprocessing module due to ease of use, however, you can spawn and control individual workers using mp.Process if you want finer control. For mere number crunching, the Pool object is very good.
While the above is now making use of all those cores, it sadly runs into memory problems once again. We specifically use apply_async function so that the pool isn’t blocked while each line processes. However, in doing so, all the data is read into memory once again; this time stored as individual lines associated with each job, waiting inline to be processed. As such, the memory will again overflow. Ideally the method will only read the line into memory when it is its turn to be processed.
import multiprocessing as mp
def process_wrapper(lineID):
with open("input.txt") as f:
for i,line in enumerate(f):
if i != lineID:
continue
else:
process(line)
break
#init objects
pool = mp.Pool(cores)
jobs = []
#create jobs
with open("input.txt") as f:
for ID,line in enumerate(f):
jobs.append( pool.apply_async(process_wrapper,(ID)) )
#wait for all jobs to finish
for job in jobs:
job.get()
#clean up
pool.close()
Above we’ve now changed the function fed to pool of workers to include opening the file, locating the specified line, reading it into memory, and then processing it. The only input now stored for each job spawned is the line number, thereby preventing the memory overflow. Sadly, the overhead involved in having to locate the line by reading iteratively through the file for each job is untenable, getting progressively more time consuming as you get further into the file. To avoid this we can use the seek function of file objects which skips you to a particular location within a file. Combining with the tell function, which returns the current location within a file, gives:
import multiprocessing as mp
def process_wrapper(lineByte):
with open("input.txt") as f:
f.seek(lineByte)
line = f.readline()
process(line)
#init objects
pool = mp.Pool(cores)
jobs = []
#create jobs
with open("input.txt") as f:
nextLineByte = f.tell()
for line in f:
jobs.append( pool.apply_async(process_wrapper,(nextLineByte)) )
nextLineByte = f.tell()
#wait for all jobs to finish
for job in jobs:
job.get()
#clean up
pool.close()
Using seek we can move directly to the correct part of the file, whereupon we read a line into the memory and process it. We have to be careful to correctly handle the first and last lines, but otherwise this does exactly what we set out, namely using all the cores to process a given file while not overflowing the memory.
I’ll finish this post with a slight upgrade to the above as there is a reasonable amount of overhead associated with opening and closing the file for each individual line. If we process multiple lines of the file at a time as a chunk, we can reduce these operations. The biggest technicality when doing this is noting that when you jump to a location in a file, you are likely not located at the start of a line. For a simple file, as in this example, this just means you need to call readline, which reads to next newline character. More complex file types likely require additional code to locate a suitable location to start/end a chunk.
import multiprocessing as mp,os
def process_wrapper(chunkStart, chunkSize):
with open("input.txt") as f:
f.seek(chunkStart)
lines = f.read(chunkSize).splitlines()
for line in lines:
process(line)
def chunkify(fname,size=1024*1024):
fileEnd = os.path.getsize(fname)
with open(fname,'r') as f:
chunkEnd = f.tell()
while True:
chunkStart = chunkEnd
f.seek(size,1)
f.readline()
chunkEnd = f.tell()
yield chunkStart, chunkEnd - chunkStart
if chunkEnd > fileEnd:
break
#init objects
pool = mp.Pool(cores)
jobs = []
#create jobs
for chunkStart,chunkSize in chunkify("input.txt"):
jobs.append( pool.apply_async(process_wrapper,(chunkStart,chunkSize)) )
#wait for all jobs to finish
for job in jobs:
job.get()
#clean up
pool.close()
Anyway, I hope that some of the above was either new or even perhaps helpful to you. If you know of a better way to do things (in python), then I’d be very interested to hear about it. In another post coming in the near future, I will expanded on this code, turning it into a parent class from which create multiple children to use with various file types.
python chunk 方式读取大文件——本质上还是file read自身支持的更多相关文章
- 在python中逐行读取大文件
在我们日常工作中,难免会有处理日志文件的时候,当文件小的时候,基本不用当心什么,直接用file.read()或readlines()就可以了,但是如果是将一个10G大小的日志文件读取,即文件大于内存的 ...
- Python之requests模块-大文件分片上传
最近在做接口测试时,拿到一个分片上传文件的接口,http接口请求头中的Content-Type为multipart/form-data.需要在客户端将大文件分片成数据块后,依次传给服务端,由服务端还原 ...
- python读取大文件只能读取部分的问题
最近准备重新研究一下推荐系统的东西,用到的数据集是Audioscrobbler音乐数据集.我用python处理数据集中artist_data.txt这个文件的时候,先读取每一行然后进行处理: with ...
- Python读取大文件的"坑“与内存占用检测
python读写文件的api都很简单,一不留神就容易踩"坑".笔者记录一次踩坑历程,并且给了一些总结,希望到大家在使用python的过程之中,能够避免一些可能产生隐患的代码. 1. ...
- Python逐块读取大文件行数的代码 - 为程序员服务
Python逐块读取大文件行数的代码 - 为程序员服务 python数文件行数最简单的方法是使用enumerate方法,但是如果文件很大的话,这个方法就有点慢了,我们可以逐块的读取文件的内容,然后按块 ...
- Python读取大文件(GB)
Python读取大文件(GB) - CSDN博客 https://blog.csdn.net/shudaqi2010/article/details/54017766
- 强悍的Python读取大文件的解决方案
这是一道著名的 Python 面试题,考察的问题是,Python 读取大文件和一般规模的文件时的区别,也即哪些接口不适合读取大文件. 1. read() 接口的问题 f =open(filename, ...
- PHP 与Python 读取大文件的区别
php读取大文件的方法 <?php function readFile($file) { # 打开文件 $handle = fopen($file, 'rb'); while (feof($ ...
- formdata方式上传文件,支持大文件分割上传
1.upload.html <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/html"> <h ...
随机推荐
- JS——if条件判断
现在只说特殊情况: 1.一个变量,例如n1=null <script> var n1 = null; alert(n1);/*弹窗的值为null*/ if (n1 == null) {/* ...
- java 操作clob
之前在学校的时候做的都是练习,小儿科,遇到的情况完全都在自己的设想范围内.最近老是遇到字段溢出的情况,但是varchar2好像最长也只有4000个字符.所以不得不另辟蹊径,就找上了clob字段. pa ...
- css特殊效果
border-radius实现特殊形状 .box{ width: 100px; height: 100px; background: orange; border: 1px solid #000; b ...
- Python操作数据库及hashlib模块
一.hashlib模块 hashlib模块,主要用于加密相关的操作,在python3的版本里,代替了md5和sha模块,主要提供 SHA1, SHA224, SHA256, SHA384, SHA51 ...
- 关于WEB开发下面DIV层被OCX控件拦住问题
控件分为有窗口控件与无窗口控件,无窗口控件很好办,如flash控件,可以通过添加wmode属性来解决挡住DIV层这个问题,添加的代码如下: 解决无窗口控件挡住DIV: 1 <param nam ...
- Spring MVC起步(一)
下图展示了请求使用Spring MVC所经历的所有站点. 在请求离开浏览器时1,会带有用户请求内容的信息,至少会包含请求的URL.但是还可能包含其他的信息,如用户提交的表单. DispatcherSe ...
- 零基础学习Python培训,应该选择哪个培训班?
近几年中,Python一直是市场上最受欢迎的编程语言之一.它语法自然,入门简单,同时应用范围又极广,无论是大火的人工智能.大数据还是传统的web开发.自动化运维,Python都能够大展拳脚.根据职友集 ...
- Django - 创建多对多及增加示例
创建多对多: 方式一: 自定义关系表 备注:自定义表Host.Application,通过自定义表,将表Host和Application进行关联(通过外键方式工): 执行语句:python manag ...
- 文本框、评论框原生js
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8 ...
- (C/C++学习)12.获取系统时间制作时钟(system()略解)
说明:通过调用函数来获取系统当前时间,并制作一个数字式的时钟,时钟的显示包括年.月.日.小时.分以及秒,通过系统屏幕的刷新来对不断更新的时间进行屏幕的显示. 一.对相关函数的学习 1.time_t t ...