Geek Answers Handbook

How to read only part of a list of strings in python

I need to find a way to read x bytes of data from a list containing strings. Each item in the list is ~ 36 MB. I need to be able to run each item in the list, but only grab about ~ 1 Kbyte of that item at a time.

Essentially, it looks like this:

for item in list:
    #grab part of item
    #do something with that part
    #Move onto next part, until you've gone through the whole item

My current code (what kind of work, but seems to be rather slow and inefficient) is this:

for character in bucket:
    print character
    packet = "".join(character)
    if(len(packet.encode("utf8")) >= packetSizeBytes):
        print "Bytes: " + str(len(packet.encode("utf8")))
        return packet

I am wondering if there is something like f.read(bufSize), but for strings.

Not sure if this is relevant, but for a larger context, this is what I am doing:

( ) ( ). f.read(chunkSize) buckets. - , , .

, . , , , . , , , - , . - , , .

+4
2

str ( byte python 3), , f.read(5) f[:5]. 5 ,

[s[:5] for s in buckets]

, . , , , , , .

import threading

def worker(chunk):
    # do stuff with chunk
    ...

def main():
    with open('file', 'r') as f:
        bucket = f.read(500)
        while bucket:
            chunk = bucket[:5]
            thread = threading.Thread(target=worker, args=(chunk,))
            thread.start()
            bucket = f.read(500)
+2

, .

l = [] # Your list
x = 0
processed = 0
while processed!=len(l):
    bts = l[x][:1024]
    l[x] = l[x][1024:]
    # Do something with bts
    if not l[x]: processed += 1
    x += 1
    if x==len(l): x = 0

, . , .

0

More articles:


All Articles