Geek Answers Handbook

Using python ijson to read a large json file with multiple json objects

I am trying to parse a large (~ 100 MB) json file using the ijson package, which allows me to interact effectively with the file. However, after writing such code,

with open(filename, 'r') as f:
    parser = ijson.parse(f)
    for prefix, event, value in parser:
        if prefix == "name":
            print(value)

I found that the code parses only the first line, not the rest of the lines from the file !!

Here is what part of my json file looks like:

{"name":"accelerator_pedal_position","value":0,"timestamp":1364323939.012000}
{"name":"engine_speed","value":772,"timestamp":1364323939.027000}
{"name":"vehicle_speed","value":0,"timestamp":1364323939.029000}
{"name":"accelerator_pedal_position","value":0,"timestamp":1364323939.035000}

In my opinion, I think that ijsonparses only one json object.

Can anyone suggest how to get around this?

+4
source share
2 answers

Since the provided fragment is more like a set of strings, each of which consists of independent JSON, it should be analyzed accordingly:

# each JSON is small, there no need in iterative processing
import json 
with open(filename, 'r') as f:
    for line in f:
        data = json.loads(line)
        # data[u'name'], data[u'engine_speed'], data[u'timestamp'] now
        # contain correspoding values
+3

, ijson (v2.3 2018 ) JSON. 1 , , : "ijson.common.JSONError: Additional data". . :

. , ( ) JSON, , :

import io
import ijson

with open(filename, encoding="UTF-8") as json_file:
    cursor = 0
    for line_number, line in enumerate(json_file):
        print ("Processing line", line_number + 1,"at cursor index:", cursor)
        line_as_file = io.StringIO(line)
        # Use a new parser for each line
        json_parser = ijson.parse(line_as_file)
        for prefix, type, value in json_parser:
            print ("prefix=",prefix, "type=",type, "value=",value)
        cursor += len(line)

- , , JSON. : ? enumerate() : Python 'for'

+2

More articles:


All Articles