Python json parser

Question

I have a JSON file that has the following structure: -

{ "contributors": null, "truncated": false, "text": "@HomeShop18 #DreamJob to professional rafter", "in_reply_to_status_id": null, "id": 421584490452893696, "favorite_count": 0, "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Mobile Web (M2)</a>", "retweeted": false, "coordinates": null, "entities": { "symbols": [], "user_mentions": [ { "id": 183093247, "indices": [ 0, 11 ], "id_str": "183093247", "screen_name": "HomeShop18", "name": "HomeShop18" } ], "hashtags": [ { "indices": [ 12, 21 ], "text": "DreamJob" } ], "urls": [] }, "in_reply_to_screen_name": "HomeShop18", "id_str": "421584490452893696", "retweet_count": 0, "in_reply_to_user_id": 183093247, "favorited": false, "user": { "follow_request_sent": null, "profile_use_background_image": true, "default_profile_image": false, "id": 2254546045, "verified": false, "profile_image_url_https": "https://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg", "profile_sidebar_fill_color": "171106", "profile_text_color": "8A7302", "followers_count": 87, "profile_sidebar_border_color": "BCB302", "id_str": "2254546045", "profile_background_color": "0F0A02", "listed_count": 1, "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png", "utc_offset": null, "statuses_count": 9793, "description": "Rafter. Rafting is what I do. Me aur mera Tablet. Technocrat of Future", "friends_count": 231, "location": "", "profile_link_color": "473623", "profile_image_url": "http://pbs.twimg.com/profile_images/413952088880594944/rcdr59OY_normal.jpeg", "following": null, "geo_enabled": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/2254546045/1388065343", "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png", "name": "Jayy", "lang": "en", "profile_background_tile": false, "favourites_count": 41, "screen_name": "JzayyPsingh", "notifications": null, "url": null, "created_at": "Fri Dec 20 05:46:00 +0000 2013", "contributors_enabled": false, "time_zone": null, "protected": false, "default_profile": false, "is_translator": false }, "geo": null, "in_reply_to_user_id_str": "183093247", "lang": "en", "created_at": "Fri Jan 10 10:09:09 +0000 2014", "filter_level": "medium", "in_reply_to_status_id_str": null, "place": null } 

There are almost 1,500 such dictionaries in this file. I want to know if there is any existing parser in python for such a file, for lazy parsing. I want the specified analyzer to return only one dictionary at a time, or, in the worst case, return line by line. What should I do?

Answer 1

The json.JSONDecoder.raw_decode workaround is used here . I want someone to suggest a better way to deal with this problem.

 import json import re nonspace = re.compile(r'\S') def iterparse(j): decoder = json.JSONDecoder() pos = 0 while True: matched = nonspace.search(j, pos) if not matched: break pos = matched.start() decoded, pos = decoder.raw_decode(j, pos) yield decoded 

Usage example:

 >>> j = ''' ... { "id": 1 } ... { "id": 2 } ... ''' >>> list(iterparse(j)) [{u'id': 1}, {u'id': 2}] 

Answer 2

You can use raw_decode from json.JSONDecoder to do this without reading the entire file in memory:

raw_decode Decode the JSON document from s (str starting with the JSON document) and return the 2-tuple of the Python view and the index at s where the document ended.

This can be used to decode a JSON document from a string that may have extraneous data at the end.

 import json def iterparse(file_obj): decoder = json.JSONDecoder() buf = "" for line in file_obj: buf += line.strip() try: res = decoder.raw_decode(buf) buf = "" yield res[0] except ValueError: pass with open("stuff.json") as f: for obj in iterparse(f): print obj 

Answer 3

Try the ijson parser:

Usage example:

 import ijson parser = ijson.parse(urlopen('url/')) stream.write('<geo>') for prefix, event, value in parser: if (prefix, event) == ('earth', 'map_key'): stream.write('<%s>' % value) continent = value elif prefix.endswith('.name'): stream.write('<object name="%s"/>' % value) elif (prefix, event) == ('earth.%s' % continent, 'end_map'): stream.write('</%s>' % continent) stream.write('</geo>')