Read and parse> 400 MB .json file in Julia without kernel failure

Question

The following destroys my core Julia. Is there a better way to read and parse a large (> 400 MB) JSON file?

using JSON data = JSON.parsefile("file.json") 

Answer 1

If no effort is made to create a smarter JSON parser, the following may work: there is a good chance that file.json has many lines. In this case, reading the file and parsing a large repeating JSON section in turns or in pieces (for the right side of the piece) can do the trick. A possible way to encode this:

 using JSON f = open("file.json","r") discard_lines = 12 # lines up to repetitive part important_chunks = 1000 # number of data items chunk_length = 2 # each data item has a 2-line JSON chunk for i=1:discard_lines l = readline(f) end for i=1:important_chunks chunk = join([readline(f) for j=1:chunk_length]) push!(thedata,JSON.parse(chunk)) end close(f) # use thedata 

There is a good chance this may be a temporary solution to your problem. Inspect file.json to find out.