Reading a large JSON file into a variable in C # .net

Question

I am trying to parse JSON files and embed them in SQL DB. My parser worked fine while the files were small (less than 5 MB).

I get a "memory exception" when I try to read large (> 5 MB) files.

if (System.IO.Directory.Exists(jsonFilePath)) { string[] files = System.IO.Directory.GetFiles(jsonFilePath); foreach (string s in files) { var jsonString = File.ReadAllText(s); fileName = System.IO.Path.GetFileName(s); ParseJSON(jsonString, fileName); } } 

I tried the JSONReader approach, but no luck getting the whole JSON into a string or variable. Please advise.

Answer 1

Use 64 bit, check RredCat answer on a similar question:

Newtonsoft.Json - memory exception when deserializing a large object

NewtonSoft Jason Performance Tips

Read David Cox's article on tokenization:

"The main approach is to use the JsonTextReader object, which is part of the Json.NET library. JsonTextReader reads the JSON file one token at a time, so it avoids the overhead of reading the entire file in the String. Since markers are read from the file, objects are created and inserted onto the stack and exit. When the end of the file is reached, the top of the stack contains one object - the top of a very large tree of objects corresponding to the objects in the source JSON file "

Parsing large records using Json.NET

Answer 2

The json file is too large to be inserted into memory in any form.

You should use a JSON reader that takes a file name or stream as input. From your question, which you use JSON Reader is unclear. What library?

If your JSON reader builds an entire JSON tree, you will still have a shortage of memory. When you read the JSON file, either cherry picks the data you are looking for, or writes the data structures to another format on disk, which can be easily queried, for example, the sqlite database.