Multiple Jackson objects and huge json files

Question

I get the feeling that the answer may be a duplicate of the word: Jackson is Json for POJOs with multiple entries , but I think the question is potentially quite different. I also use raw data binding, not full data binding.

So, like my question, I have several objects in the file, and I'm trying to turn them into POJO and put them into the database of my project so that I can quickly access the data, and not slowly.

The files here are about tens of GB, with up to a million files in each file. Anyway, here is what I still have:

ObjectMapper mapper = new ObjectMapper(); Map<String,Object> data = mapper.readValue(new File("foo.json"), Map.class); System.out.println(data.get("bar")); 

And this works fine for printing the bar element of the first object in foo, but I need a way to iterate over each element so that it does not eat up all my memory.

Thanks.

Answer 1

Use this sample code to see the main idea.

 final InputStream in = new FileInputStream("json.json"); try { for (Iterator it = new ObjectMapper().readValues( new JsonFactory().createJsonParser(in), Map.class); it.hasNext();) System.out.println(it.next()); } finally { in.close();} } 

Answer 2

You do not need to choose between streaming ( JsonParser ) and ObjectMapper , do both! Walk a little with the parser, but then call JsonParser.readValueAs(MyType.class) to bind a separate JSON object.

Or call the ObjectMapper readValue() method, passing JsonParser at the appropriate points. Or use ObjectMapper.reader(Type.class).readValues() and iterate this way.

Answer 3

Assuming you have an array wrapping your objects, create a JsonParser, and then call readValuesAs with the appropriate type. It returns an Iterator to you with all your objects that read the file as objects are used.