To increase the speed of updating existing records (~ 11 000) in the base data

Question

I am parsing a ton of data that I originally inserted into the master data warehouse.

In the future, I will parse the same XML, although some of them may have been updated. What I do then is check the existing record with the same tag, and if it already exists, I update the record with data.

However, although my initial parsing (about 11,000 entries) takes 8 seconds or so, the update seems expensive and takes 144 seconds (this simulator works, therefore, much more on real devices).

While the first time is normal (I show the progress bar), the second is unacceptably long, and I would like to do something to improve speed (even if this happens in the background in a separate thread).

Unfortunately, this is not a find-or-create question, since the data in XML can be modified for individual records, so each one may essentially require an update.

I have indexed attributes that speed up the initial parsing and updating, but they are still slow (the numbers above are indexed). I noticed that parsing / updating seems to be slowly slowing down. Although initially fast, it becomes slower and slower as more and more records are processed.

So finally, my question is, do I have any suggestions on how I can improve the speed with which I update my dataset? I am using MagicalRecord to retrieve a record. Here is the code:

Record *record; if (!isUpdate) { record = [NSEntityDescription insertNewObjectForEntityForName:@"Record" inManagedObjectContext:backgroundContext]; } else { NSPredicate *recordPredicate = [NSPredicate predicateWithFormat:@"SELF.tag == %@", [[node attributeForName:@"tag"] stringValue]]; record = [Record findFirstWithPredicate:recordPredicate]; } 

Answer 1

Instead of performing many selections, execute one query for each entity type and save them in the dictionary by tag, and then just check the dictionary if there is an object with this key. You should be able to set the ToFetch property to just add a tag, and this should reduce overhead.

Answer 2

One thing you could try is to use the NSPredicate pattern so that you don't re-parse the format string for every search / fetch you do.

So, before you enter the loop:

 NSPredicate *template = [NSPredicate predicateWithFormat:@"SELF.tag == $RECORD_TAG"]; 

inside the loop:

 Record *record; if (!isUpdate) { record = [NSEntityDescription insertNewObjectForEntityForName:@"Record" inManagedObjectContext:backgroundContext]; } else { NSPredicate *recordPredicate = []; record = [Record findFirstWithPredicate:[template predicateWithSubstitutionVariables:[NSDictionary dictionaryWithObject:[[node attributeForName:@"tag"] stringValue] forKey:@"RECORD_TAG"]]; } 

See the Apple Predicate Programming Guide for more information.

Answer 3

You can also try combining the older answer with property hashing.

Insert the hash properties and save this hash as a kind of Record checksum property.
When updating, you set the selected properties as a tag and checksum and select all the elements. Then, when you iterate over your dataset, if the checksum is different from the one that was selected, you can get this Record and update it.

Answer 4

The initial answer to ALL performance issues is to run tools. Then, using this data, you can identify your problem areas. From there, you may have other, specific questions about some aspects of improving productivity.

We humans do not know much about performance bottlenecks. So, first use the tools. This will certainly tell you where your time is being spent.