Best iPhone Data Sync Strategy

Question

I am working on a regular iPhone application that retrieves data from a server (XML, JSON, etc.), and I am wondering what is the best way to implement data synchronization. The criteria are speed (less data exchange over the network), reliability (data recovery in the event of an update failure), offline access and flexibility (adapted when the database structure changes a bit like a new column). I know that it depends on the application and the application, but can you share your strategy / experience?

For me, I am thinking of something like this:

1) Save last modified date to iPhone

2) After starting, send a message, for example getNewData.php? lastModifiedDate = ...

3) The server will process and send back only the changed data from the last time.

4) This data is formatted as follows:

<+><data id="..."></data></+> // add this to SQLite/CoreData

<-><data id="..."></data></-> // remove this

<%><data id="..."><attribute>newValue</attribute></data></%> // new modified value

I don’t want to do <+>, ↔, <%> ... for each attribute, since that would be too complicated, so probably when you get the <%> field, I will just delete the data with the given identifier, and then add it again (it is assumed that here is not some kind of automatically increasing field).

5) After everything is downloaded and updated, I will update the Last Modified Date field.

The main problem with this strategy is: if the network drops, when I update something => The last changed date is not updated yet => the next time I restart the application, I will have to go through the same thing again, not to mention potential inconsistent data. If I use a temporary table to update and make everything atomic, it will work, but then again, if the update is too large (many data changes), the user must wait a long time until new data appears. Should I use Last-Modified-Date for each data field and update the data gradually?

Answer 1

I would start by making the upgrade process atomic, since you will have enough on your hands to figure out how to work correctly with client-server communication.

After this, it's time to think about setting it up incrementally, but only after you do some testing to find out if this is really necessary. If you configure the update protocol as low as possible, you may find that even a “large” update downloads fast enough.

Another way to look at this is to ask yourself, how often does a network problem occur when the average user synchronizes? You probably don't want to tune into unlikely scenarios.

If you are trying to optimize (minimize) data transfer, you may consider a different format than XML, since XML is pretty verbose. Or, at the very least, you can exchange XML readability for space by making each name and attribute of each element as small as possible and eliminate all unnecessary spaces.

Answer 2

Your basic scheme is good. What you need to do is somehow make your updates idempotent so that you can restart a partially completed transfer without risk. This is a better way to go than to try to implement some kind of true atomic commit (although you can do it too, using, for example, the SQLite database).

In our experience, fairly large updates (10 with KB) can be downloaded pretty quickly if the server is fast enough. No need to flip updates to tiny bits. But, of course, it will not hurt to try to minimize the amount of data transferred, while retaining more detailed information about the "latest update".

(And definitely, you should use JSON, not XML, as the transmitted data representation.)

Answer 3

I wonder if you decide to use the Sync Framework to manage synchronization. If this interests you, you can take a look at the open source project, the OpenMobster Sync service. You can perform the following synchronization operations.

• double sided
• one-way client
• one-way device
• Bootup

In addition, all changes are automatically tracked and synchronized with the cloud. You can disconnect your application when the network connection is disconnected. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization, such as iCloud, across multiple devices.

In addition, changes in the cloud are synchronized using push notifications, so the data is always up to date, even if it is stored locally.

In your case

 Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access 

• Speed: Only changes are sent over the network in both directions.

• Reliability: it stores data in a transaction store such as sqlite, and any failed updates are reported in the SyncML payload. Only successful operations are performed, while failed operations are re-checked during the next synchronization.

Here is a link to an open source project: http://openmobster.googlecode.com

Here is a link to the iPhone Sync App: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp