What is the current programming standard for synchronizing data between a web service and a client?

Question

The question is a little general, therefore, to narrow the focus, I will share my current setting, which motivates this question. I have a LAMP web service using the RESTful API. We have two client implementations: one browser based on a javascript client (local storage) and one iOS-based client (master data storage). Obviously, these two clients store data in different ways, but the data itself should be stored in two-way synchronization with the remote server as often as possible.

Currently, our “synchronous” process is a bit dumb (as in, non-smart). Conceptually, it looks like this:

• The client periodically requests a server for ALL the latest data.
• The server sends remote data that overwrites the current set of local data in the client store.
• Any local creation / update / deletion after this point is processed as gold and immediately sent to the server.

The data itself is stored relationally and periodically updated by client users. Clients in my particular case do not care too much about the relationship itself (therefore, we can leave with local storage in the browser at the moment).

Obviously, this is not true synchronization. I want to go to a system where, in fact, the diff from the latest changes is periodically sent to the server, and the server sends back the diff from the latest changes that it knows about. It seems very difficult to get to this point, but maybe I just don’t understand the problem very well.

REST feels like a good start, but REST only talks about how the two data stores talk to each other, and not how the data itself is synchronized between them. (This synchronization process is left to the developer of each store.) What is the best way to implement this process? Is there a modern set of software design patterns that are used to communicate a specific solution to this problem? I am mainly interested in the general (technological agnostic) approach, if possible ... but the specific framework would be useful to view if they exist.

Answer 1

Replication with multiple masters is always (and always will be) complex and custom-made, as conflict handling will be specific to your application.

IMO A more reliable approach is to use Master-slave replication, and your web service is both a leader and clients as subordinates. To synchronize clients, use the archived atomic change flow (see event sourcing) in accordance with RFC5005 . This is your closest access to the modern standard for this type of replication, and it is RESTful.

When clients are online, they do not update their replica directly; instead, they send commands to the server and update their replication through the Atom channel.

When customers offline get harder. Your customers should have a behavior model for your web service. It will need to have a standalone copy of your replica, which must be copied when recording from an online replica (an online replica is one that is updated with an atom feed). When a client executes commands that modify data, it must store the command (for subsequent playback against the web service), the expected result (for verification during playback) and updating the offline replica.

When the client returns to the network, he must reproduce the commands, compare the result with the expected result and notify the client of any deviations. How these deviations will be handled depends on your application. You can then turn off the offline replica.

Answer 2

CouchDB replication works through HTTP and does what you are looking for. After the databases are synchronized at both ends, it will send diff to add / update / delete.

Couch can do this with other Couch computers or with a mobile platform such as TouchDB.

https://github.com/couchbaselabs/TouchDB-iOS

I have done a lot, but you can always configure CouchDB on one machine, configure TouchDB on a mobile device, and then watch HTTP traffic back and forth to get an idea of ​​how they do it.

Or read this: http://guide.couchdb.org/draft/replication.html

Perhaps some of the link above will help you understand how to make your own differences for your REST service. (Since both of them over HTTP consider this to be useful.)

Answer 3

You might want to explore the Dropbox Datastore API:

https://www.dropbox.com/developers/datastore

It seems like this could be very well suited for your purposes. They have iOS and javascript clients.

Answer 4

I have been interested in Meteor recently.

The platform installs Mongo on the server and minimongo in the browser. The client subscribes to some data, and when this data changes, the platform automatically sends new data to the client.

This is a smart solution to the synchronization problem, and it solves other problems as well. It will be interesting to know if this will make more platforms in the future.