Efficient, consistent client-side apps with optimistic concurrency & JSON patch

November 06, 2014Aaron "Amac" McCall

While working on a line of business application for a client recently, I was asked to research and implement two different approaches towards improving data updating efficiency and consistency.

The first is JSON Patch. The idea here is to reduce data transfer by only sending the operations needed to make the remote resource identical to the local one. Even though both resources are represented as JSON objects, applying patches means we don't have to replace the entire entity on every update. This also reduces the risk of accidental changes to data that stays the same.

The second is optimistic concurrency control. This approach allows multiple users to open a data record for editing at the same time, and determines whether there are any conflicts at save time.

Our working hypothesis was that combining these two approaches would enable us to build a more bandwidth-efficient, data-consistent application while also providing a more pleasant user experience.

The Research

Given these new ideas, I wanted to get a basic understanding of the theory behind both, so I did some searching and reading. Here's what I learned.

A bit of background on optimistic concurrency via HTTP

Pessimistic concurrency, the traditional approach to protecting data integrity, requires locking the data resource until the update is committed. Optimistic concurrency, on the other hand, does not lock the resource, preferring instead to check for violations of integrity rules just-in-time as the last step before committing. The pessimistic approach is best in situations where there is a high likelihood of conflicting edits because the lock is acquired and released regardless of any actual conflicts to be prevented, making it significantly slower in most cases than a non-locking approach. In situations where the likelihood of conflicts is low, the optimistic approach is faster because it only follows the slow path—a rolled-back transaction—in the event of an actual conflict.

In our case, we would be using HTTP headers to pass the integrity check data for our optimistic concurrency system, so I started reading up on it. A quick Google search turned up a few useful resources:

These articles all assume that you will be using ETag and If-Match for transporting version information, but the header field definitions in the official HTTP specs also discuss another pair for similar uses: Last-Modified and If-Unmodified-Since. Depending on the situation, either option may be appropriate. In the case of our app, Last-Modified was the path of least resistance because our REST API was already sending that header and the limited resolution of HTTP date was not likely to be an issue.

To summarize, our server sends something like Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT and our client app needs to store the value and send it back as If-Unmodified-Since: Sun, 26 Sep 2010 22:04:35 GMT when it sends updated data to the server. In addition, the client would need to handle a 412 error response from the server as an invalid version error, meaning that the If-Unmodified-Since check had failed.

JSON Patch basics

There are a number of operations that can be modeled via JSON Patch: add, remove, replace, copy, move, and test. Each operation is modeled using a specially formed JSON object. Every operation type has an "op" and a "path" property. The copy and move operation also have a "from" property. Finally, the add, replace, and test operations have "value" property. For our purposes, client-server communication is entirely modeled via add, replace and remove. Here are some examples of those operation objects:

var original = {
  "cars": [
    {id: 1, "model":"De Ville"},
    {id: 2, "model": "Fleetwood"}
  ]
};

var add = {"op": "add", "path": "/cars/2", "value": {"model": "Eldorado"}};
var remove = {"op": "remove", "path": "/cars/1"};
var replace = {"op": "replace", "path": "/cars/0/model", "value": "Coupe De Ville"};

// The result of applying the above would be:

var patched = {
  "cars": [
    {id: 1, "model":"Coupe De Ville"},
    {id: 3, "model": "El Dorado"}
  ]
};

Our client application would need to accumulate any changes to a model or its child models and collections as JSON Patch operations and send them to the server on save.

The sites that helped me the most to understand JSON Patch were:

Prior Art and Priorités d'Art

Armed with this new knowledge and ready to tackle implementation, I began by searching for previous Backbone implementations of these concepts. I saw that Backbone supported HTTP's PATCH verb, but it had no additional support for tracking what had changed since the last sync with the server. I found a Stack Overflow question and a jsonpatch library that mentioned Backbone, but no fully-baked generic solutions for JSON Patch. When I went looking for optimistic concurrency solutions, I couldn't even find a Stack Overflow question about it. Since I couldn't find any solutions, I'd have to build my own from scratch.

As I thought through my approach, I kept these things in mind:

It had to work both with Backbone and Ampersand, which we were planning to migrate to in the next few months.
It should require minimal configuration from developers for most cases.
It must be modular, allowing the use of optimistic concurrency without JSON Patch and vice versa.

In my next post, I'll describe how I went about implementing these solutions, and the operational results in the efficiency of updating data in our application.

COMING SOON: "Part II: Building the Mixins"

Do you like reading fun stuff like this? Then why not sign up for our mailing list? Check it out below.

&yet

The Blog