&yet

The &yet Blog

It used to all make sense.

The web was once nothing but documents.

Just like you'd want some type of file browser UI to dig through files on your operating system, obviously, you need some type of document browser to view all these web-addressable "documents".

But over time, those "documents" have become a lot more. A. lot. more.

But I can now use one of these "documents" to have a 4 person video/audio conference on Talky with people anywhere in the world, play incredible full-screen first-person shooters at 60fps, write code in a full-fledged editor, or {{ the reader may insert any number of amazing web apps here }} using nothing but this "document viewer".

Does calling them "documents" seem ridiculous to anyone else? Of course it does. Calling them "sites" is pretty silly too, actually because a "site" implies a document with links and a URL.

I know the "app" vs. "site" debate is tired and worn.

Save for public, content-heavy sites, all of the apps that I'm asked to write by clients these days at &yet are fully client-side rendered.

The browser is not an HTML renderer for me, it's the world's most ubiquitous, yet capable, runtime. With the amazing capabilities of the modern web platform, it's to the point where referring to a browser as a document viewer is a insult to the engineers who built it.

There is a fundamental difference when you treat the browser as a runtime instead of a document renderer.

I typically send it nothing but a doctype, a script tag, and a stylesheet with permanent cache headers. HTML just happens to be the way I tell the browser to download my app. I deal with the initial latency issues by all-but-ensuring visitors hit the app with a primed cache. This is pretty easy for apps that are opened frequently or are behind a static login page in which you prefetch the app resources. With proper cache headers the browser won't even do the 304 not-modified dance. It will simply start executing code.

This makes some people cringe, and many web purists (luddites?! #burn) would argue that everything should gracefully degrade and that there isn't, or at least there shouldn't be, any distinction between a JavaScript app and site. When I went to EdgeConf in NYC the "progressive enhancement" panel said a lot of things like "your app should still be usable without JS enabled". Often "javascript is disabled" is really the time when the browser is downloading your javascript. To this I say:

WELL, THEN SHOW ME A TALKY.IO CLONE THAT GRACEFULLY DEGRADES!

It simply cannot be done. Like it or not, the web has moved on from that myopic view of it. The blanket graceful degradation view of the web no longer makes sense when you can now build apps whose core use case is fully dependent on a robust JavaScript runtime.

I had a great time at Chrome Dev Summit, but again, the core message of the "Instant Mobile Apps" talk was: "render your html on the server to avoid having your render blocking code require downloading your JS before it can start executing."

For simple content-driven sites, I agree. Completely. The demo in that particular talk was the Chrome developer documentation. But it's a ridiculously easy choice to render documentation server side. (In fact the notion that there was ever a client-side rendered version to begin with was surprising to me.)

If your view of the web lacks a distinction between clientside apps and sites/documents, I'd go as far as to say that you're now part of the problem.

Why?

Because that view enables corporate IT departments to argue for running old browsers without getting laughed out of the building.

Because that view keeps some decision makers from adopting 100% JavaScript apps and instead spending money on native apps with web connectivity.

Because that view wastes precious developer time inventing and promoting hacks and workarounds for shitty browsers when they could be building next-generation apps.

Because that view enables you to argue that your proficiency of browser CSS hacks for IE7 is still relevant.

Because that view will always keep the web locked into the browser.

What about offline?

I'm writing this on a plane without wifi and of course, using a native app to do so. There are two primary reasons for this:

  1. The offline web is still crap. See offlinefirst.org and this hood.ie post for more.
  2. All my favorite web-based tools are still stuck in the browser.

The majority of users will never ever open a browser without an Internet connection, type in a URL and expect ANYTHING to happen.

Don't get me wrong, I'm very supportive of the offline first efforts and they are crucial for justifying that

We have a very different view of apps that exist outside of the browser. In fact, the expectation is often reversed: "Oh right, I do need a connection for this to work".

Chrome OS is one approach, but I think its 100% cloud-based approach is more hardcore than the world is ready to adopt and certainly is never going to fly with the indie data crowd or the otherwise Google-averse.

So, have I ranted enough yet?

According to Jake Archibald from Google, ServiceWorkers will land in Canary sometime early 2014. This work is going to fundamentally change what the web can do.

If you're unfamiliar with ServiceWorkers (previously called Navigation Controllers), they let you write your own cache control layer in javascript for your web application. ServiceWorkers promise to serve the purpose that appcache was intended for: truly offline web apps.

At a high level, they now let javascript developers building clientside apps to treat the existence of a network connection as an enhancement rather than an expectation.

You may think, "Oh, well, the reason we use the web is because access to the network provides our core value as an app."

While I'd tend to agree that most of the useful apps fundamentally require data from the internet to be truly useful, you're missing the point.

Even if the value of your app depends entirely on a network connection, you can now intercept requests and choose to answer them from caches that you control, while in parallel attempting to fetch newer versions of those resources from the network.

If you think about it, that capability is no different than something like Facebook for iOS or Android.

That Facebook app's core value is unquestioningly derived from seeing your friends' latest updates and photos, which you're obviously not going to get without a connection. But the fundamental difference is this: the native app will still open the app and show you all the cached content it has. As a result (and for other reasons) the OS has given those types of apps a privileged status.

With full programmatic cache control for the web that ServiceWorkers will offer, you'll be able to choose to load your app and whatever latest content you had downloaded from cache first while optionally trying to connect and download new things from the network. The addition of a controllable cache layer in web apps means that an app like facebook really has no compelling reason to be a native app. I mean, really. If you break it down, that app is mostly a friend timeline browser, right? (the key word there being browser).

BUT, even with the addition of ServiceWorkers, there's another extremely important difference: user perception.

We've spent years teaching users that things they use in their web browser simply do not work offline. Users understand (at least at on some unconscious level) that the browser is the native app that gets sites/documents from the Internet. From a user experience standpoint, trying to teach the average user anything different is attempting to roll a quarry full of rocks up a hill.

This is where it starts to become apparent that failing to draw a distinction between a fully client "apps" and a website really starts to become a disservice to all these new capabilities of the web platform. It doesn't matter how good the web stack becomes, it will never compete with native apps in the "native" space while it stays stuck in the browser.

The addition of "packaged" chrome apps is an admirable, but in my opinion, still inadequate attempt at addressing this issue.

At the point where a user on a mobile device opts to "add to home screen" the intent from the user is more than just a damn bookmark, they're saying: "I want access to this on the same level as my native apps". It's a user's request for an installation of that app, but in reality it's treated as a shitty, half-assed install that's really just a bookmark. But the intent from the user is clear: "I want a special level of quick and easy access to this specific app".

So why not just embrace that what they're actually trying to do is "install" that web application into their operating system?

Apple sort of does this for Mac Apps. After you first "sideload" (a.k.a. download from the web and try to run) a native Mac desktop app, they treat it a bit like an awkward stepchild when you first open it. They warn you and tell you: hey, this was an app downloaded from the Internet, are you sure you want to let this thing run?

While I'm not a fan of the language or the FUD involved with that, the timing makes perfect sense to me. At the point I've opted to "install" something to my homescreen on my mobile device (or the equivalent to that for desktop), that seems like the proper inflection point to verify with the user that they do, in fact, want to let this app have access to specific "privileged" OS APIs.

Without a simple way to install and authorize a clientside web app, these kinds of apps will always get stuck in the uncanny valley of half-assed, semi-installed apps.

So why bother in the first place? Why not just do native whenever you want to build an "app"? Beyond providing a way to build for multiple platforms, there's one more thing the web has that native apps don't have: a URL.

The UNIVERSAL RESOURCE LOCATOR concept is easy to take for granted, but it's insanely useful to be able to reference things like links to emails inside gmail, or a tweet, or a very specific portion of documentation. Being able to naturally link between apps on the web is what gives the web it's power. It's unfortunate that many, when they first start building single page applications don't update URLs as they go and fail to respect the "back" button, thus breaking the web.

But when done properly, the blending the rich interactivity of native apps with the addressability and ubiquity of the web is a thing of beauty.

I cannot understate how excited I am about Service Workers. Because finally, we'll have the ability to build web applications that treat network resources the same way that good native applications do: as an enhancement.

Of course, the big IF is whether platforms play along and actually treat these types of apps as first class citizens.

Call me an optimist, but I think the capabilities that ServiceWorkers promise us, will shine a light on the bizarre awkwardness of the concept of opening a browser to access offline apps.

The web platform's capabilities have outgrown the browser.

Let's help the web to make its next big push.

--

I'm @HenrikJoreteg on twitter. I'd love to hear your thoughts on this.

For further reading on ServiceWorkers, here is a great explainer doc.

Also, check out my book on building sanely structured single page applications.

I had the privilege to attend EdgeConf 2013 as a panelist and opening speaker for the Realtime Data discussion.

It was an incredible, deeply technical conference with an interesting discussion/debate format.

Here's the video from the panel:

The slides from my talk can be found on speakerdeck.

It was a privilege to attend — I'm very grateful to Andrew Betts and FT Labs for the opportunity to be there.

A big driver for And Bang was that it was all supposed to be "glowingly" realtime. Many apps that have some sort of "realtime" component have it sort of "tacked-on" to a more traditional request/response based infrastructure. We wanted the whole experience and the whole technology stack to be realtime.

When I first started tinkering around with building a team task tool a few years ago, I grabbed a set of what were my familiar tools at the time. Django, MySQL and a bit of jQuery.

As it turned out, if you want to build a highly interactive, multi-user, realtime application, Django or Rails and relational databases are not necessarily the right tools for the job.

We quickly realized that we needed to find and learn, or create, a new set of tools to solve the types of problems this presented.

You can trace our progress and experiments a bit by reading past blog posts on the process of finding or building those tools up until we finally launched 1.0 a year ago.

When we launched at last year's Realtime Conference. We had something that worked. It worked really well, in fact. The amount of maintenance we've had to do has been minimal and the tool has become a part of the daily workflow for quite a few happy customers.

We accomplished our goal of making a highly synchronized, completely realtime app. Everything is synchronized. Even when you upload a new avatar your teammates will see it change live as soon as it's uploaded.

Although we're proud of, and excited about, what we built for 1.0, something about it didn't feel quite right to us. As we went on to start building more features, we kept hitting roadblocks. For example:

  1. It wasn't very extensible.

  2. It was hard for people on our team to contribute to. The whole system was too complex and required too much knowledge of the whole stack in order to feel confident touching any of it.

  3. The server code and application data was very tightly coupled to the presentation of the data in the app.

  4. In fact, in order to get this "full stack eventing" that we wanted, everything was quite tightly coupled.

  5. It was difficult to visualize or manage the data.

  6. It was hard to build alternate interfaces or native apps.

So, after doing some initial iterations on the launched 1.0 version we did some soul searching. We still didn't have this ideal realtime platform that we wanted. Being the nerds and tinkerers that we are, we went back to the drawing board.

The core lesson

If I had to pick the biggest, most valuable lesson we learned from this whole experience, it is something that every computer science professor in the world could probably have told us:

SEPARATION OF CONCERNS!

Unless your goal is to build a completely decentralized system where each node is a server and a client I'd suggest following a few simple rules:

  1. Servers are for data

  2. Clients are for presentation

  3. Don't mix

Divide and conquer

To illustrate my point, instead of a single respository that contained the entire AndBang 1.0 application. We now have the following codebases:

  • andbang-spec

The single purpose of this repo is to describe the functionality of the API — in intense detail.

  • accounts.andbang.com

The application that does oauth and account managment. This is where you log in to manage the teams you're on, etc.

  • api.andbang.com

The API. It speaks nothing but data. It doesn't know even how to render HTML.

  • docs.andbang.com

It's entire purpose is to document and demonstrate the API. And yet it doesn't actually contains any details about it. Instead it consumes a JSON spec that the API serves that describes its own capabilities. The docs app just renders it into a nice application you can use to explore the capabilities exposed.

  • andbang.com

The "official" desktop web client. But it's just that, a client built on the public API. It doesn't contain any non-public information or get any special access. Any one of our customers will be capable of building something like this this without requesting any special access.

  • andbang-iphone

Yup. Another "official" client. This time for iOS.

  • andbang.js

The "drop-in" JS sdk that andbang.com uses to talk to the API. This is generated from consuming the andbang-spec.

  • andbang-express-auth

Drop in oauth middleware. If you happen to want to build an express.js app on our API. It makes the OAuth piece ridiculously simple.

So, we went from a single repository that did everything. To eight distinct codebases.

We'll have a bunch of other posts that go into more detail about the various items above.

Why this matters

Obviously, seperation of concerns is not a new concept by any means. However, when surveying the landscape of people who are wanting to build fully realtime apps I'm seeing tight coupling of frontends to backends and interfaces to data structures.

Ultimately, what we want is the ability to access our data lots of ways and to build a variety of interesting integrations and clients on it. The fact that it's also "realtime" shouldn't limit that.

Frankly, in today's landscape with a plethora of client platforms building for only desktop web, or iOS, or Android, or tablets is silly. The most interesting services all provide this.

If you want to be able to control your app from Google Glass or the HUD in your car, a tightly coupled desktop web app probably won't cut it.

For more on all of this, see my talk from realtime conf:

RTC 2012 - 13 - Henrik Joreteg from &yet on Vimeo.

Be sure to put yourself on the list for access to AndBang 2.0. Hope you like it!

Add your email to join the And Bang 2.0 private beta invite list:

These days, more and more HTML is rendered on the client instead of sent pre-rendered by the server. So If you're building a web app that uses a lot of client side javascript you'll doubtlessly want to create some HTML in the browser.

How we used to do it

First a bit of history. When I first wrote ICanHaz.js I was just trying to ease a pain point I was having: generating a bunch of HTML in a browser is a pain.

Why is it a pain? Primarily because JS doesn't cleanly support multi-line strings, but also because there isn't an awesome string interpolation system built into JS.

To work around that, ICanHaz.js, as lots of other template clientside template systems do, uses a hack to make it easier to send arbitrary strings to the browser. As it turns out, browsers ignore content in <script> tags if you give them a type attribute that isn't text/javascript. So, ICanHaz reads the content of tags on the page that say: <script type=“text/html”> which can contain templates or any other multi-line strings for that matter. So, ICanHaz will reads those templates and turns each of them into a function that you can call to render that string with your data mixed into it. For example:

This html:

<script id="user" type="text/html">
  <li>
    <p class="name">Hello I'm {{ name }}</p>
    <p><a href="http://twitter.com/{{ twitter }}">@{{ twitter }}</a></p>
  </li>
</script>

Is read by ICanHaz and turned into a function you call with your own like this:

// your data
var data = {
  first_name: "Henrik",
  last_name: "Joreteg"
}

// I can has user??
html = ich.user(data)

This works, and lots of people clearly thought the same as it's been quite a popular library.

Why that's less-than-ideal

It totally works, but if you think about it, it's a bit silly. It's not super fast and you're making the client do a bunch of extra parsing just to turn text into a function. You also have to send the entire template engine to the browser which is a bunch of wasted bandwidth.

How we're doing it now

What I finally realized is that all you actually want when doing templating on the client is the end result that ICanHaz gives you: a function that you call with your data that returns your HTML.

Typically, smart template engines, like the newer versions of Mustache.js, do this for you. Once the template has been read, it gets compiled into a function that is cached and used for subsequent rending of that same template.

Thinking about this leaves me asking: why don't we just send the javascript template function to the client instead of doing all the template parsing/compiling on the client?

Well, frankly, because I didn't really know of a great way to do it.

I started looking around and realized that Jade (which we already use quite a bit at &yet) has support for compiling as a separate process and, in combination with a small little runtime snippet, this lets you create JS functions that don't need the whole template engine to render. Which is totally awesome!

So, to make it easier to work with, I wrote a little tool: templatizer that you can run on the server-side (using node.js) to take a folder full of jade templates and turn them into a javascript file that you can include in your app that has just has the template rendering functions as javascript.

The end result

From my tests the actual rendering of templates is 6 to 10 times faster. In addition you're sending way less code to the browser (because you're not sending a whole templating engine) and you're not making the browser do a bunch of work you could have already done ahead of time.

I still need to write more docs and use it for a few more projects before we have supreme confidence in it, but I've been quite happy with the results so far and wanted to share it.

I'd love to hear your thoughts. I'm @HenrikJoreteg on twitter and you should follow @andyet as well and check out our awesome team same-pagification tool And Bang.

See you on the Internet. Go build awesome stuff!


The single biggest challenge you'll have when building complex clientside application is keeping your code base from becoming a garbled pile of mess.

If it's a longer running project that you plan on maintaining and changing over time, it's even harder. Features come and go. You'll experiment with something only to find it's not the right call.

I write lots of single page apps and I absolutely despise messy code. Here are a few techniques, crutches, coping mechanisms, and semi-pro tips for staying sane.

Separating views and state

This is the biggest lesson I've learned building lots of single page apps. Your view (the DOM) should just be blind slave to the model state of your application. For this you could use any number of tools and frameworks. I'd recommend starting with Backbone.js (by the awesome Mr. @jashkenas as it's the easiest to understand, IMO.

Essentially, you'll build up a set of models and collections in memory in the browser. These models should be completely oblivious to how they're used. Then you have views that listen for changes in the models and update the DOM. This could be a whole giant blog post in an of itself. But this core principal of separating your views and your application state is vital when building large apps.

Common JS Modules

I'm not going to get into a debate about module styles and script loaders. But I can tell you this: I haven't seen any cleaner, simpler mechanism for splitting your code into nice isolated chunks than Common JS modules.

It's the same style/concept that is used in node.js. By following this style I get the additional benefit of being able to re-use modules written for the client on the server and vice versa.

If you're unfamiliar with the Common JS modules style, your files end up looking something like this:

// you import things by using the special `require` function and you can
// assign the result to a variable

var StrictModel = require('strictModel'),
    _ = require('underscore');

// you expose functionality to other modules by declaring your main export
// like this.
module.exports = StrictModel.extend({
    type: 'navItem',
    props: {
        active: ['boolean', true, false],
        url: ['string', true, ''],
        position: ['number', true, 200]
    },
    init: function () {
        // some, something
    }
});

Of course, browsers don't have support for these kinds of modules out of the box (there is no window.require). But, luckily that can be fixed. I use a clever little tool called stitch written by Sam Stephenson of 37signals. There's also another one by @substack called browserify that lets you use a lot of the node.js utils on the client as well.

What they do is create a require function and bundle up a folder of modules into an app package.

Stitch is written for node.js but you could just as easily just use another server-side language and just use node to build your client package. Ultimately it's just creating a single JS file and of course at that point you can just serve it like any other static file.

You set up Stitch in a simple express server like this:

// require express and stitch
var express = require('express'),
    stitch = require('stitch');

// define our stitch package
 var appPackage = stitch.createPackage({
    // you add the folders whose contents you want to be “require-able”
    paths: [
        __dirname + '/clientmodules',  // this is where i put my standalone modules
        __dirname + '/clientapp' // this is where i put my modules that compose the app
    ],
    // you can also include normal dependencies that are not written in the 
    // commonJS style
    dependencies: [
        somepath + '/jquery.js',
        somepath + '/bootstrap.js'
    ]
});

// init express
var app = express.createServer();

// define a path where you want your JS package to be server
app.get('/myAwesomeApp.js', appPackage.createServer());

// start listening for requests
app.listen(3000);

At this point you can just go to http://localhost:3000/myAwesomeApp.js in a browser and you should see your whole JS package.

This is handy while developing because you don't have to re-start or recompile anything when you make changes to the files in your package.

Once you're ready to go to production you can use the package and uglify JS to write a minified file to disk to be served staticly:

var uglifyjs = require('uglify-js'),
    fs = require('fs');

function uglify(code) {
    var ast = uglifyjs.parser.parse(code);
    ast = uglifyjs.uglify.ast_mangle(ast);
    ast = uglifyjs.uglify.ast_squeeze(ast);
    return uglifyjs.uglify.gen_code(ast);
}

// assuming `appPackage` is in scope of course, this is just a demo
appPackage.compile(function (err, source) {
    fs.writeFileSync('build/myAwesomeApp.js', uglify(source));
});

Objection! It's a huge single file, that's going to load slow!

Two things. Don't write a huge app with loads and loads of giant dependencies. Second, cache it! If you do your job right, your users will only download that file once and you can probably do it while they're not even paying attention. If you're clever you can even prime their cache by lazy-loading the app on the login screen, or some other such cleverness.

Not to mention, for single page apps, speed once your app has loaded is much more important than the time it takes to do the initial load.

Code Linting

If you're building large JS apps and not doing some form of static analysis on your code, you're asking for trouble. It helps catch silly errors and forces code style consistency. Ideally, no one should be able to tell who wrote what part of your app. If you're on a team, it should all be uniform within a project. How do you do that? We use a slick tool written by Nathan LaFreniere on our team called, simply, precommit-hook. So all we have to do is:

npm install precommit-hook

What that will do is create a git pre-commit hook that uses JSHint to check your project for code style consistency before each commit. Once upon a time there was a tool called JSLint written by Mr. Crockford. Nowadays (love that silly word) there's a less strict, more configurable version of the same project called JSHint.

The neat thing about the npm version of JSHint is that if you run it from the command line it will look for a configuation file (.jshintrc) and an ignore file (.jshintignore) both of which the precommit hook will create for you if they don't exist. You can use these files to configure JSHint to follow the code style rules that you've defined for the project. This means that you can now run jshint . at the root of your project and lint the entire thing to make sure it follows the code styles you've defined in the .jshintrc file. Awesome, right!?!

Our .jshintrc files usually looks something like this:

{
    "asi": false,
    "expr": true,
    "loopfunc": true,
    "curly": false,
    "evil": true,
    "white": true,
    “undef": true,
    "predef": [
        "app",
        "$",
        "require",
        "__dirname",
        "process",
        "exports",
        "module"
    ]
}

The awesome thing about this approach is that you can enforce consistency and that the rules for the project are contained and actually checked into the project repo itself. So if you decide to have a different set of rules for the next project, fine. It's not a global setting it's defined and set by whomever runs the project.

Creating an "app" global

So what makes a module? Ideally, I'd suggest each module being in it's own file and only exporting one piece of functionality. Only having a single export helps you keep clear what purpose the module has and keeps it focused on just that task. The goal is having lots of modules that do one thing really well and then your app just combines modules into a coherent story.

When I'm building an app, I intentionally have one main controller object of sorts. It's attached to the window as “app” just for my own. For modules that I've written specifically for this app (stuff that's in the clientapp folder) I allow myself the use of that global to perform app-level actions like navigating, etc.

Using events: Modules talking to modules

How do you keep your modules cleanly separated? Sometimes modules are dependant on other modules. How do you keep them loosely coupled? One good technique is triggering lots of events that can be used as hooks by other code. Many of the core components in node.js are extensions of EventEmitter the reason is that you can register handlers for stuff that happens to those items just like you can register a handler for someone clicking a link in the browser. This pattern is really useful when building re-usable compenents yourself. By exporting things that inherit from event emitters means that the code using your module can specify what they care about rather than the module having to know. For example, see the super simplified version of the And Bang js library below.

There are lots of implementations of event emitters. We use a modified version of one from the LearnBoost guys: @tjholowaychuk, @rauchg and company. It's wildemitter on my github if you're curious. But the same concept works for any of the available emitters. See below:

// require our emitter
var Emitter = require('wildemitter');

// Our main constructor function
var AndBang = function (config) {
    // extend with emitter
    Emitter.call(this);
};

// inherit from emitter
AndBang.prototype = new Emitter();

 // Other methods
AndBang.prototype.setName = function (newName) {
    this.name = newName;
    // we can trigger arbitrary events
    // these are just hooks that other
    // code could chose to listen to.
    this.emit('nameChanged', newName);
};

// export it to the world
module.exports = AndBang;

Then, other code that wants to use this module can listen for events like so:

var AndBang = require('andbang'),
    api = new AndBang();

// now this handler will get called any time the event gets triggered
api.on('nameChanged',  function (newName) { /* do something cool */ });

This pattern makes it easy to expose functionality without having to know anything about the consuming code.

More?

I'm tired of typing so that's all for now. :)

But I just thought I'd share some of the tools, techniques and knowledge we've acquired through blood, sweat and mistakes. If you found it helpful, useful or if you want to yell at me. You can follow me on twitter: @HenrikJoreteg.

See ya on the interwebs! Build cool stuff!


The other day, DHH[1] tweeted this:

Forcing your web ui to be "just another client" of your API violates the first rule of distributed systems: Don't write distributed systems.

-- DHH (@dhh) June 12, 2012

In building the new Basecamp, 37signals chose to do much of the rendering on the server-side and have been rather vocal about that, bucking the recent trend to build really richly interrative, client-heavy apps. They cite speed, simplicity and cleanliness. I quote DHH, again:

It's a perversion to think that responding to Ajax with HTML fragments instead of JSON is somehow dirty. It's simple, clean, and fast.

-- DHH (@dhh) June 12, 2012

Personally, I think this generalization is a bit short-sighted.

The "rule" that is cited in the first tweet about distributed sytstems is from Martin Fowler who says:

First Law of Distributed Object Design: Don't distribute your objects!

So, yes, duplicating state into the client is essentially just that: you're distributing your objects. I'm not saying I'm wiser than Mr. Fowler, but I do know that keeping client state can make an app much more useful and friendly.

Take Path for iPhone. It caches state, so if you're offline you can read posts from your friends. You can also post new updates while offline that just seemlessly get posted when you're back on a network. That kind of use case is simply impossible unless you're willing to duplicate state to the client.

As application developers we're not trying to dogmatically enforce "best practices" of computer science just for the sake of dogma, we're trying to build great experiences. So as soon as we want to support that type of use case, we have to agree that it's OK to do it in some circumstances.

As some have pointed out and DHH acknowledged later, even Basecamp goes against his point with the calendar. In order to add the type of user experience they want, they do clientside MVC. They store some state in the client and do some client-side rendering. So, what's the difference in that case?

I'm not saying all server side rendering is bad. I'm just saying, why not pick one or the other? It seems to me (and I actually speak from experience here) that things get really messy once you start mixing presentation and domain logic.

As it turns out, Martin Fowler actually wrote A WHOLE PAPER about separating presentation from domain logic.

The other point I'd like to make is this: What successful, interesting web application do you know/use/love that doesn't have multiple clients?

As soon as you have any non-web client, such as an iPhone app, or a dashboard widget or a CLI or some other webapp that another developer built, you now need a seperate data API anyway.

Obviously, 37signals has an API. But, gauging by the docs, there are pieces of the API that are incomplete. Another benefit of dog-fooding your own API is that you can't ship with an incomplete API if you built your whole app on it.

We're heads-down on the next version of And Bang which is built entirely on what will be our public API. This re-engineering has been no small undertaking, but we feel it will be well worth the effort.

The most interesting apps we use are not merely experienced through a browser anymore. APIs are the one true cross-platform future you can safely bank on.

I'm all ears if you have a differing opinion. Hit me up on twitter @HenrikJoreteg and follow @andyet and @andbang if you're curious about what else we're up to.


[1] DHH (David Heinemeier Hansson) of Ruby on Rails and 37signals fame is not scared to state his opinions. I think everyone would agree that his accomplishments give him the right to do so. To be clear, I have nothing but respect for 37 Signals. Frankly, their example is a huge source of inspiration for bootstrapped companies like ours at &yet.

This last year, we've learned a lot about building scalable realtime web apps, most of which has come from shipping &bang.

&bang is the app we use to keep our team in sync. It helps us stay on the same page, bug each other less and just get stuff done as a team.

The process of actually trying to get something out the door on a bootstrapped budget helped us focus on the most important problems that needed to be solved to build a dynamic, interactive, real-time app in a scaleable way.

A bit of history

I've written a couple of posts on backbone.js since discovering it. The first one introduces Backbone.js as a lightweight client-side framework for building clean, stateful client apps. In the second post I introduced Capsule.js. Which is a tool that I built on top of Backbone that adds nested models and collections and also allows you to keep a mirror of your client-side state on a node.js server to seemlessly synchronize state between different clients.

That approach was great for quickly prototyping an app. But as I pointed out in that post, that's a lot of in memory state being stored on the server and simply doesn't scale very well.

At the end of that post I hinted at what we were aiming to do to ultimately solve that problem. So this post is meant to be a bit of an update on those thoughts.

Our new approach

Redis is totally freakin' amazing. Period. I can't say enough good things about it. Salvatore Sanfilippo is a god among men, in my book.

Redis can scale.

Redis can do PubSub.

PubSub just means events. Just like you can listen for click events in Javascript in a browser you can listen for events in Redis.

Redis, however is a generic tool. It's purposely fairly low-level so as to be broadly applicable.

What makes Redis so interesting, from my perspective, is that you can treat it as a shared memory between processes, languages and platforms. What that means, in a practical sense, is that as long as each app that uses it interacts with it according to a pre-defined set of rules, you can write a whole ecosystem of functionality for an app in whatever language makes the most sense for that particular task.

Enter Thoonk

My co-worker, Nathan Fritz, is the closest thing you can get to being a veteran of realtime technologies.

He's a member of the XSF council for the XMPP standard and probably wrote his first chat bot before you knew what chat was. His Sleek XMPP Python library is iconic in the XMPP community. He has a self-declared un-natural love for XEP-60 which describes the XMPP PubSub standard.

He took everything he learned from his work on that standard and built Thoonk. (In fact, he actually kept the PubSub spec open as he built the Javascript and Python implementations of Thoonk.)

What is Thoonk??

Thoonk is an abstraction on Redis that provides higher-level datatypes for a more approachable interface. Essentially, staring at Redis as a newbie is a bit intimidating. Not that it's hard to interface with, it's just kind of tricky to figure out how to logically structure and retrieve your data. Thoonk simplifies that into a few data-types that describe common use cases. Primarly "feeds", "sorted feeds", "queues" and "jobs".

You can think of a feed as an ad-hoc database table. They're "cheap" to create and you simply declare them to make them or use them. For example, in &bang, we have all our users in a feed called "users" for looking up user info. But also, each user has a variety of individual feeds. For example, they have a "task" feed and a "shipped" feed. This is where it veers from what people are used to in a relational database model, because each user's tasks are not a part of a global "tasks" feed. Instead, each user has a distinct feed of tasks because that's the entity we want to be able to subscribe to.

So rather than simply breaking down a model into types of data, we end up breaking things into groups of items (a.k.a. "feeds") that we want to be able to track changes to. So, as an example, we may have something like this:

// our main user feed
var userFeed = thoonk.feed('users');

// an individual task feed for a user
var userTaskFeed = thoonk.sortedFeed('team.andyet.members.{{memberID}}.tasks');

Marrying Thoonk and Capsule

Capsule was actually written with Thoonk in mind. In fact that's why they were named the way they did: You know these lovely pneumatic tube systems they use to send cash to bank tellers and at Costco? (PPSHHHHHHH--THOONK! And here's your capsule.)

Anyway, the integration didn't end up being quite as tight as we had originally thought but it still works quite well. Loose coupling is better anyway right?

The core problem I was trying to solve with Capsule was unifying the models that are used to represent the state of the app in the browser and the models you use to describe your data on the server--ideally, not just unifying the data structure, but also letting me share behavior of those objects.

Let me explain.

As I mentioned, we recently shipped &bang. It lets a group of people share their task lists and what they're actively working on with each other.

It spares you from a lot of "what are you working on?" conversations and increases accountability by making your work quite public to the team.

It's a realtime, keyboard-driven, web app that is designed to feel like a desktop app. &bang is a node.js application built entirely with the methods described here.

So, in &bang, a team model has attributes as well as a couple of nested backbone collections such as members and chat messages. Each member has attributes and other nested collections, tasks, shipped items, etc.

Initial state push

When a user first logs in we have to send the entire model state for the team(s) they're on so we can build out the interface (see my previous post for more on that). So, the first thing we do when a user logs in is subscribe them to the relevant Thoonk feeds and perform the the initial state transfer to the client.

To do this, we init an empty team model on the client (a backbone/capsule model shared between client/server) . Then we recurse through our Thoonk feed structures on the server to export the data from the relevant feeds into a data structure that Capsule can use to import that data. The team model is inflated with the data from the server and we draw the interface.

From there, the application is kept in sync using events from Thoonk that get sent over websockets and applied to the client interface. Events like "publish", "change", "retract" and "position".

Once we got the app to the point where this was all working, it was kind of a magical moment, because at this point, any edits that happen in Thoonk will simply get pushed out through the event propagation all the way to the client. Essentially, the inteface that a user sees is largely a slave to the server. Except, of course, the portions of state that we let the user manipulate locally.

At this point, user interactions with the app that change data are all handled through RPC calls. Let's jump back to the server and you'll see what I mean.

I thought you were still using Capsule on the server?

We do, but differently, here's how that is handled.

In short... it's a job system.

Sounds intimidating right? As someone who started in business school, then gradually got into front-end dev, then back-end dev, then a pile of JS, job systems sounded scary. In my mind they're for "hardcore" programmers like Fritzy or Nate or Lance from our team. Job systems don't have to be that scary.

At a very high level you can think of a "job" as a function call. The key difference being, you don't necessarily expect an immediate result. To continue with examples from &bang: a job may be to "ship a task". So, what do we need to know to complete that action? We need the following:

  • member Id of the user shipping the task
  • the task id being completed (we call this "shipping", because it's cooler, and it's a reminder a reminder that finishing is what's important)

We can derive everything else we need from those key pieces of information.

So, rather than call a function somewhere:

shipTask(memberId, taskId)

We can just describe a job as a simple JSON object:

{
    userId: <user requesting the job>,
    taskId: <id of task to 'ship'>,
    memberId: <id of team member>
}

The we can add that to our "shipTask" job queue like so:

thoonk.job('shipTask').put(JSON.stringify(jobObject));

The cool part about the event propagation I talked about above is we really don't care so much when that job gets done. Obviously fast is key, but what I mean is, we don't have to sit around and wait for a synchronous result because the event propagation we've set up will handle all the application state changes.

So, now we can write a worker that listens for jobs from that job queue. In that worker we'll perform all the necessary related logic. Specifically stuff like:

  • Validating that the job is properly formatted (contains required fields of the right type)
  • Validating that the user is the owner of that task and is therefore allowed to "ship" it.
  • Modifying Thoonk feeds accordingly.

Encapsulating and reusing model logic

You'll notice that part of that list requires some logic. Specifically, checking to see if the user requesting the action is allowed to perform it. We could certainly write that logic right here, in this worker. But, in the client we're also going to want to know if a user is allowed to ship a given task, right? Why write that logic twice?

Instead we write that logic as a method of a Capsule model that describes a task. Then, we can use the same method to determine whether to show the UI that lets the user perform the action in the browser as we use on the back end to actually perform the validation. We do that by re-inflating a Capsule model for that task in our worker code and calling the canEdit() method on it and passing it the user id requesting the action. The only difference being, on the server-side we don't trust the user to tell us who they are. On the server we roll the user id we have for that session into the job when it's created rather then trust the client.

Security

One other, hugely important thing that we get by using Capsule models on the server is some security features. There are some model attributes that are read only as far a the client is concerned. What if we get a job that tries to edit a user's ID? In a backbone model if I call:

backboneModelInstance.set({id: 'newId'});

That will change the ID of the object. Clearly that's not good in a server environment when you're trusting that to be a unique ID. There are also lots of other fields you may want on the client but you don't want to let users edit.

Again, we can encapsulate that logic in our Capsule models. Capsule models have a safeSet method that assumes all inputs are evil. Unless an attribute is whitelisted as clientEditable it won't set it. So when we go to set attributes within the worker on the server we use safeSet when dealing with untrusted input.

The other important piece of securing a system that lets users indirectly add jobs to your job system is ensuring that the job you receive validate your schema. I'm using a node implementation of JSON Schema for this. I've heard some complaints about that proposed standard, but it works really well for the fairly simple usecase I need it for.

A typical worker may look something like this:

workers.editTeam = function () {
  var schema = {
    type: "object",
    properties: {
      user: {
        type: 'string',
        required: true
      },
      id: {
        type: 'string',
        required: true
      },
      data: {
        type: 'object',
        required: true
      }
    }
  };

  editTeamJob.get(0, function (err, json, jobId, timeout) {
    var feed = thoonk.feed('teams'), 
      result,
      team,
      newAttributes,
      inflated;

    async.waterfall([
      function (cb) {
        // validate our job
        validateSchema(json, schema, cb);
      },
      function (clean, cb) {
        // store some variables from our cleaned job
        result = clean;
        team = result.id;
        newAttributes = result.data;
        verifyOwnerTeam(team, cb);
      },
      function (teamData, cb) {
        // inflate our capsule model
        inflated = new Team(teamData);
        // if from the server user normal 'set'
        inflated.safeSet(newAttributes);
      },
      function (cb) {
        // do the edit, all we're doing is storing JSON strings w/ ids
        feed.edit(JSON.stringify(inflated.toJSON()), result.id, cb);
      }
    ], function (err) {
      var code;
      if (!err) {
        code = 200;
        logger.info('edited team', {team: team, attrs: newAttributes});
      } else if (err === 'notAllowed') {
        code = 403;
        logger.warn('not allowed to edit');
      } else {
        code = 500;
        logger.error('error editing team', {err: err, job: json});
      }
      // finish the job 
      editTeamJob.finish(jobId, null, JSON.stringify({code: code}));
      // keep the loop crankin'
      process.nextTick(workers.editTeam);
    });
  });
};

Sounds like a lot of work

Granted, writing a worker for each type of action a user can perform in the app with all the related job and validation is not an insignificant amount of work. However, it worked rather well for us to use the state syncing stuff in Capsule while we were still in the prototyping stage, then converting the server-side code to a Thoonk-based solution when we were ready to roll out to production.

So why does any of this matter?

It works.

What this ultimately means is that we now push the system until Redis is our bottleneck. We can spin up as many workers as we want to crank through jobs and we can write those workers in any language we want. We can put our node app behind HA proxy or Bouncy and spin up a bunch of 'em. Do we have all of this solved and done? No. But the core ideas and scaling paths seem fairly clear and doable.

[update: Just to add a bit more detail here, from our tests we feel confident that we can scale to tens of thousands of users on a single server and we believe we can scale vertically after doing some intelligent sharding with multiple servers.]

Is this the "Rails of Realtime?"

Nope.

Personally, I'm not convinced there ever will be one. Even Owen Barnes (who originally set out to build just that with SocketStream) said at KRTConf: "There will not be a black box type framework for realtime." His new approach is to build a set of interconnected modules for structuring out a realtime app based on the unique needs of its specific goals.

The kinds of web apps being built these days don't fit into a neat little box. We're talking to multiple web services, multiple databases, and pushing state to the client.

Mikeal Rogers gave a great talk at KRTConf about that exact problem. It's going to be really, really hard to create a framework that solves all those problems in the same way that Rails or Django can solve 90% of the common problems with routes and MVC.

Can you support a BAJILLION users?

No, but a single Redis db can handle a fairly ridiculous amount of users. At the point that actually becomes our bottleneck, (1) we can split out different feeds for different databases, and (2) we'd have a user base that would make the app wildly profitable at that point--certainly more than enough to spend some more time on engineering. What's more, Salvatore and the Redis team are putting a lot of work into clustering and scaling solutions for Redis that very well may outpace our need for sharding, etc.

Have you thought about X, Y, Z?

Maybe not! The point of this post is simply to share what we've learned so far.

You'll notice this isn't a "use our new framework" post. We would still need to do a lot of work to cleanly extract and document a complete realtime app solution from what we've done in &bang--particularly if we were trying to provide a tool that can be used to quickly spin up an app. If your goal is to find a tool like that, definitely check out what Owen and team are doing with SocketStream and what Nate and Brian are doing with Derby.

We love the web, and love the kinds of apps that can be built with modern web technologies. It's our hope that by sharing what we've done, we can push things forward. If you find this post helpful, we'd love your feedback.

Technology is just a tool, ultimately, it's all about building cool stuff. Check out &bang and follow me @HenrikJoreteg, Adam @AdamBrault and the whole @andyet team on the twitterwebz.


If you're building a single page app, keep in mind that &yet offers consulting, training and development services. Hit us up (henrik@andyet.net) and tell us what we can do to help.

Last week we launched our newest product, &!, at KRTConf. It's a realtime, single-page app that empowers teams to bug each other less and get more done as a team.

One of our speakers, Scott Hanselman from Microsoft tried to open the app in IE9 and was immediately redirected to a page that tells users they need WebSockets to use the app. He then wrote a post criticizing this choice, his argument being that users don't care about the underlying technology, they just want it to work. He thinks we should provide reasonable fallbacks so that it works for as wide of an audience as possible.

I completely agree with his basic premise: users don't care about the technology.

Users care about their experience.

I think this is something the web has ignored for far too long so I'll say it again:

Users only care about their experience.

In this case, we're not building a website with content. We're building an experience.

We didn't require Web Sockets because we're enamored with the technology, we actually require it precisely because it provides the best user experience.

The app simply doesn't feel as responsive when long-polling. There's enough of a difference in lag and responsiveness that we made the choice to eliminate the other available transports in Socket.io. (We're doing a lot more with our data transport than simply sending chats.) Additionally, we're also using advanced HTML5 and CSS3 that simply isn't available yet in IE9. It turns out that checking for WebSockets is a fairly good litmus test of the support of those other features (namely CSS3 transitions and animations). The app is just plain more fun to use because of those features.

Apple beat Microsoft by focusing on user experience. They unapologetically enforced minimum system requirements and made backward incompatible changes. Why is it considered "acceptable" to require minimum hardware (which costs money), but it's somehow not acceptable to require users to download a free browser?

I've said this over and over again: web developers who are building single-page applications are in direct competition with native applications.

If we as web developers continue to limp along support for less-than-top-notch browsers, the web will continue to lose ground to the platforms that build for user experience first. Why should we, as a small bootstrapped company invest our limited resources building less-than-ideal fallbacks?

All this, of course, depends on your audience. We created &! for small, forward-thinking teams, not necessarily their moms. :)

Realtime is becoming a central part of Internet technology.

It's sneaking it's way into our lives already with push notifications, Facebook and Google's web chats, and it's a core focus for startups like Convore, Pusher, Superfeedr, Browserling, NowJS, Urban Airship, Learnboost, our own &! (andbang), and many more.

What's most interesting to me is how accessible this is all becoming for developers. In my presentation at NodeConf I mentioned that the adoption of new technology seems directly related to how easy it is to tinker with it. So, as realtime apps get easier and easier to build, I'm convinced that we're going to see a whole slew of new applications that tap this power in new, amazing ways.

We at &yet have built five or so realtime apps in the past year, and we're super excited about this stuff. We've also discovered that there are a slew of different methods and tools for building these kinds of apps--we've used a number of them. Different developer communities have been solving the same problems with different tools and it's been amazing to see how much mindblowingly awesome code has been so freely shared. However, there's still a bit of a disconnect, because it often happens within a given dev community. We always find that we learn the most when we talk to and learn from people who are doing things differently than we are.

So what can we do to encourage more of this?

That's exactly the conversation Adam and I were having when we went to the XMPP Summit in Brussels, Belgium. That conversation culminated into a crazy idea: We should put on a conference entirely focused on realtime web stuff!

It's crazy, for a couple of reasons. First, we've never organized a conference before and secondly we're in Eastern Washington, not exactly a tech hotspot (although, we're working on that too). Luckily we're fortunate to have made some awesome friends as we've attended conferences, written blogposts and worked on pretty cool projects for our clients.

We're teaming up with Julien Genestoux and Superfeedr to make this all happen. Julien is a pioneer and incredible visionary when it comes to realtime technology. Superfeedr was one of the early startups in the realtime web world. Whether you know it or not, you've probably benefitted from superfeedr's technology while using other services like gowalla, tumblr, etsy, posterous and many more.

Together we've manage to line up a ridiculously awesome list of speakers, that we're gradually announcing. So far, we've announced Guilleremo Rauch (creator of socket.io), Leah Culver (founder of Convore and previously pownce.com), James Halliday (JS hacker and creator of dnode, browserify, and a bunch of other awesome stuff under the alias "substack"). Also, we've just added realtime veteran Jack Moffit (@metajack) and NowJS's Sridatta Thatipamala (@sridatta).

Personally, I'm way more excited about attending this event than I am being part of organizing it. These people are my heroes. We've got several more really interesting folks on the TBA list as well.

We've been getting some great advice from Chris Williams (JSConf's daddy) on how to put on a kick-ass conference. We don't know if we'll make any money, in fact, our main goal is just to not lose money. We just want to bring together all of these amazing people from various communities that are pushing the envelope of what can be done in a browser. We need to listen to each other, learn from each other and push each other to solve the problems that can make more awesome apps a possibility.

In order for attendees to get the most value possible, we're going to do a presentation track (on the top floor) and then a hack-track (on the lower floor), where the presenters can do smaller, follow-up sessions, how-to's, training, etc. Multiple hack-tracks will be going on simultaneously. The goal being for people to be able to get more in-depth knowledge on the topics that interest them most.

We're also trying hard to get representatives of various dev communities, so that no one stack is touted as the "One True Way". That's just silly. We all have our favorites, I get that, but ultimately we're better off if we learn from each other, especially from those who are not using our tools of choice. There's a whole batch of new problems to solve in building (and scaling) rich, real-time applications that work on as many devices as possible.

The details

KRTConf will be Nov. 7-8 in Portland, OR, all the details are on krtconf.com and new stuff is being announced as it happens on twitter at @krtconf and on this blog.

If you wanna be there, you can get tickets on eventbrite and if you're interested in speaking, sponsoring or otherwise being involved in the event, email Adam adam@krtconf.com or myself henrik@krtconf.com or hit us up on twitter @adambrault @henrikjoreteg.

I'm super excited to be a part of this and hopefully I'll see you there!

Quick intro, the hype and awesomeness that is Node

Node.js is pretty freakin' awesome, yes. But it's also been hyped up more than an Apple gadget. As pointed out by Eric Florenzano on his blog a LOT of the original excitement of server-side JS was due to the ability to share code between client and server. However, instead, the first thing everybody did is start porting all the existing tools and frameworks to node. Faster and better, perhaps, but it's still largely the same 'ol thing. Where's the paradigm shift? Where's the code reuse?!

Basically, Node.js runs V8, the same JS engine as Chrome, and as such, it has fairly decent ECMA Script 5 support. Some of the stuff in "5" is super handy, such as all the iterator stuff forEach, map, etc. But – and it's a big "but" indeed – if you use those methods you're no longer able to use ANY of your code in older browsers, (read "IE").

So, that is what makes underscore.js so magical. It gives you simple JS fallbacks for non-supported ECMA Script 5 stuff. Which means, that if you use it in node (or a modern browser), it will still use the faster native stuff, if available, but if you use it in a browser that doesn't support that stuff your code will still work. Code REUSE FTW!

So what kind of stuff would we want to share between client and server?

Enter Backbone.js

A few months ago I got really into Backbone and wrote this introductory post about it that made the frontpage of HN. Apparently, a LOT of other people were interested as well, and rightfully so; it's awesome. Luckily for us, Jeremy Askenas (primary author of backbone, underscore, coffeescript and all around JS magician) is also a bit of a node guru and had the foresight to make both backbone and underscore usable in node, as modules. So once you've installed 'em with npm you can just do this to use them on the server:

var _ = require('underscore')._,
    backbone = require('backbone');

So what?! How is this useful?

State! What do I mean? As I mentioned in my introductory backbone.js post, if you've structured your app "correctly" (granted, this my subjective opinion of "correct"), ALL your application state lives in the backbone models. In my code I go the extra step and store all the models for my app in a sort of "root" app model. I use this to store application settings as attributes and then any other models or collections that I'm using in my app will be properties of this model. For example:

var AppModel = Backbone.Model.extend({
  defaults: {
    attribution: "built by &yet",
    tooSexy: true
  },

  initialize: {
    // some backbone collections
    this.members = new MembersCollection();
    this.coders = new CodersCollection();

    // another child backbone model
    this.user = new User();
  }
});

Unifying Application State

By taking this approach and storing all the application state in a single Backbone model, it's possible to write a serializer/deserializer to extract and re-inflate your entire application state. So that's what I did. I created two recursive functions that can export and import all the attributes of a nested backbone structure and I put them into a base class that looks something like this:

var BaseModel = Backbone.Model.extend({
  // builds and return a simple object ready to be JSON stringified
  xport: function (opt) {
    var result = {},
      settings = _({
        recurse: true
      }).extend(opt || {});

    function process(targetObj, source) {
      targetObj.id = source.id || null;
      targetObj.cid = source.cid || null;
      targetObj.attrs = source.toJSON();
      _.each(source, function (value, key) {
        // since models store a reference to their collection
        // we need to make sure we don't create a circular refrence
        if (settings.recurse) {
          if (key !== 'collection' && source[key] instanceof Backbone.Collection) {
            targetObj.collections = targetObj.collections || {};
            targetObj.collections[key] = {};
            targetObj.collections[key].models = [];
            targetObj.collections[key].id = source[key].id || null;
            _.each(source[key].models, function (value, index) {
              process(targetObj.collections[key].models[index] = {}, value);
            });
          } else if (source[key] instanceof Backbone.Model) {
            targetObj.models = targetObj.models || {};
            process(targetObj.models[key] = {}, value);
          }
        }
      });
    }

    process(result, this);

    return result;
  },

  // rebuild the nested objects/collections from data created by the xport method
  mport: function (data, silent) {
    function process(targetObj, data) {
      targetObj.id = data.id || null;
      targetObj.set(data.attrs, {silent: silent});
      // loop through each collection
      if (data.collections) {
        _.each(data.collections, function (collection, name) {
          targetObj[name].id = collection.id;
          Skeleton.models[collection.id] = targetObj[name];
          _.each(collection.models, function (modelData, index) {
            var newObj = targetObj[name]._add({}, {silent: silent});
            process(newObj, modelData);
          });
        });
      }

      if (data.models) {
        _.each(data.models, function (modelData, name) {
          process(targetObj[name], modelData);
        });
      }
    }

    process(this, data);

    return this;
  }
});

So, now we can quickly and easily turn an entire application's state into a simple JS object that can be JSON stringified and restored or persisted in a database, or in localstorage, or sent across the wire. Also, if we have these serialization function in our base model we can selectively serialize any portion of the nested application structure.

Backbone models are a great way to store and observe state.

So, here's the kicker: USE IT ON THE SERVER!

How to build models that work on the server and the client

The trick here is to include some logic that lets the file figure out whether it's being used as a CommonJS module of if it's just in a script tag.

There are a few different ways of doing this. For example you can do something like this in your models file:

(function () {
  var server = false,
    MyModels;
  if (typeof exports !== 'undefined') {
    MyModels = exports;
    server = true;
  } else {
    MyModels = this.MyModels = {};
  }

  MyModels.AppModel...

})()

Just be aware that any external dependencies will be available if you're in the browser and you've got other <script> tags defining those globals, but anything you need on the server will have to be explicitly imported.

Also, notice that I'm setting a server variable. This is because there are certain things I may want to do in my code on the server that won't happen in the client. Doing this will make it easy to check where I am (we try to keep this to a minimum though, code-reuse is the goal).

State syncing

So, if we go back to thinking about the client/server relationship, we can now keep an inflated Backbone model living in memory on the server and if the server gets a page request from the browser we can export the state from the server and use that to rebuild the page to match the current state on the server. Also, if we set up event listeners properly on our models we can actually listen for changes and send changes back and forth between client/server to keep the two in sync.

Taking this puppy realtime

None of this is particularly interesting unless we have the ability to send data both ways – from client to server and more importantly from server to client. We build real-time web apps at &yet–that's what we do. Historically, that's all been XMPP based. XMPP is awesome, but XMPP speaks XML. While JavaScript can do XML, it's certainly simpler to not have to do that translation of XMPP stanzas into something JS can deal with. These days, we've been doing more and more with Socket.io.

The magical Socket.io

Socket.io is to Websockets what jQuery is to the DOM. Basically, it handles browser shortcomings for you and gives you a simple unified API. In short, socket.io is a seamless transport mechanism from node.js to the browser. It will use websockets if supported and fall back to one of 5 transport mechanisms. Ultimately, it goes all the way back to IE 5.5! Which is just freakin' ridiculous, but at the same time, awesome.

Once you figure out how to set up socket.io, it's fairly straightforward to send messages back and forth.

So on the server-side we do something like this on the new connection:

io.on('connection', function(client){
  var re = /(?:connect.sid\=)[\.\w\%]+/;
  var cookieId = re.exec(client.request.headers.cookie)[0].split('=')[1]
  var clientModel = clients.get(cookieId)

  if (!clientModel) {
    clientModel = new models.ClientModel({id: cookieId});
    clients.add(clientModel);
  }

  // store some useful info
  clientModel.client = client;

  client.send({
    event: 'initial',
    data: clientModel.xport(),
    templates: 
  });

...

So, on the server when a new client connection is made, we immediately send the full app state:

io.on('connection', function(client) {
  client.send({
    event: 'initial',
    data: appModel.xport()
  });
};

For simplicity, I've decided to keep the convention of sending a simple event name and the data just so my client can know what to do with the data.

So, the client then has something like this in its message handler.

socket.on('message', function (data) { 
  switch (data.event) {
    case 'initial':
      app.model.mport(data.data);
      break;
    case 'change'

      ...  

  }


});

So, in one fell swoop, we've completely synced state from the server to the client. In order to handle multiple connections and shared state, you'll obviously have to add some additional complexity in your server logic so you send the right state to the right user. You can also wait for the client to send some other identifying information, or whatnot. For the purposes of this post I'm trying to keep it simple (it's long already).

Syncing changes

JS is built to be event driven and frankly, that's the magic of Backbone models and views. There may be multiple views that respond to events, but ultimately, all your state information lives in one place. This is really important to understand. If you don't know what I mean, go back and read my previous post.

So, now what if something changes on the server? Well, one option would be to just send the full state to the clients we want to sync each time. In some cases that may not be so bad – especially if the app is fairly light, the raw state data is pretty small as well. But still, that seems like overkill to me. So what I've been doing is just sending the model that changed. So I added the following publishChange method to my base model:

publishChange: function (model, collection) {
  var event = {};

  if (model instanceof Backbone.Model) {
    event = {
      event: 'change',
      model: {
        data: model.xport({recurse: false}),
        id: model.id
      }
    }
  } else {
    console.log('event was not a model', e);
  }

  this.trigger('publish', event);
},

Then added something like this to each model's init method:

initialize: function () {
  this.bind('change', _(this.publishChange).bind(this));
}

So now, we have an event type in this case change and then we've got the model information. Now you may be wondering how we'd know which model to update on the other end of the connection. The trick is the id. What I've done so solve this problem is to always generate a UUID and set it as the id when any model or collection is instantiated on the server. Then, always register models and collections in a global lookup hash by their id. That way we can look up any model or collection in the hash and just set all our data on it. Now my client controller can listen for publish events and send them across the wire with just an id. Here's my register function on my base model (warning, it's a bit hackish):

register: function () {
  var self = this;
  if (server) {
    var id = uuid();
    this.id = id;
    this.set({id: id});
  }

  if (this.id && !Skeleton.models[this.id]) Skeleton.models[this.id] = this;

  this.bind('change:id', function (model) {
    if (!Skeleton.models[this.id]) Skeleton.models[model.id] = self;
  });
},

Then, in each model's initialize method, I call register and I have a lookup:

initialize: function () {
    this.register();    
}

So now, my server will generate a UUID and when the model is sent to the client that id will be the same. Now we can always get any model, no matter how far it's nested by checking the Skeleton.models hash. It's not hard to deduce that you could take a similar approach for handling add and remove events as long as you've got a way to look up the collections on the other end.

So how should this be used?

Well there's are three choices that I see.

  1. Send model changes from either the server or the client in the same way. Imagine we're starting with an identical state on the server and client. If we now modify the model in place on the client, the publish event would be triggered and its change event would be sent to the server. The change would be set to the corresponding model on the server, which would then immediately trigger another change event, this time on the server echoing back the change to the client. At that point the loop would die because the change isn't actually different than the current state so no event would be triggered. The downside with this approach is that it's not as fault tolerant of flaky connections and it's a bit on the noisy side since each change is getting sent and then echoed back. The advantage of this approach is that you can simply change the local model like you normally would in backbone and your changes would just be synced. Also, the local view would immediately reflect the change since it's happening locally.

  2. The other, possibly superior, approach is to treat the server as the authority and broadcast all the changes from the server. Essentially, you would just build the change event in the client rather than actually setting it locally. That way you leave it up to the server to actually make changes and then the real change events would all flow from the server to the client. With this approach, you'd actually set the change events you got from the server on the client-side, your views would use those changes to update, but your controller on the client-side wouldn't send changes back across the wire.

  3. The last approach is just a hybrid of the other two. Essentially, there's nothing stopping you from selectively doing both. In theory, you can sync the trivial state information for example simple UI state (whether an item in a list is selected or not) using method #1 and then do more important interactions by sending commands to the server.

In my experiments option 2 seems to work the best. By treating the server as the ultimate authority, you save yourself a lot of headaches. To accommodate this I simply added one more method to my base model class called setServer. It builds a change event and sends it through our socket. So now, in my views on the client, when I'm responding to a user action instead of calling set on the model I simply call setServer and pass it a hash of key/value pairs just like I would for a normal set.

setServer: function(attrs, options) {
  socket.send({
    event: 'set',
    id: this.id,
    change: attrs
  });
}

Why is this whole thing awesome?

It lets you build really awesome stuff! Using this approach we send very small changes over an already established connection, we can very quickly synchronize state from one client to the other or the server can get updates from an external data source, modify the model on the server and those changes would immediately be sent to the connected clients.

Best of all – it's fast. Now, you can just write your views like you normally would in a Backbone.js app.

Obviously, there are other problems to be solved. For example, it all gets a little bit trickier when dealing with a multiple states. Say, for instance you have a portion of application state that you want to sync globally with all users and a portion that you just want to sync with other instances of the same user, or the same team, etc. Then you have to either do multiple socket channels (which I understand Guillermo is working on), or you have to sync all the state and let your views sort our what to respond to.

Also, there's persistence and scaling questions some of which we've got solutions for, some of which, we don't. I'll save that for another post. This architecture is clearly not perfect for every application. However, in the use cases where it fits, it's quite powerful. I'm neck-deep in a couple of projects where I'm explore the possibilities of this approach and I've gotta say, I'm very excited about the results. I'm also working on putting together a bit of a real-time framework built on the ideas in this post. I'm certainly not alone in these pursuits, it's just so cool to see more and more people innovating and building cool stuff with real-time technologies. I'm thankful for any feedback you've got, good or bad.

If you have thoughts or questions, I'm @HenrikJoreteg on twitter. Also, my buddy/co-worker @fritzy and I have started doing a video podcast about this sort of stuff called Keeping It Realtime. And, be sure to follow @andyet and honestly, the whole &yet team for more stuff related to real-time web dev. We're planning some interesting things that we'll be announcing shortly. Cheers.


If you're building a single page app, keep in mind that &yet offers consulting, training and development services. Hit us up (henrik@andyet.net) and tell us what we can do to help.

If you'd asked most developers 5 years ago, most of them would have said: "Why would anyone want to write JS on the server?!" The luddites still do.

But we, at &yet, have fallen in love with node. Our particular schtick is the real-time web (see our podcast). We've been building real-time web apps for a while, mostly with XMPP and Strophe.js. Recently, however we've started using node + socket.io.

Frankly, we couldn't be happier and can't wait to see what the future holds as these technologies continue to mature.

We're humbled and excited to get to be a part of what promises to be an amazing conference. Given that it sold out in about 4 minutes, clearly we're not the only ones!

See you there!

Overview

We've been finding ourselves building more and more JS heavy apps here at &yet. Until recently, we've pretty much invented a custom app architecture for each one.

Not surprisingly, we're finding ourselves solving similar problems repeatedly.

On the server side, we use django to give us an MVC structure to follow. But there's no obvious structure to your client-side code. There are some larger libraries that give you this, but usually have a ton of widgets etc. I'm talking about solutions like Sproutcore, YUI, or Google Closure and there are toolkits like GWT and Cappuccino that let you compile other code to JS.

But for those of us who want to lovingly craft the UI exactly how we want them in the JavaScript we know and love, and yet crave quick lightweight solutions, those toolkits feel like overkill.

Recently something called Backbone.js hit my "three tweet threshold" I decided to take a look. Turns out it's a winner in my book and I'll explain why and how we used it.

The Problem

There are definitely some challenges that come with building complex, single-page apps, not the least of which is managing an ever increasing amount of code running in the same page. Also, since JavaScript has no formal classes there is no self-evident approach for structuring an entire application.

As a result of these problems new JS devs trying to build these apps typically goes through a series of realizations that goes something like this:

  1. Get all excited about jQuery and attempt to use the DOM to store data and application state.
  2. Realizing the first approach gets really tricky once you have more than a few things to keep track of and so instead you attempt to store that state in some type of JS model.
  3. Realizing that binding model changes to the UI can get messy if you call functions directly from your model setters/getters. You know you have to react to model changes in the UI somehow but not really knowing where to do those DOM manipulations and ending up something that's looking more and more like spaghetti code.
  4. Building some type of app structure/framework to solve these problems.
  5. And... finally realizing that someone's already solved many of these problems for you (and open sourced their code).

The Goals

So, how do we want our app to behave? Here are the ideals as I see them.

  1. All state/models for your app should live in one place.
  2. Any change in that model should be automatically reflected in the UI, whether that's in one place or many.
  3. Clean/maintainable code structure.
  4. Writing as little "glue code" as possible.

Enter Backbone.js

Backbone doesn't attempt to give you widgets or application objects or even really give you views. It basically gives you a few key objects to help you structure your code. Namely, Models, Collections and Views. Ultimately what it provides is some basic tools that you can use to build a clean MVC app in the client. We get some useful base objects for those and an event architecture for handling changes. Let's take a look at each of those.

The Model object

The model object just gives you a way to set and retrieve arbitrary attributes. So, all you really need to create a fully functioning and useful model is the following:

var Movie = Backbone.Model.extend({});

Now you can instantiate, and set and get attributes all you want:

matrix = new Movie();

matrix.set({
    title: "The Matrix",
    format: "dvd'
});

matrix.get('title');

You can also pass it attributes directly when you instantiate like so:

matrix = new Movie({
    title: "The Matrix",
    format: "dvd'
});

If you need to enforce that certain required attributes when you build it, you can do so by providing an initialize() function to provide some initial checks. By convention the initialize function gets called with the arguments you pass the constructor.

var Movie = Backbone.Model.extend({
    initialize: function (spec) {
        if (!spec || !spec.title || !spec.format) {
            throw "InvalidConstructArgs";
        }

        // we may also want to store something else as an attribute
        // for example a unique ID we can use in the HTML to identify this
        // item's element. We can use the models 'cid' or 'client id for this'.
        this.set({
            htmlId: 'movie_' + this.cid
        })
    }
});

You can also define a validate() method. This will get called anytime you set attributes and you can use it to validate your attributes (surprise!). If the validate() method returns something it won't set that attribute.

var Movie = Backbone.Model.extend({
    validate: function (attrs) {
        if (attrs.title) {
            if (!_.isString(attrs.title) || attrs.title.length === 0 ) {
                return "Title must be a string with a length";
            }
        }
    }
});

Ok, so there's a quite a few more goodies you get for free from the models. But I'm trying to give an overview, not replace the documentation (which is quite good). Let's move on.

Collections

Backbone collections are just an ordered collection of models of a certain type. Rather than just storing your models in a JS Array, a collection gives you a lot of other nice functionality for free. Functionality such as conveniences for retrieving models and a way to always keep in sorted according to the rules you define in a comparator() function.

Also, after you tell a collection which type of model it holds then adding a new item to the collection is a simple as:

// define our collection
var MovieLibrary = Backbone.Collection.extend({
    model: Movie,

    initialize: function () {
        // somthing
    }
});

var library = new MovieLibarary();

// you can add stuff by creating the model first
var dumb_and_dumber = new Movie({
    title: "Dumb and Dumber",
    format: "dvd"
});

library.add(dumb_and_dumber);

// or even by adding the raw attributes
library.add({
    title: "The Big Lebowski",
    format: "VHS"
});

Again, there's a lot more goodies in Collections, but their main thing is solving a lot of the common problems for maintaining an ordered collection of models.

Views

Here's where your DOM manipulation (read jQuery) takes place. In fact, I use that as a compliance check: The only files that should have jQuery as a dependency are Views.

A view is simply a convention for drawing changes to a model to the browser. This is where you directly manipulate the HTML. For the initial rendering (when you first add a new model) you really need some sort of useful client-side templating solution. My personal biased preference is to use ICanHaz.js and Mustache.js to store and retrieve them. (If you're interested there's more on ICanHaz.js on github.) But then, your view just listens and responds to changes in the model.

Here's a simple view for our Movie items:

var MovieView = Backbone.View.extend({
    initialize: function (args) {
        _.bindAll(this, 'changeTitle');

        this.model.bind('change:title', this.changeTitle);
    },

    events: {
        'click .title': 'handleTitleClick'
    },

    render: function () {
        // "ich" is ICanHaz.js magic
        this.el = ich.movie(this.model.toJSON());

        return this;
    },

    changeTitle: function () {
        this.$('.title').text(this.model.get('title'));
    },

    handleTitleClick: function () {
        alert('you clicked the title: ' + this.model.get('title'));
    }
});

So this view handles two kinds of events. First, the events attribute links user events to handlers. In this case, handling the click for anything with a class of title in the template. Also, this just makes sure that any changes to the model will automatically update the html, therein lies a lot of the power of backbone.

Putting it all together

So far we've talked about the various pieces. Now let's talk about an approach for assembling an entire app.

The global controller object

Although you may be able to get away with just having your main app controller live inside the AppView object, I didn't like storing my model objects in the view. So I created a global controller object to store everything. For this I create a simple a singleton object named whatever my app is named. So, to continue our example I might look something like this.

var MovieAppController = {
    init: function (spec) {
        // default config
        this.config = {
            connect: true
        };

        // extend our default config with passed in object attributes
        _.extend(this.config, spec);

        this.model = new MovieAppModel({
            nick: this.config.nick,
            account: this.config.account,
            jid: this.config.jid,
            boshUrl: this.config.boshUrl
        });
        this.view = new MovieAppView({model: this.model});

        // standalone modules that respond to document events
        this.sm = new SoundMachine();

        return this;
    },

    // any other functions here should be events handlers that respond to
    // document level events. In my case I was using this to respond to incoming XMPP
    // events. So the logic for knowing what those meant and creating or updating our
    // models and collections lived here.
    handlePubSubUpdate: function () {};
};

Here you can see that we're storing our application model that holds all our other models and collections and our application view.

Our app model would in this example would hold any collections we may have, as well as store any attributes that our application view may want to respond to:

var MovieAppModel = Backbone.Model.extend({
    initialize: function () {
        // init and store our MovieCollection in our app object
        this.movies = new MovieCollection();
    }
});

Our application view would look something like this:

var MovieAppView = Backbone.View.extend({
    initialize: function () {
        // this.model refers the the model we pass to the view when we
        // first init our view. So here we listen for changes to the movie collection.
        this.model.movies.bind('add', this.addMovie);
        this.model.movies.bind('remove', this.removeMovie);
    },

    events: {
        // any user events (clicks etc) we want to respond to
    },

    // grab and populate our main template
    render: function () {
        // once again this is using ICanHaz.js, but you can use whatever
        this.el = ich.app(this.model.toJSON());

        // store a reference to our movie list
        this.movieList = this.$('#movieList');

        return this;
    },

    addMovie: function (movie) {
        var view = new MovieView({model: movie});

        // here we use our stored reference to the movie list element and
        // append our rendered movie view.
        this.movieList.append(view.render().el);
    },

    removeMovie: function (movie) {
        // here we can use the html ID we stored to easily find
        // and remove the correct element/elements from the view if the 
        // collection tells us it's been removed.
        this.$('#' + movie.get('htmlId')).remove();
    }
});

Ok so now for a snapshot of the entire app. I've included all my dependencies, each in their own file (read notes on this below). We also include the ICanHaz.js templates. Then on $(document).ready() I would simply call the init function and pass in whatever variables the server-side code may have written to my template. Then we draw our app view by calling its render() method and appending that to our <body> element like so:

<!DOCTYPE html>
<html>
    <head>
        <title>Movies App</title>

        <!-- libs -->
        <script src="jquery.js"></script>
        <script src="underscore.js"></script>
        <script src="backbone.js"></script>

        <!-- client templating -->
        <script src="mustache.js"></script>
        <script src="ICanHaz.js"></script>

        <!-- your app -->
        <script src="Movie.js"></script>
        <script src="MovieCollection.js"></script>
        <script src="MovieView.js"></script>
        <script src="MovieAppModel.js"></script>
        <script src="MovieAppView.js"></script>
        <script src="MovieAppController.js"></script>

        <!-- ICanHaz templates -->
        <script id="app" type="text/html">
            <h1>Movie App</h1>
            <ul id="movies"></ul>
        </script>

        <script id="movie" type="text/html">
            <li id="{{ htmlId }}"><span class="title">{{ title }}</span> <span>{{ format }}</span></li>
        </script>

        <script>        
            $(document).ready(function () {
                // init our app
                window.app = MovieAppController.init({
                    account: '{{ user.get_profile.account.slug }}',
                    // etc, etc
                });

                // draw our main view
                $('body').append(app.view.render().el); 
            });     
        </script>

    </head>
    <body></body>
</html>

All said and done, now if we add or remove any movies from our collection or change their titles in the model those changes will just be reflected in the HTML like magic. With all the proper behaviors you'd defined for them in your MovieView.

General Tips

  1. Store all your objects in their own files for development.
  2. Compress and minify all your JS into one file for production.
  3. Use JSLint
  4. Extend underscore.js for any of your global utility type functions instead of modifying native objects. More on that in this gist.
  5. jQuery should only be used in your views. That way you can potentially unit test your model and collections on the server side.
  6. Use generic jQuery events to signal other views in your app. That way you don't tightly couple them.
  7. Keep your models as simple as possible.

I know I've covered a lot of stuff here, and some of it duplicates what's available in the backbone.js documentation but when I started working on building an app with it, I would have wanted a complete, yet somewhat high-level walkthrough like this. So I figured I'd write one. Big props to DocumentCloud and Jeremy Ashkenas for creating and sharing backbone with us.

If you have any thoughts, comments or reactions I'm @HenrikJoreteg on twitter. Thanks and good luck.


If you're building a single page app, keep in mind that &yet offers consulting, training and development services. Hit us up (henrik@andyet.net) and tell us what we can do to help.

We make web software for human people.
(And have a nearly inappropriate amount of fun doing it.)

Blog feedFollow us on Twitter