Google Drive Realtime API - First Thoughts

Technologies tend to arrive with a bang of hyperbole, and then settle into the valley of despair before reaching the plains of enlightenment. It's a well known process. Look it up.

Google's Realtime API is probably going to buck that trend. Well, unless a lot more people read my blog than I know about, because this is a technology that is so sufficiently advanced it's essentially magic. But few people know about it.

I'm assuming you've used Google Docs at some point, and had the pleasure of watching other peoples' cursors (or your own) wandering through the document. Collaborative text editing is older than Sub-etha-edit, but Docs took it to the point where it "just worked" across the entire Internet. Do not underestimate that achievement.

The Realtime API is that.

In short, you can write Javascript web pages that hold a shared 'document' object, and every copy of that data structure in every other browser in the world updates to include any changes.

Seriously, stop and think about that. Text boxes which update when anyone changes their content. We used to laugh about such things being depicted in bad 80's action movies, but that's now state-of-the-art.

I have made my own attempts at such a technology, which is why I find their solution to be so familiar. And I also understand the limitations... that while the API does its best to appear magical in operation, you pay for it in other ways: atomicity being the primary one.

So before going into it's strengths, let's go over the weaknesses of the OT (Operational Transform) approach, in order to better dance around the landmines.

The big one: binary data. OT depends on having a structured understanding of the data it's transforming - it wants to be 'git' (the version control system) but without the possibility of ever having 'unresolved edit conflicts' that require manual intervention.

Binary blobs are - by definition - unstructured, and the realtime API cannot patch large blocks of binary data without fundamentally stepping all over the toes of everyone else trying to do the same thing. the upshot: patches can't be combined, so changes go missing.

So, don't keep BLOBs in Realtime.

A DOM-like object tree is the exact opposite. It is so structured that every branch insertion, every node deletion, can be tracked as a separate "mutation". That's great! The OT system has a stream of micro-operations that can be 'transformed' against each other in a more granular way. Google sat down and figured out the full "theory of patches" for that limited case.

Text strings are a kind of half-way between the two, and one where the OT 'rules' are simple enough that most programmers could sit down and work them out in a half hour.

Creating an OT 'grammar' is necessary for each datatype. The rules which work when combining text edits in a "description" box are not adequate to make sure that two "legal' edits to a JSON string result in a syntax legal combination. The strings may combine to produce invalid JSON... bad if that field is storing program config data.

If you know that two binary blobs represent GIF images, then extracting the 'differences' between two versions (with the intention of applying the 'difference' to a third image) is a simple set of photoshop operations. Without that knowledge, a 'standard binary merge' is only going to corrupt the GIF file.

Clearly, the OT rules for combining images are useless for combining XML data. Every datatype needs an OT definition, and it's not proven (or possibly proveable) that all datatypes can have one.

The academic area that looks into this is called the "Theory of Patches". If you read the papers in the hopes of finding a solution, what you tend to get is "Oh no, it is so much worse than you thought... have you considered these pathological merge cases?" and then your head hurts.

The best thing about the "Theory of Patches" is that, in academic style, at least it lays out the general shape of the minefield, and mentions some particularly impressive craters from past attempts to get through it.

For the moment, the Drive API only has built-in rules for three datatypes: Strings, Lists, and Maps. ('Custom' objects are possible, but they're really Maps in disguise) And frankly, Lists are a pain in the ass.

But since you can build pretty much any data tree structure you want out of those, you're generally good. And by doing so, your 'document model' is granular enough that OT can merge your micro-patches with everyone else's version and keep that tree in synch.

Then there are the consequences... because you have a data structure that doesn't only change when you tell it to, but when any bloody person in the world does. What happens when, three steps into a dialog wizard, someone else deletes the file? Well, you're going to have to code a listener for that.

There are other hard limits: 10Mb per 'realtime document', and I think it was 500k per 'patch'. But you should plan to never hit those limits: if you store bulk data in realtime, you're doing something wrong. (That's what the normal drive API is for.) Realtime is for co-ordination and structure, not streaming, and not data transfer.

Google handles all the authentication, permissions can be set for files through the usual Drive web interface, which is nice. Realtime documents get their permissions from the 'base' drive file they're attached to - like 'conversations' that can be started about any existing file. (actually, file version. If you change the base file, you invoke a new realtime branch - watch that.)

Although the OAuth sign-in process is a lot slicker than it used to be, it still has problems... Mostly caused by pop-up blockers. But that's part of a much bigger discussion I want to save for another day.

And they have automatic undo! Undoooo! Do you know how hard that is? How much of your time that saves? How happy your users will be?

What the "Realtime API" does is make Google into the biggest 'Chat Server' in the world. Every document is hooking up to its own little social networking hub, to discuss what buttons their users are pressing, and how their cursors are moving today. A billion tiny brainstorming sessions, between browsers.

There's a lot of guff being written about how people are leaving the social networks. That's fine... Google's social network isn't just for people. It's a peer message-passing layer for our software, arguably more useful and important to the internet's long-term future.

I really encourage you to start writing code that depends on this modern view. Spend time with the paradigm, learn its' flaws and graces - you can probably have a shared 'ToDo' list app working in a few hours that instantly scales to millions of users. But be prepared to let go of a lot of baggage. This is Star-Trek level technology, so don't try applying 20th century thinking. Start fresh.

Enlightenment awaits.

FireFox still missing MessageChannel

I've managed to 'find' three browser bugs in the last few weeks. One is wending it's way through the chromium bug process (it's nice when you get a good one) one was already logged, and the third was FireFox.

Well, I say "bug" but really it's "unimplemented HTML5 feature"... specifically the MessageChannel object necessary for cross-document messaging. Without those, you can still use .postMessage for everything, but you take a performance hit because all the "marshalling" required, and security goes out the window.

Oh, so FireFox has not implemented a bleeding-edge new feature. Aw, cry me a river, you might say. Except this is a rare case where even Internet Explorer has managed to get over the line, making Mozilla the last holdout (except opera mini.) and it's such a simple feature!

In fact, if you look at the great repository of knowledge on such things:
http://caniuse.com/#feat=channel-messaging
You'll see that Chrome has had it for five version, IE has had it for two, even the Blackberry browser has always had it. It's one of the easier parts of HTML5 spec, and is a fundamental part of security in future apps.

So why is it missing from FireFox?

Reading through the bugtrack is quite illuminating, and paints a winding story. I think the tone was set in the initial days by one developer saying "I don't see the point of this." and another replying "Because it's in the HTML5 spec." and the first replying, "Oh well." and then nothing happening for three months.

Recently a flurry of people reporting 'bugs' prompted some work, and the feature was mostly implemented except for web workers, and then pulled back at the last moment. From what I can tell, the patches are sitting in some kind of purgatory until someone cares enough again.

In the meantime, the fallback for FireFox is to continue using the single window.sendMessage event to route everything. Why is this bad? Perhaps the best reason I can give is that, once I attached my "onmessage" handler to the outer window to receive messages back from the inner frame, that handler was passed all inter-frame traffic... including the Google+ widget messages that I didn't even know were there... and it was clear that the Google+ widget was also receiving all the messages intended for my scripts, (filled with things like authentication tokens) so I was essentially trusting them to ignore messages intended for my origin and not be evil. (just like they were trusting me)

Google, I do trust. But what happens when I add a Facebook or other third-party web-library-component-widget things to my page? Will they start conducting industrial espionage on each other because of the globally accessible message queue? How would you know?

So it's not just me working around the missing features in FireFox, or the dozen other edge-dancing coders complaining in the forums, It's Google. How much effort is being put into coping with this out in the field?

node.js finally arrives, thanks to RedHat OpenShift

I've been messing with node.js for about two years now. It's a beautiful system. Most people don't get it, but that's fine. The bigger problem had been there was no-where to run your code, if you didn't maintain your own servers.

And who seriously wants to do that, if you're not getting paid for it? It's a time sink.

I've been waiting for a major provider like Dreamhost to enable node.js. A couple of small providers have started up, but it's kind of a premium service. And that's a shame, because the first job of node.js is not to take over completely from your old site, but initially to "plug a few gaps" in the big internet puzzle - mostly all this crazy new 'real time' websockets and messaging stuff.

...like a chat client. Everyone loves a little chat client. But you're not going to redevelop your entire site and chuck Apache, with all the pain that would cause, to get one. You want a cheap little 'experimental' side-server that fills that gap, so cheap you essentially forget about it.

Well, along comes RedHat with their cloud computing platform called "OpenShift". I think they changed it recently, because I remember it having a much more sensible name and a lot less features the last time I checked.

I'll skip over explaining their funky new words for everything (like "cartridge", clearly intended to harken back to your halcyon Nintendo days) and just say that, FOR FREE, you can have THREE servers spun up for you, pre-loaded with all kinds of linux-based server setups such as Apache, PHP, JBOSS, or node.js.

If you're used to FTPing into your hosting server, then OpenShift is going to be a shock, due to the heavy use of GIT to transfer files. It seems crazy at first, especially setting up the SSH access keys if you don't have a unix box handy, but there is a method to the madness which becomes apparent when you start to 'scale'. Using GIT to deploy files to one server seems completely excessive - although it does do some cool 'merging' of your code with the latest server upgrades - but the moment you need to deploy to half a dozen instances, you'll see there's no better way.

All other providers I've tried work in a similar way. If anything, RedHat has streamlined the process to it's simplest. But why does the typical node.js deployment system 'force' you into being scaleable? Because if you don't start out that way, you never will.

Yes, socket.io and websockets work, I've tested it. (You have to be aware it appears on port 8000 externally, regardless of what you thought it would be. And only the websocket connection method works - no comet fallback that I know of.)

Widespread use of node.js and websockets is likely to reduce global internet traffic, by removing all the inefficient ajax 'polling' or comet 'long poll' IP connections being made and remade, a billion times a minute. node.js can handle more users and page hits than PHP. But none of that mattered while we didn't have any damn servers to run the code on.

So, Thanks RedHat! You've made my day, and solved one of my longest-standing network plumbing problems, for free. And you've made the internet better. Thank you.

Steam Family Sharing renegotiates copyright in our favour

Steam (the video-game distribution service) just renegotiated your copyright with them. And unusually, they've taken a step on the long road towards ending the pointless copyright wars, instead of escalating further.

You can now "lend out" digital games to your family and friends. It's a small step, but a good one:
http://store.steampowered.com/sharing/

Jaron Lanier gave a talk recently where he pointed out one consequence of the "paper-thin" Digital Rights Management around DVDs, compared with the "open-access" data on CD.  "Imagine that 20 years ago, you put a CD and DVD into safe." he thought-experimented. They cost about the same when you bought them, and twenty years later you open the safe and ask "what capabilities does owning this medium offer?"

Well, with DVDs, you can do pretty much what you could do back in the day - play them on your region-locked DVD player, unskippable FBI warnings and all.

But with CDs, you can shift the music to your iPod, digital music collections, Spotify... there's more you can do with a CD today (again, legally speaking) than ever. Amazingly, the unprotected CD has increased in utility value, thanks to technology. That investment you made back in the day keeps paying off. Weird.

Valve and Steam have just done something equivalent. They "renegotiated" your copyright with them - as is allowed in that EULA you clicked through, as could be done by any media company at any time - but instead of screwing us, they have taken the opportunity to act on our behalf.

(In this way, they are outperforming many actual governments.)

Basically, you can now lend your game library to other people as easily as letting them sit down at your own computer to play a game, or loaning them the game media. (with the added advantage they can't lose or step on it, and you can yank it back any time you need to.)

What this does, once again, is increase the utility value of your Steam digital library. All those games you bought can convert into favours amongst friends - one of the hardest currencies of all.

Steam gave you more digital rights, rather than trying to restrict them even further. Isn't that refreshing? Being treated with respect? Feels good, doesn't it?

We're closer to the day that "baby's first steps" can't get yanked from YouTube because a digitally-bought-and-paid-for copy of "Finding Nemo" was on in the background. Where buying an album means you can sing your own terrible karaoke version in public, or do a heavy-metal guitar cover that many think is better than the original, without fear of prosecution.

More generally, the digital rights afforded us need to align with the way we (and our children) actually use the technology in our modern daily lives, or the copyright industry is just a way of criminalizing the entire population for being alive in the 21st century, for profit.