Real-world real-time web apps with Python equals Django + Orbited + Twisted

December 17, 2009

Some things are better in real-time. Day-to-day activities such as conversations, multi-user games, and collaborative projects are far more exciting and productive when immediate feedback from others is available. Many web applications could gain significant new value by adding real-time features.

Currently though, it’s a significant challenge to create real-time web applications. In this post I’ll give a Python-centric overview of what exists now, what is being done to improve things, and conclude with some details of a project I’m working on called Hotdot that is focused on creating real-time webapps with Django, Orbited, and Twisted.

Comet, Orbited, and the current state of real-time web applications

Comet is a catch-all term for the multitude of methods to “simulate” real-time communication in the web browser. See the Wikipedia page on Comet to understand all the permutations of the Comet technique. Fact of the matter is, Comet works pretty good - certaintly ‘good enough’ if you do it right.

Comet is cool, and it’s the best we have right now (although the future looks very bright). To appreciate why Comet is challenging (more than say standard Ajax apps), it helps to understand what makes Comet more complex to implement: <div class="quote"> The complixity in Comet comes from the fact you need both the client (a Comet JavaScript client) and the server (preferably an asynchronous server) that intimaltely know and work in conjunction with one another. </div>

This is what you get with Orbited - both the javascript client and the asyncronous server. Orbited is a big step forward in easing the difficulty of creating a Comet application. But what Orbited (mostly) does not provide is a clear path towards hooking this into a real-world application. By real-world I mean:

  • Users (Authentication + Permissions. Real-time web apps are commonly user-centric)
  • CRUD (Standard Create/Retrieve/Update/Delete logic sites always have. The bread and butter of frameworks like Django)
  • Real-time (The new and interesting piece. How does this smoothly fit in with the Users and CRUD?)

Orbited isn’t the only solution out there, but for Python it’s one of the best. Many people are using it in production with good success, and interest is definitely on the upswing.

Interest in real-time web apps is growing, and Python has the goods.

Orbited isn’t the only player in Python real-time web app land. Most notably, there is Tornado, which is the ‘real-time web framework’ used to build FriendFeed. Tornado is minimal and clean, and does include some very nice features beyond just being a non-blocking webserver.

Here is an excerpt from a comment on the python-tornado mailing list by one of Tornado’s main developers Bret Taylor: <div class="quote"> I hope that Tornado evolves more along the lines of Django in terms of focus (web development) than Twisted (networking). </div>

But what if we could have things both ways? That is, what if we could have a robust and featureful web development environment, and networking library under the covers of our real-time web framework? I believe we do have that if Django + Orbited + Twisted are combined correctly. And furthermore, we would have a real-time web framework with a greater community and total features than can be found in Tornado. The sum is greater than the parts in this case.

Scaling Comet down and up - Orbited plays well with Django and Twisted.

Developers love to talk about scaling up. It’s a topic that is synomunous with success, and it’s extremely important at the right time. Taking into account the relative number or developers that just need to get stuff done for their small to medium sized application, compared to those that need to support apps with huge growth rates and demand. But probably of most importance is this:

Scaling down brings mass interest, and thus more mindshare and adoption.

An issue that I still see with Comet is it does not easily scale down, or there aren’t ‘frameworks’ to speed development and reduce boilerplate. Here is where I really see the combination of Django, Orbited, and Twisted being hugely beneficial. Each application alone has a large set of awesome features, but when put together, they do in fact contain all the necessary parts to build a real-time web framework:

  • Django: Excellent web framework for creating the backbone of a great web application.
  • Orbited: Real-time web (Comet) library to build the real-time components with.
  • Twisted: Scalable asynchronous network library, for serving Orbited (and Django too, with WSGI!)

In Django we have the default WSGI server and the SQLite database that aid with development. Similarily, with Django, Orbited, Twisted, we can scale it nicely down by using Twisted to run both Orbited and Django (using the new and excellent twisted.web.wsgi support), as well as use MorbidQ as the Queue for Orbited messages. This is a extremely important enabler for developers. We need to get started quickly and be convinced that the effort we are about to put in is worth our time.

And not to fear, there exists well known strageties to scale up. This could involve replacing MorbidQ with the RabbitMQ Stomp Gateway, you can scale Orbited out horizontally (twisted.web processes) scale Django up using one of the many server configurations and the Database, well, just throw more RAM at it until you get really big :-).

There is a time and place for scaling up. Before that time it’s far better to build a great app, with awesome features that people actually want.

BOSH and XMPP - another important real-time web app stack

I do want to mention another “stack” of technologies that are very good, and that is the excellent javascript XMPP library Strophe in combination with an XMPP server like ejabberd The guys from stanziq even provide a very nice full-featured example to get started with, and I definitely recommed checking it out.

I’ve built an XMPP based app using Strophe, Ejabberd, and Django and the experience was (and still is) a fun challenge.

One thing I felt was lacking in this stack is a pure Python XMPP server (just as Orbited has MorbidQ) to avoid the extra complexity of the larger, more robust and full-featured XMPP servers, and to aid in development and small scale deployments. There are other complexities too, like understanding XMPP (which does pays off, but has a learning curve), and choosing how you do BOSH, using either Punjab or the ejabberd http_bind support.

Taking what I’ve learned from the above mentioned stack of technologies, I’ve recently began to explore what is possible with another stack of technologies consisting of Django, Orbited and Twisted.

Introducing Hotdot: Real-time webapps using Django + Orbited + Twisted

I’ve been using and experimenting with different Comet technologies for a couple years now, and although there’s been some great progress, there’s still room for improving the process of creating a real-time web app in Python. I’ve set out creating Hotdot, as a real-world working example that demostrates combining Django, Orbited, and Twisted to make something ‘real-world’. Additionaly, I hope to gain enough understanding of the problem space to generalize the code, so others can begin to plug in their logic, and create apps of their own.

The code of Hotdot is currently just a glorified example, and has some very nice parts, as well as some crufty parts, and certaintly will be improving in features and generality. Since Django provides, as the introductory tutorial, a project based on Polls and voting, I decided to following this pattern with Hotdot. So, as of now, Hotdot is mainly an example: “Real-time Voting Booth”

Some features I’m maintaining for Hotdot right now:

  • As simple as possible, but be as close to ‘real-world’ as possible.
  • Fundamental features of a ‘real-world’ web app:
    • Users: Login, basic security and data access/restriction.
    • CRUD: Create, Retrieve, Update, Delete all Polls.
    • Real-time aspect: Real-time Edit + Notification of Voting, and Chat.
  • No external dependancies beside pure Python packages (Running Django from twisted.web.wsgi a huge win here)

Keep in mind Hotdot is not yet a ‘full featured real-time web app solution’, it’s more of a solid move in that direction. I’ll possibly be writing a more detailed post on Hotdot, especially as it improves, but for now the best way to learn more is to get Hotdot from Github, and have some real-time fun.