In various presentations throughout 2007, the Twitter team has made reference to a pure Ruby message queue server called Starling, written by our own Blaine Cook.
Starling is at the core of what we do at Twitter; it moves small messages around to daemons that work on jobs like processing updates, delivering messages, archiving user accounts, and so forth. An asynchronous messaging solution is becoming a necessity for big web applications, and Starling fits the particular needs we have at Twitter. It's fast, it's stable, it speaks the memcache protocol so it doesn't need a new client library, and it's disk-backed for persistence. When other parts of the Twitter site go down, Starling stays up. It's a champ, and we love it.
Until now, Starling has lived a sheltered life in the Twitter code base. We're happy to announce that Starling is now open source and freely available for anyone to use, modify, and improve. We're eager to see patches and to start a proper open source community around Starling.
To give Starling a try today, just sudo gem install starling on your favorite Ruby development box. Let's see some serious queues!
Wednesday, January 16, 2008
Announcing Starling
Subscribe to: Post Comments (Atom)
5 comments:
Good job on getting this going!
looking forward to seeing more starling in the wild.
Thumbs up to using the memcache protocol!
My main worry about Twitter when I first saw it was that it would not scale. Glad I was proved to be wrong!
Are the rdocs online anywhere? Maybe put them on http://starling.rubyforge.org/
I've been reading through the source, and was wondering what the normal lifecycle of a message queue was for you guys. From my reading, it looks like the transaction logfile on disk continues to grow until you call the 'rotate_log' method on the queue, but I don't see anywhere in the codebase where that happens.
How do you clean up stale queues and prevent the logfiles from growing ever-larger? Is there a background cleanup process which truncates them?
Post a Comment