Caching for AOL Journals

We're continuing to work on improving the scalability of the AOL Journals servers.  Our major problem recently has been traffic spikes to particular blog pages or entries caused by links on the AOL Welcome page.  Most of this peak traffic is from AOL clients using AOL connections, meaning they're using AOL caches, known as Traffic Servers.  Unfortunately, there was a small compatibility issue that prevented the Traffic Servers from caching pages served with only ETags.  That was resolved last week in a staged Traffic Server rollout.  Everything we've tested looks good so far; the Traffic Servers are correctly caching pages that can be cached, and not the ones that can't.  We're continuing to monitor things, and we'll see what happens during the next real traffic spike.

The good thing about this type of caching is that our servers are still notified about every request through validation requests, so we'll be able to track things fairly closely, and we're able to add caching without major changes to the pages.

The down side is of course that our servers can still get hammered if these validation requests themselves go up enough.  This is actually the case even for time-based caching if you get enough traffic; you still have to scale up your origin server farm to accommodate peak traffic, because caches don't really guarantee they won't pass through traffic peaks. 

We're continuing to carefully add more caching to the system while monitoring things.  We're currently evaluating using a Squid reverse caching proxy.  It's open source and has been used in other situations -- and it would give us a lot of flexibility in our caching strategy.  We're also making modifications to our databases to both distribute load and take advantage of in-memory caching, of course.  But we can scale much higher and more cheaply by pushing caching out to the front end and avoiding any work at all in the 80% case where it's not really needed.

Tags: caching, etags, squid, scalability, traffic, performance, aol journals, blogs

Jonathan Miller: 破釜沉舟

Jonathan Miller wasn't a charismatic leader.  But he recognized the need to fundamentally change AOL's strategy, mapped out a new direction, and got people moving the way that he was pointing.  Since last year, especially, he had been executing well and really seemed to "get it".  Giving AOL's services away for free this summer forced the organization to focus on the new world rather than the old; a Rubicon crossing, or as Jon's Sifu might say, "Break the woks and sink the boats(破釜沉舟)".  And as Ted notes, the strategy that Jon architected is starting to show good results.

Given all of this, Jon's departure was a shock. Neither the communications to the rank and file nor to Jon himself were handled well.  There are plenty of rumors and speculation flying around.  I hope that the Time Warner leadership team handles the situation going forward with the openness and honesty that are due to the people who have worked so hard to turn AOL around.

Jon, you'll be missed.

The AIM Network: AIM6, AIMPages, Buddy Feeds

I love launches.  This week, we're launching AIM 6 along with a major upgrade to the AIMPages beta.  And they work together!

The new AIM is a big improvement; I've been running the various betas over the past many months and they've been both rising in quality and slimming down in footprint.  And the UI is finally reasonable: I can now once again edit my buddy list right there in the main window.  And buddies now have a little (i) icon that tells you when your buddy has published something -- anywhere.  Like blog entries!  Profile updates!  Or, if they've set things up, new Flickr photos, Diggs, Myspace or Blogger or Xanga updates, or any custom Atom or RSS feeds you care to add.

There are some problems:  It keeps telling me about their away message status, and I don't really care that Kevin Lawver is away at lunch.  And I see that Kevin posted a photo but there's no thumbnail in the feed...  but this is a first release, we can fix these nits.  (The feed data is available at http://buddyfeed.oscar.aol.com/rss-push/aol:buddy_feed?request=user&sn=<screenname>, and as an AIMPage module named "What's New" which I demoed at Widgets Live last week.)

There's a good review of all of this on GigaOM by Liz Gannes.

The only downside of all this is that I'm mostly on a Mac these days, and there's no Mac AIM 6 client right now.  On the other hand, Adium plus a feed reader works pretty well too.

AIMPages has also added AIM Pages Buddies.  I don't think this is the best name, but the concept is good.  It's two way, meaning that both people have to opt in to it.  And by default your AIM Pages Buddies are shown on your profile (so you really don't want your whole Buddy List showing up there).  The invitation mechanism is easy:  When you start to add, the system sends an IM to the buddy asking if they agree.  A lot better than email, if they're online.  If not, they'll get reminded with a little status link at the top of their AIM Page: "Buddy Requests (42)".

Aside:  By default, for newly created profiles, only your AIM Pages Buddies can post comments on your profile.  You can change this in the settings to open it up if you want to.  On the other hand, it's also a way to get more buddy requests.

Tags: aim, aim6, aimpages, buddy feeds, social networks, launch

Gold stars for good feed readers!

Over the past few weeks, we've been rolling out support for proper cache control on the Journals blogging platform, which will help reduce both bandwidth and database load, and also make things faster in general.  Last week we added ETag based cache control for both Atom and RSS feeds.  Today I took a quick peek at our logs to see how things are going...

The good news is that a lot of feed readers are being great citizens.  Around 33% of our feed requests get satisfied with 304 responses, meaning that clients only need to do a quick validation that they have the latest content, rather than fetching everything all over again.  Here's a quick list of feed readers, in no particular order, which are doing the right thing with our servers.  Gold stars for everybody!

• Bloglines (http://www.bloglines.com)
• FeedDemon/2.0
• NewsGatorOnline/2.0
• NetNewsWire/2.0
• LiteFeeds/2.0
• http://www.squeet.com
• Firefox/1.5.0.7
• Thunderbird/1.5.0.8
• FacebookFeedParser/1.0 (UniversalFeedParser/4.1;) +http://facebook.com/
• Windows-RSS-Platform/1.0
• LiveJournal.com
• Planet GBT +http://planetgbt.priyadi.net Planet/1.0~pre1-terasi +http://www.planetplanet.org UniversalFeedParser/3.3 +http://feedparser.org/
  Google Desktop

Pretty clearly, the UniversalFeedParser library should get a lot of the credit.  It's great that the Windows RSS Platform is doing the right thing, considering the likely amount of traffic it's going to generate.

Surprisingly, we actually have a lower cache hit ratio for Atom feeds than RSS ones... mostly due to one major crawler that seems to prefer Atom feeds and never gets a 304, presumably because it's never sending an If-None-Match header:

• Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)

Hey Google: It would really help the stats if you guys could support ETags and If-None-Match.

Tags: feeds, caching, etags, gold star, google-why-oh-why

Web 2.0: 2 Quick Takes

• Digg Labs' Swarm demo is hypnotic.
• Marissa Mayer is all about the data.  Also, she has hypnotic sparkly things hanging from her belt.
  

Launching AOL Developer Network

We've been talking about the launch of our new dev.aol.com site all week.  The goal is to provide a place to talk about our open services in one standard place.  (We've tried the strategy of hiding our APIs in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the leopard', but it didn't work out too well.)

We really want to get feedback on both the site itself and the APIs that we're exposing.  Both are evolving rapidly and I anticipate that we'll be adding some new APIs there in the very near future -- there are a couple I'm going to be pushing for.  So please, give us feedback, or just link to dev.aol.com and we'll pick it up.

Web 2.0, Sudoku, and EC2

I'm at Web 2.0 now -- it took secret ninja moves to actually get a pass, even though AOL is a triple iridium sponsor.  The place is so thick with VPs you can barely reach the free food.

In breaking news, our own Michael Chowla just won HCL's Sudoku contest and will be taking home a very nice trophy.  I'll add the picture... as soon as the network actually lets my camera phone upload it... oh darn.  Apparently the Ning demo suffered from some network issues earlier today too.  Infrastructure!

Speaking of infrastructure:  Amazon's Elastic Compute Cloud looks very cool.  You could run a startup now with nothing more than a laptop and a table at a WiFi-enabled cafe.

Tags: web20, sudoku, ec2, amazon, infrastructure

AIM Pages Blog Widgets

Just finished the Blog sidebar widgets panel at Widgets Live!.  I was the only one foolish enough to try a live demo... which of course crashed and burned.  I'm so glad I made a backup animation to show instead.  I'm also told that I was nearly inaudible during most of the presentation.  Note to self: Lean into the mike! Also, I forgot my VGA converter cable and had to jump off the platform right as we started to go grab it. 

On the other hand, the panel discussion was fun.  We're certainly in early stages; we had four people on the panel, and four completely separate terms for essentially the same things (widgets, gadgets, modules, parts).

Link: Widgets Live! AIM Pages Blog Widgets Presentation (html).

At Widgets Live!

It's pretty packed and I'm in the back next to the power outlets.  It was great to see Alex Russell mention AOL's support for Dojo this morning.  I've been struggling a bit with the WiFi -- it's better than at some conferences, but something's definitely wrong when it takes 2 seconds to ping the router.  It's still a little slow even now but not horrible:

PING 172.17.0.1 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: icmp_seq=0 ttl=64 time=188.881 ms
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=135.392 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=74.55 ms
64 bytes from 172.17.0.1: icmp_seq=3 ttl=64 time=63.111 ms
64 bytes from 172.17.0.1: icmp_seq=4 ttl=64 time=237.2 ms

REST Web Services: The Book

Something to look for on Amazon soon: REST Web Services.  The outline looks good; I would love to have a book to hand to people who want to find out about REST.

Especially Chapter 8: Resource-Oriented versus Service-Oriented Architectures:

The main event. We single-handledly take on the SOA, WSDL, and the WS-* stack: an enormous, multi-billion-dollar project that has alrady had about 25 books written about it. We believe this conglomorate will ultimately end up like CORBA, the OSI protocol suite, ATM, and other overengineered big-money flops; and that simpler, more flexible solutions will swallow anything good to come out of it.

In the spirit of comity and friendship, following the smackdown will be a section describing what REST services can learn from the SOA and take from the WS-* technologies.

Tags: REST, Web Services

Widgets Live!: Blog sidebar widgets

I'll be on a Blog sidebar widgets panel at Widgets Live! next week, doing a short demo and discussing the nuts and bolts of how to more easily hook up widgets and blogs.  Widgets are to web pages what feeds are to HTTP -- a technical shift that opens up new possibilities and is a potential game changer.  It should be fun, barring demo gremlins.

(Feed)

Tags: widgets, widgets live, blogs, panel

Journals R9 Update: Shiny!

Stephanie just blogged about our latest Journals R9 update, which has escaped from beta and is currently rampaging throughout our production servers on journals.aol.com.  The most visible change is the new, less blue color scheme.  There are a lot of less visible changes.  One of them is that we now support HTTP authentication for both RSS and Atom feeds -- so if you are a reader of someone's private Journal and use a feed reader that supports it, you can get their daily thoughts delivered to your computer automatically, just like you can for public Journals.  Does your feed reader support it?  If it's web based, probably not.  Otherwise, it probably does.

Here's a bonus Easter Egg, just for fun.  You can apply a theme to any Journals page by adding an argument "?skin=css url" to the page URL.  Like this.  Or this, or this... or anything over at the AIMPages themes directory.  It shows a direction we're headed; not the end of a journey, but the start of a trip.

We've also updated our whitelist to allow more cool widgets to be put in your posts and About Me box.  I just added one of the neater ones -- MyBlogLog's Recent Readers widget.  I've put it on my sidebar as an experiment; is it cool, or just weird, to see your visits being tracked and displayed?  (Hi Frank!)