Caching for AOL Journals

We're continuing to work on improving the scalability of the AOL Journals servers.  Our major problem recently has been traffic spikes to particular blog pages or entries caused by links on the AOL Welcome page.  Most of this peak traffic is from AOL clients using AOL connections, meaning they're using AOL caches, known as Traffic Servers.  Unfortunately, there was a small compatibility issue that prevented the Traffic Servers from caching pages served with only ETags.  That was resolved last week in a staged Traffic Server rollout.  Everything we've tested looks good so far; the Traffic Servers are correctly caching pages that can be cached, and not the ones that can't.  We're continuing to monitor things, and we'll see what happens during the next real traffic spike.

The good thing about this type of caching is that our servers are still notified about every request through validation requests, so we'll be able to track things fairly closely, and we're able to add caching without major changes to the pages.

The down side is of course that our servers can still get hammered if these validation requests themselves go up enough.  This is actually the case even for time-based caching if you get enough traffic; you still have to scale up your origin server farm to accommodate peak traffic, because caches don't really guarantee they won't pass through traffic peaks. 

We're continuing to carefully add more caching to the system while monitoring things.  We're currently evaluating using a Squid reverse caching proxy.  It's open source and has been used in other situations -- and it would give us a lot of flexibility in our caching strategy.  We're also making modifications to our databases to both distribute load and take advantage of in-memory caching, of course.  But we can scale much higher and more cheaply by pushing caching out to the front end and avoiding any work at all in the 80% case where it's not really needed.

Tags: caching, etags, squid, scalability, traffic, performance, aol journals, blogs

Jonathan Miller: 破釜沉舟

Jonathan Miller wasn't a charismatic leader.  But he recognized the need to fundamentally change AOL's strategy, mapped out a new direction, and got people moving the way that he was pointing.  Since last year, especially, he had been executing well and really seemed to "get it".  Giving AOL's services away for free this summer forced the organization to focus on the new world rather than the old; a Rubicon crossing, or as Jon's Sifu might say, "Break the woks and sink the boats(破釜沉舟)".  And as Ted notes, the strategy that Jon architected is starting to show good results.

Given all of this, Jon's departure was a shock. Neither the communications to the rank and file nor to Jon himself were handled well.  There are plenty of rumors and speculation flying around.  I hope that the Time Warner leadership team handles the situation going forward with the openness and honesty that are due to the people who have worked so hard to turn AOL around.

Jon, you'll be missed.

The AIM Network: AIM6, AIMPages, Buddy Feeds

I love launches.  This week, we're launching AIM 6 along with a major upgrade to the AIMPages beta.  And they work together!

The new AIM is a big improvement; I've been running the various betas over the past many months and they've been both rising in quality and slimming down in footprint.  And the UI is finally reasonable: I can now once again edit my buddy list right there in the main window.  And buddies now have a little (i) icon that tells you when your buddy has published something -- anywhere.  Like blog entries!  Profile updates!  Or, if they've set things up, new Flickr photos, Diggs, Myspace or Blogger or Xanga updates, or any custom Atom or RSS feeds you care to add.

There are some problems:  It keeps telling me about their away message status, and I don't really care that Kevin Lawver is away at lunch.  And I see that Kevin posted a photo but there's no thumbnail in the feed...  but this is a first release, we can fix these nits.  (The feed data is available at http://buddyfeed.oscar.aol.com/rss-push/aol:buddy_feed?request=user&sn=<screenname>, and as an AIMPage module named "What's New" which I demoed at Widgets Live last week.)

There's a good review of all of this on GigaOM by Liz Gannes.

The only downside of all this is that I'm mostly on a Mac these days, and there's no Mac AIM 6 client right now.  On the other hand, Adium plus a feed reader works pretty well too.

AIMPages has also added AIM Pages Buddies.  I don't think this is the best name, but the concept is good.  It's two way, meaning that both people have to opt in to it.  And by default your AIM Pages Buddies are shown on your profile (so you really don't want your whole Buddy List showing up there).  The invitation mechanism is easy:  When you start to add, the system sends an IM to the buddy asking if they agree.  A lot better than email, if they're online.  If not, they'll get reminded with a little status link at the top of their AIM Page: "Buddy Requests (42)".

Aside:  By default, for newly created profiles, only your AIM Pages Buddies can post comments on your profile.  You can change this in the settings to open it up if you want to.  On the other hand, it's also a way to get more buddy requests.

Tags: aim, aim6, aimpages, buddy feeds, social networks, launch

Gold stars for good feed readers!

Over the past few weeks, we've been rolling out support for proper cache control on the Journals blogging platform, which will help reduce both bandwidth and database load, and also make things faster in general.  Last week we added ETag based cache control for both Atom and RSS feeds.  Today I took a quick peek at our logs to see how things are going...

The good news is that a lot of feed readers are being great citizens.  Around 33% of our feed requests get satisfied with 304 responses, meaning that clients only need to do a quick validation that they have the latest content, rather than fetching everything all over again.  Here's a quick list of feed readers, in no particular order, which are doing the right thing with our servers.  Gold stars for everybody!

• Bloglines (http://www.bloglines.com)
• FeedDemon/2.0
• NewsGatorOnline/2.0
• NetNewsWire/2.0
• LiteFeeds/2.0
• http://www.squeet.com
• Firefox/1.5.0.7
• Thunderbird/1.5.0.8
• FacebookFeedParser/1.0 (UniversalFeedParser/4.1;) +http://facebook.com/
• Windows-RSS-Platform/1.0
• LiveJournal.com
• Planet GBT +http://planetgbt.priyadi.net Planet/1.0~pre1-terasi +http://www.planetplanet.org UniversalFeedParser/3.3 +http://feedparser.org/
  Google Desktop

Pretty clearly, the UniversalFeedParser library should get a lot of the credit.  It's great that the Windows RSS Platform is doing the right thing, considering the likely amount of traffic it's going to generate.

Surprisingly, we actually have a lower cache hit ratio for Atom feeds than RSS ones... mostly due to one major crawler that seems to prefer Atom feeds and never gets a 304, presumably because it's never sending an If-None-Match header:

• Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)

Hey Google: It would really help the stats if you guys could support ETags and If-None-Match.

Tags: feeds, caching, etags, gold star, google-why-oh-why

Web 2.0: 2 Quick Takes

• Digg Labs' Swarm demo is hypnotic.
• Marissa Mayer is all about the data.  Also, she has hypnotic sparkly things hanging from her belt.
  

Launching AOL Developer Network

We've been talking about the launch of our new dev.aol.com site all week.  The goal is to provide a place to talk about our open services in one standard place.  (We've tried the strategy of hiding our APIs in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the leopard', but it didn't work out too well.)

We really want to get feedback on both the site itself and the APIs that we're exposing.  Both are evolving rapidly and I anticipate that we'll be adding some new APIs there in the very near future -- there are a couple I'm going to be pushing for.  So please, give us feedback, or just link to dev.aol.com and we'll pick it up.

Web 2.0, Sudoku, and EC2

I'm at Web 2.0 now -- it took secret ninja moves to actually get a pass, even though AOL is a triple iridium sponsor.  The place is so thick with VPs you can barely reach the free food.

In breaking news, our own Michael Chowla just won HCL's Sudoku contest and will be taking home a very nice trophy.  I'll add the picture... as soon as the network actually lets my camera phone upload it... oh darn.  Apparently the Ning demo suffered from some network issues earlier today too.  Infrastructure!

Speaking of infrastructure:  Amazon's Elastic Compute Cloud looks very cool.  You could run a startup now with nothing more than a laptop and a table at a WiFi-enabled cafe.

Tags: web20, sudoku, ec2, amazon, infrastructure