Random Note: DNA's Dark Matter

Scientific American's The Hidden Genetic Program of Complex Organisms grabbed my attention last week.  This could be the biological equivalent of the discovery of dark matter.  Basically, the 'junk' or intron DNA that forms a majority of our genome may not be junk at all, but rather control code that regulates the expression of other genes. 

The programming analogy would be, I think, that the protein-coding parts of the genome would be the firmware or opcodes while the control DNA is the source code that controls when and how the opcodes are executed.  Aside from the sheer coolness of understanding how life actually works, there's a huge potential here for doing useful genetic manipulation.  It's got to be easier to tweak control code than to try to edit firmware... (Free link on same subject: The Unseen Genome.)

Things in Need of a Feed

Syndicated feeds are much bigger than blogs and news stories; they're a platform.  A bunch of use cases, several of which actually exist in some form, others just things I'd like to see:

• Blog entries for blogs I'm interested in
• Feed of all comments on entries I've authored
• News stories matching a custom filter I've set up
• Traffic conditions on my customary route(s)
• Fedex shipping feed giving status and history for all of my packages
• Customer support feed giving status and history for all my issues (any company)
• Product safety/recall information for everything I buy
  
• Amazon feed of new books matching my preferences
• All new material by a specific author (on any blog or online source)
• Feed of new feeds, of various types:
•   Just my friends
•   Authored by people whose blogs I already subscribe to
•   Filtered on personal profile/interests
• House for sale listings
• Newly discovered prime numbers (okay, a niche audience)
• Airport flight status alerts
• Movies in my Netflix queue and recommendations
  
• Audio / video content pushed onto my iPod (Podcasting)
• Auction information

Addendum 11/11:

• Multiplayer game results feed
  
• New government publications feed
• New computer virus alerts feed (with metadata giving virus signatures)
• Book queue
  

Niche Markets

Niche markets are where it's at: Chris Anderson's The Long Tail is exactly right. The Internet not only eliminates the overhead of physical space but also, more importantly, reduces the overhead of finding what you want to near-zero. When your computer tracks your preferences and auto-discovers new content that you actually want, it enables new markets that couldn't otherwise exist.

Update 10/11: Joi Ito's take.

Network Protocols and Vectorization

Doing things in parallel is one of the older performance tricks.  Vector SIMD machines -- like the Cray supercomputers -- attack problems that benefit from doing the same thing to lots of different pieces of data simultaneously.  It's just a performance trick, but it drove the design and even the physical shape of those machines because the problems they're trying to tackle -- airflow simulation, weather prediction, nuclear explosion simulation, etc. -- are both important and difficult to scale up.  (More recently, we're seeing massively parallel machines built out of individual commodity PCs; conceptually the same, but limited mostly by network latency/bandwidth.)

So what does this have to do with network protocols?  Just as the problems of doing things like a matrix-vector multiply very, very fast drove the designs of supercomputers, the problems of moving data from one place to another very quickly, on demand drive the designs of today's network services.  The designs of network APIs (whether REST, SOAP, XML-RPC, or whatever) need to take these demands into account.

In particular, transferring lots of small pieces of data in serial fashion over a network can be a big problem.  Lots of protocols that are perfectly fine when run locally or over a LAN fail miserably when expected to deal with 100-200ms latencies on a WAN or the Internet.  HTTP does a decent job of balancing out performance/latency issues for retrieving human readable pages -- a page comes down as a medium-sized chunk of data, followed by, if necessary, associated resources such as scripts, style sheets, and binary images, which can all be retrieved in parallel/behind the scenes.  Note, that this is achieved only through lots of work on the client side and deep knowledge of the interactions between HTML, HTTP, and the final UI.  The tradeoff is complexity of protocol and implementation.

How does this apply to network protocols in general?  One idea is to carefully scrutinize protocol requests that transfer a single small piece of data.  Often a single small piece of data isn't very useful on its own.  Are there common use cases where a system will do this in a loop, perhaps serially, to get enough data to process or present to a user?  If so, perhaps it would be a good idea to think of "vectorizing" that part of the protocol.  Instead of returning a single piece of data, for example, return a variable-length collection of those pieces of data.  The semantics of the request may change only slightly -- from "I return an X" to "I return a set of X".  Ideally, the length should be dynamic and the client should be able to ask for "no more than N" on each request.

For example, imagine a protocol that requires a client to first retrieve a set of handles (say, mailboxes for a user) then query each one in turn to get some data (say, the number of unread messages).  If this is something that happens often -- for example, automatically every two minutes -- there are going to be a lot of packets hitting servers.  If multiple mailboxes are on one server, it would be fairly trivial to vectorize the second call and effectively combine the two queries into one -- call it "get mailbox state(s)".  This would let a client retrieve the state for all mailboxes on a given server, with better latency and far less bandwidth than the first option.  Of course there's no free lunch; if a client is dealing with multiple servers, it now has to group the mailboxes for each server for purposes of retrieving state.  But conceptually, it's not too huge of a leap.

There are other trade-offs.  If the "extra" data is large -- like a binary image -- it might well be better to download it separately, perhaps in parallel with other things.  If it's cacheable, but the main data isn't, it may again be better to separate it out so you can take advantage of things like HTTP caching. 

To summarize, one might want to vectorize part of a network protocol if:

• Performance is important, and network latency is high and/or variable;
  
• The data to be vectorized are always or often needed together in common use cases;
  
• It doesn't over-complexify the protocol;
• There's no other way to achieve similar performance in other ways (parallel requests, caching, etc.)

Of course, this applies to the Atom API.  There's a fair amount of vectorization in the Atom API from the start, since it's designed to deal with feeds as collections of entries.  I think there's a strong use case for being able to deal with collections of feeds as part of the Atom API as well, for all the reasons given above.  Said collections of feeds might be feeds I publish (so I want to know about things like recent comments...) or perhaps feeds I'm tracking (so I want to be able to quickly determine which feeds have something interesting, before downloading all of the most recent data).  It would be interesting to model this information as a synthetic feed, since of course that's already nicely vectorized.  But there are plenty of other ways to achieve the same result.

Office Space

How important is the physical workspace to knowledge workers generally, and software developers specifically?  Everybody agrees it's important.  Talk to ten people, though, and you'll get nine different opinions about what aspects are important and how much they impact effectiveness.  But there are some classic studies that shed some light on the subject; looking around recently, they haven't been refuted.  At the same time, a lot of people in the software industry don't seem to have heard of them.

Take the amount and kind of workspace provided to each knowledge worker.  You can quantify this (number of square feet, open/cubicle/office options).  What effects should you expect from, say, changing the number of square feet per person from 80 to 64?  What would this do to your current project's effort and schedule?

There's no plug-in formula for this, but based on the available data, I'd guesstimate that the effort would expand by up to 30%.  Why?

"Programmer Performance and the Effects of the Workplace" describes the Coding War Games, a competition in which hundreds of developers from dozens of companies compete on identical projects.  (Also described in Peopleware: Productive Projects and Teams.) The data is from the 1980's, but hasn't been replicated since as far as I can tell. The developers were ranked according to how quickly they completed the projects, into top 25%, middle 50%, and bottom 25%.  The competition work was done in their normal office environments.

• The top 25% had an average of 78 square feet of dedicated office space.
• The bottom 25% had an average of 46 square feet of dedicated office space.
• The top 25% finished 2.6 times faster, on average, than the bottom 25%, with a lower defect rate.
• They ruled out the idea that top performers tended to be rewarded with larger offices.
  

Now, whether larger workspaces improve productivity, or whether more productive people tend to gravitate to companies with larger workspaces, doesn't really matter to me as a manager.  Either way, the answer is the same: Moving from 46 square feet per person to 78 square feet per person can reduce the time to complete a project by a factor of up to 2.6x.  That's big.  (Of course there were other differences between the environment of the top 25% and the bottom 25%, but they are largely related to issues like noise, interruptions, and privacy.  It seems reasonable to assume these are correlated with people density.)

It itself, this doesn't give us an answer for the question we started out with (changing from 80 square feet to 64 square feet per person, and bumping up the people density commensurately).  As a first approximation, let's assume a linear relationship between dedicated area per person and productivity ratios.  64 is just over halfway between 46 and 78, so it seems reasonable to use half of the 2.6 factor, or 1.3, as a guesstimate.  So using this number, a project that was going to take two weeks in the old environment would take 1.3 times as long, or around two and a half weeks, in the new environment.  (In the long term, of course.)

To put this into perspective, it appears that increasing an organization's CMM level by one generally results in an 11% increase in productivity, and that the ratio of effort between worst and best real-world processes appears to be no more than 1.43.

You can't follow the numbers blindly here.  This probably depends a lot on the kind of work you actually do, and I can think of dozens of caveats.  My gut feeling is that the penalty is likely to be more like 10% than 30%, assuming you're really holding everything else (noise, interruptions, etc.) as constant as possible.  I suspect that the organizations which are squeezing people into ice cube sized cubicles are likely to be destroying productivity in other ways as well.  But, these numbers do provide some guidance as to what to expect in terms of costs and consequences of changing the workplace environment.

Links and references:

• In How office space affects programming productivity (IEEE Computer  Vol. 28 No. 1; Jan 1995, pp. 7676) Capers Jones gives a guideline of at least 80 square feet of space per person, with full walls and doors, for optimal productivity.
  
• The most well-documented planning exercise for knowledge worker facilities is IBM's Santa Teresa facility; a discussion is here.
• Steve McConnell gives a good overview of this and other issues in Quantifying Soft Factors (IEEE Software Vol. 17 No. 6: Nov/Dec 2000, pp. 9-11).
• T. DeMarco and T. Lister , "Programmer Performance and the Effects of the Workplace", Proc. 8th Int'l Conf. Software Eng., ACM Press, New York,1985,, pp. 268-272.
• A great anecdote: Joel Spolsky, Bionic Office.  He's betting a lot of money that it's effective to equip his company with spacious, private offices.
  
  
  

Community, social networks, and technology at Supernova 2004

Some afterthoughts from the Supernova conference, specifically about social networks and community.  Though it's difficult to separate the different topics. 

A quick meta-note here: Supernova is itself a social network of people and ideas, specifically about technology -- more akin to a scientific conference than an industry conference.  And, it's making a lot of use of various social tools: http://www.socialtext.net/supernova/, http://supernova.typepad.com/moblog/.

Decentralized Work (Thomas Malone) sounds good, but I think there are powerful entrenched stakeholders that can work against or reverse this trend (just because it would be good doesn't mean it will happen).  I'm taking a look at The Future of Work right now; one first inchoate thought is how some of the same themes are treated differently in The Innovator's Solution.

The Network is People - a panel with Chrisopher Allen, Esther Dyson, Ray Ozzie, and Mena Trott.  Interesting/new thoughts:

• Chris Allen on spreadsheets:  They are a social tool for convincing people with numbers and scenarios, just like presentation software is for convincing people with words and images.  So if you consider a spreadsheet social software, well, what isn't social software?
• "43% of time is spent on grooming in large monkey troupes."  (But wait, what species of monkeys are we talking about here?  Where are our footnotes?)  So, the implication is that the amount of overhead involved in maintaining true social ties in large groups is probably very high.  Tools that would actually help with this (as opposed to just growing the size of your 'network' to ridiculous proportions) would be a true killer app. 
  
• Size of network is not necessarily a good metric, just one that's easy to measure.  Some people really only want a small group.
  

Syndication Nation - panel with Tim Bray, Paul Boutin, Scott Rosenberg, Kevin Marks, Dave Sifry.  I felt that this panel had a lot of promise but spent a lot of time on background and/or ratholing on imponderables (like business models).  Kevin and Tim tried to open this up a bit to talk about some of the new possibilities that automatic syndication offers.  At the moment, it's mostly about news stories and blogs and cat pictures.  Some interesting/new thoughts:

• Kevin stated that # of subscribers to a given feed follows a power law almost exactly, all the way down to 1.  So even having a handful of readers is an accomplishment.  One might also note that this means the vast majority of subscriptions are in this 'micropublishing' area.
• New syndication possibilities mentioned: Traffic cameras for your favorite/current route. 
  
• The Web is like a vast library; syndicated feeds are about what's happening now (stasis vs. change).  What does this mean?
• The one interesting thing to come out of the how-to-get-paid-for-this discussion: What if you could subscribe to a feed of advertising that you want to see?  How much more would advertisers pay for this?  (Reminds me of a discussion I heard recently about radio stations going back to actually playing more music and less talk/commercials: They actually get paid more per commercial-minute because advertisers realize their ad won't be buried in a sea of crap that nobody is listening to.)

More on some of the other topics later. 

Supernova 2004 midterm update

I'm at the Supernova 2004 conference at the moment.  I'm scribbling notes as I go, and plan to go back and cohere the highlights into a post-conference writeup.  First impressions:  Lots of smart and articulate people here, both on the panels and in the 'audience'.  I wish there were more time for audience participation, though there is plenty of time for informal interactions between and after sessions.  The more panel-like sessions are better than the formal presentations.

The Syndication Nation panel had some good points, but it ratholed a bit on standard issues and would have benefited from a longer term/wider vision.  How to pay for content is important, but it's a well trodden area.  We could just give it a code name, like a chess opening, and save a lot of discussion time...

I am interested in the Autonomic Computing discussion and related topics, if for no other reason than we really need to be able to focus smart people on something other than how to handle and recover from system issues.  It's addressing the technical complexity problem.

Next problem: The legal complexity problem (IP vs. IP: Intellectual Property Meets the Internet Protocol) - I think this problem is far harder because it's political.  There's no good solution in sight for how to deal with the disruptions technology are causing business models and the structure of IP law.

And, on a minor note, I learned the correct pronunciation of Esther Dyson's first name.