Theory P or Theory D?

Which theory fits the evidence (Raganwald):

Theory P adherents believe that there are lies, damned lies, and software development estimates. ... Theory P adherents believe that the most important element of successful software development is learning.

Maybe I'm an extreme P adherent; I say that learning is everything in software development.  The results of this learning are captured in code where possible, human minds where not.  Absolutely everything else associated with software development can and will be automated away.

Finally:

To date, Theory P is the clear winner on the evidence, and it’s not even close. Like any reasonable theory, it explains what we have observed to date and makes predictions that are tested empirically every day.

Theory D, on the other hand, is the overwhelming winner in the marketplace, and again it’s not even close. The vast majority of software development projects are managed according to Theory D, with large, heavyweight investments in design and planning in advance, very little tolerance for deviation from the plan, and a belief that good planning can make up for poor execution by contributors.

Does Theory D reflect reality? From the perspective of effective software development, I do not believe so. However, from the perspective of organizational culture, theory D is reality, and you ignore it at your peril.

So this is a clear contradiction.  Why is it that theory D is so successful (at replicating itself if nothing else) while theory P languishes (at replicating)?  Perhaps D offers clear benefits to its adherents within large organizations -- status, power, large reporting trees...  and thus P can't gain a foothold despite offering clear organization-level benefits. 

But I suspect that it's simpler than that; I think that people simply don't really evaluate history or data objectively.  Also, it may be difficult for people without the technical background to really how difficult some problems are; past a certain level of functionality, it's all equally magic.  The size of the team that accomplished a task then becomes a proxy for its level of difficulty, in the way that high prices become a proxy for the quality of a product in the marketplace for the majority of consumers.  So small teams, by this measure, must not be accomplishing much, and if they do, it's a fluke that can be explained away in hindsight with a bit of work.

Somebody should do a dissertation on this...

Tags: software development, theory p, theory d, statistics, evidence, process