The Delphi Age

The Blogosphere is proud these days, in the wake of their collective defrocking of a news media institution. As well they should be. The Killian affair reveals traditional news media as tottering toward irrelevance. There are parallels between the Blogosphere’s decentralized participatory structure and an ongoing software development trend started by Linus Torvalds, known as Open Source.

A 1997 essay by Eric Steven Raymond identified the strengths of developing complex software in an open, collaborative environment---The Cathedral and the Bazaar:

Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time.

Linus Torvalds's style of development---release early and often, delegate everything you can, be open to the point of promiscuity---came as a surprise. No quiet, reverent cathedral-building here---rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches out of which a coherent and stable system could seemingly emerge only by a succession of miracles...The Linux world not only didn't fly apart in confusion, but seemed to go from strength to strength at a speed barely imaginable to cathedral-builders.

Mr Raymond identifies an efficacious method of managing code design, debugging and implementation through a committed open developer community. The Cathedral and Bazaar model is applicable not only to software design, but to the analyzation and dissemination of all information, including news. By Mr. Raymond’s analogy, the ‘cathedral’ in the Killian case is CBS News---established, esoteric, top-down, institutional and stodgy---extremely protective of its sources and fact-gathering processes. The ‘bazaar’ is the Blogosphere. News blogs are self-correcting systems composed of broad bases of readers who also contribute knowledge to issues. The bloggers themselves provide focus as guides and inspiration, much like Linus Torvalds provided leadership to the legions of volunteer coders building Linux.

Mr Raymond contrasts the different approaches of 'cathedral' and 'bazaar' knowledge building. Though he is talking about software development, he might just as easily be talking about news gathering and analysis in the Blogosphere:

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, "Given enough eyeballs, all bugs are shallow.'' I dub this: "Linus's Law''.

In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out [leading to] long release intervals.

In the bazaar view, you assume that bugs are generally shallow phenomena---or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.

Sociologists years ago discovered that the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly-chosen one of the observers. They called this the Delphi Effect.

Contributors for any given [Linux] project are self-selected. Contributions are received not from a random sample, but from people who are interested enough to use the software, learn about how it works, attempt to find solutions to problems they encounter, and actually produce an apparently reasonable fix. Anyone who passes all these filters is highly likely to have something useful to contribute.

The Delphi Effect appears to be affecting news gathering and analysis, and the promulgation of knowledge in general. There are armies of volunteers brought together on blogs whose averaged opinions and knowledge create a formidable challenge to traditional cathedral-style news organizations. The trend extends into other areas of knowledge, such as the Wikipedia---a collaboratively developed free encyclopedia that is created and updated by its users. No article is finished in the Wikipedia. It has a self-healing quality that gradually extracts false data. The Delphi Effect keeps the Wikipedia current, accurate and dynamic. Wikipedia’s competitors are centuries-old cathedral-style knowledge bureaucracies like Encyclopedia Brittanica. They spend millions maintaining their knowledge base, releasing it in large, expensive sets once a year. Wikipedia costs little to maintain, is far more dynamic, current, and perhaps covers a broader knowledge gamut.

We are seeing the Delphi Effect route around faulty news evidence from CBS just as it does buggy code, rendered anachronistic. Competing against CBS’s ‘cathedral’ style of news gathering and reporting is a vibrant, stealthy and reliable watchdog: blogs. Where CBS stonewalls over time, blogs self-correct, nearly instantaneously.

Mr Raymond distills the basic laws behind the Open Source movement:

• Release early and often;


• Grow a beta list by adding to it everyone who contacts you;


• Send chatty announcements to the beta list whenever releasing, encouraging people to participate;


• Listen to beta-testers, polling them about design decisions and stroking them whenever they sent in patches and feedback;


• If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource.

We should expect to see the Delphi Effect continue to challenge traditional strongholds of knowledge. We should also be aware of potential pitfalls with Blogospheric news. There are questions that should be considered as we move into the Delphi Age:

• Is Al Qaeda an example of the Delphi Effect applied to extremists? Does this account for much of their power to challenge sovereign nations?


• If so, is a sovereign country a 'cathedral' to the terrorist's 'bazaar'? In other words, can Delphi-style terrorists be defeated by traditional top-down applications of power?


• Dan Rather's 'cathedral' career is on the line; is there equivalent accountability in the Blogosphere, where most users are anonymous?


• Is the war on terror a war against asymmetrical opposition? If so, how can we embrace asymmetry in technological and social development while we fight it's darkest sociological side-effects?


• Does the power of evaluation created by blogs always serve the cause of truth? What about Al Quaeda’s blogs, or ones in the Arab world? No doubt that in those blogospheres Jews are pigs and the Protocols of the Elders of Zion is the de facto truth. Can the Delphi Effect work against itself in a bazaar composed of closed minds laboring under a consensus of delusion?


• Can the Blogosphere become the ultimate medium for a new kind of demagoguery? With over 100,000 readers a day, could someone like Charles Johnson of Little Green Footballs organize the ultimate flash mobs, if so inclined? That's real power in this era. What are the limits of power available to bloggers?

We live in momentous times.