The Delphi Age

The Blogosphere is proud these days, in the wake of their collective defrocking of a news media institution. As well they should be. The Killian affair reveals traditional news media as tottering toward irrelevance. There are parallels between the Blogosphere’s decentralized participatory structure and an ongoing software development trend started by Linus Torvalds, known as Open Source.

A 1997 essay by Eric Steven Raymond identified the strengths of developing complex software in an open, collaborative environment---The Cathedral and the Bazaar:

Linux overturned much of what I thought I knew. I had been preaching the Unix gospel of small tools, rapid prototyping and evolutionary programming for years. But I also believed there was a certain critical complexity above which a more centralized, a priori approach was required. I believed that the most important software needed to be built like cathedrals, carefully crafted by individual wizards or small bands of mages working in splendid isolation, with no beta to be released before its time.

Linus Torvalds's style of development---release early and often, delegate everything you can, be open to the point of promiscuity---came as a surprise. No quiet, reverent cathedral-building here---rather, the Linux community seemed to resemble a great babbling bazaar of differing agendas and approaches out of which a coherent and stable system could seemingly emerge only by a succession of miracles...The Linux world not only didn't fly apart in confusion, but seemed to go from strength to strength at a speed barely imaginable to cathedral-builders.

Mr Raymond identifies an efficacious method of managing code design, debugging and implementation through a committed open developer community. The Cathedral and Bazaar model is applicable not only to software design, but to the analyzation and dissemination of all information, including news. By Mr. Raymond’s analogy, the ‘cathedral’ in the Killian case is CBS News---established, esoteric, top-down, institutional and stodgy---extremely protective of its sources and fact-gathering processes. The ‘bazaar’ is the Blogosphere. News blogs are self-correcting systems composed of broad bases of readers who also contribute knowledge to issues. The bloggers themselves provide focus as guides and inspiration, much like Linus Torvalds provided leadership to the legions of volunteer coders building Linux.

Mr Raymond contrasts the different approaches of 'cathedral' and 'bazaar' knowledge building. Though he is talking about software development, he might just as easily be talking about news gathering and analysis in the Blogosphere:

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone. Or, less formally, "Given enough eyeballs, all bugs are shallow.'' I dub this: "Linus's Law''.

In the cathedral-builder view of programming, bugs and development problems are tricky, insidious, deep phenomena. It takes months of scrutiny by a dedicated few to develop confidence that you've winkled them all out [leading to] long release intervals.

In the bazaar view, you assume that bugs are generally shallow phenomena---or, at least, that they turn shallow pretty quickly when exposed to a thousand eager co-developers pounding on every single new release. Accordingly you release often in order to get more corrections, and as a beneficial side effect you have less to lose if an occasional botch gets out the door.

Sociologists years ago discovered that the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly-chosen one of the observers. They called this the Delphi Effect.

Contributors for any given [Linux] project are self-selected. Contributions are received not from a random sample, but from people who are interested enough to use the software, learn about how it works, attempt to find solutions to problems they encounter, and actually produce an apparently reasonable fix. Anyone who passes all these filters is highly likely to have something useful to contribute.

The Delphi Effect appears to be affecting news gathering and analysis, and the promulgation of knowledge in general. There are armies of volunteers brought together on blogs whose averaged opinions and knowledge create a formidable challenge to traditional cathedral-style news organizations. The trend extends into other areas of knowledge, such as the Wikipedia---a collaboratively developed free encyclopedia that is created and updated by its users. No article is finished in the Wikipedia. It has a self-healing quality that gradually extracts false data. The Delphi Effect keeps the Wikipedia current, accurate and dynamic. Wikipedia’s competitors are centuries-old cathedral-style knowledge bureaucracies like Encyclopedia Brittanica. They spend millions maintaining their knowledge base, releasing it in large, expensive sets once a year. Wikipedia costs little to maintain, is far more dynamic, current, and perhaps covers a broader knowledge gamut.

We are seeing the Delphi Effect route around faulty news evidence from CBS just as it does buggy code, rendered anachronistic. Competing against CBS’s ‘cathedral’ style of news gathering and reporting is a vibrant, stealthy and reliable watchdog: blogs. Where CBS stonewalls over time, blogs self-correct, nearly instantaneously.

Mr Raymond distills the basic laws behind the Open Source movement:

• Release early and often;


• Grow a beta list by adding to it everyone who contacts you;


• Send chatty announcements to the beta list whenever releasing, encouraging people to participate;


• Listen to beta-testers, polling them about design decisions and stroking them whenever they sent in patches and feedback;


• If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource.

We should expect to see the Delphi Effect continue to challenge traditional strongholds of knowledge. We should also be aware of potential pitfalls with Blogospheric news. There are questions that should be considered as we move into the Delphi Age:

• Is Al Qaeda an example of the Delphi Effect applied to extremists? Does this account for much of their power to challenge sovereign nations?


• If so, is a sovereign country a 'cathedral' to the terrorist's 'bazaar'? In other words, can Delphi-style terrorists be defeated by traditional top-down applications of power?


• Dan Rather's 'cathedral' career is on the line; is there equivalent accountability in the Blogosphere, where most users are anonymous?


• Is the war on terror a war against asymmetrical opposition? If so, how can we embrace asymmetry in technological and social development while we fight it's darkest sociological side-effects?


• Does the power of evaluation created by blogs always serve the cause of truth? What about Al Quaeda’s blogs, or ones in the Arab world? No doubt that in those blogospheres Jews are pigs and the Protocols of the Elders of Zion is the de facto truth. Can the Delphi Effect work against itself in a bazaar composed of closed minds laboring under a consensus of delusion?


• Can the Blogosphere become the ultimate medium for a new kind of demagoguery? With over 100,000 readers a day, could someone like Charles Johnson of Little Green Footballs organize the ultimate flash mobs, if so inclined? That's real power in this era. What are the limits of power available to bloggers?

We live in momentous times.

Faust's Soul

Mayor Richard M. Daley unveiled plans for a surveillance network for the city of Chicago---Mayor Outlines Elaborate Camera Network for City:

From a hi-tech command center, the City of Chicago plans to monitor a vast security network. Thousands of surveillance cameras will be linked -- and authorities will be alerted to crimes and terrorist acts. The mayor unveiled the plans for this new security network at a news conference this morning.

Some people are concerned about "Big Brother" invading their privacy but Mayor Daley says the cameras will be located in public areas.

The technology that is now so much a part of crime-fighting and anti-terrorism has gone -- as one police spokesman says -- from Stone Age to Star Wars in less than a decade. This step in the evolution will link more than 2,000 public surveillance cameras in Chicago into a unified system.

There are well over 2,000 cameras that the city and its sister agencies -- like the school system -- monitor everyday. The city is adding another 250 cameras to potential high risk areas, most of them downtown. For instance, if there is a crime on a CTA platform-- most of which are or will be equipped with surveillance cameras, a call to 911 will activate a video link-up.

"When the system determines there's a camera in the vicinity of the 911 call, it will automatically beam back an image to the call-taker of the origin of where it occurred," said Ron Huberman, Emergency Mgt. and Com. Dir.

The 911 dispatcher will have -- in many cases -- the ability to remotely control cameras at the scene of a crime miles away. The system is also equipped with software that can alert the 911 Center to changes in traffic flow, or the presence of people where they're not supposed to be.

In reading about vast, robust and expensive surveillance networks like Chicago’s, its worth pondering how winning the war against terrorism will threaten the freedoms we so cherish. ‘Winning’ seems to mean reclaiming parity with historical forces by creating a system---the Anti Fog---which acts as a sieve to ferret out potential disruption. The Anti Fog will be a synthesis of networks composed of people and surveillance systems that report to authorities for safeguarding the status quo. That’s a verbose way of saying ‘spy network,’ in the employ of the powers-that-be---corporate, government, American, Chinese, etc.

‘Status quo’ is defined as the state in which anything is already; to preserve extant property and systems. Terrorists threaten many things. At their most basic level, they threaten what already exists. Anti-terror is therefore a force to preserve the status quo. Status quo preservation risks hegemonic results, however. The powers-that-be are on the verge of acquiring the most pervasive anti-disruption system in history. It might serve to keep the peace; it can also keep power in the hands of institutions that would otherwise evolve from the pressure of change---governments and corporations alike.

The economy’s next boom will focus technological genius towards the task of protecting property, people and perhaps even ideas from disruption. In the networking sector, this has long since begun. Go to any MIS department of a sizeable organization and what will be found is a veritable war room, replete with daily battles and skirmishes, long term strategies, and force allocations on a scaling budget. The onslaught of spam, viruses and cyber attacks is the best window to the future that can be found today. Internet attackers are unidentified, numerous, endlessly replicate, and constantly metamorphize. The goal of the attackers is to simply bring down the system, and disrupt its status quo. They are generally unaligned and have individual causes. Organizations spend billions of dollars combating mosquitoes that never seem to end; some carry malaria, many just act as sand in the gears. At the end of the day, the soldiers of MIS ‘win’ if the status quo of their employer’s infrastructure is intact and running. The daily battles are kept as transparent as possible to reduce disruption.

Disturbingly, if an MIS director of a major American institution were to compare security strategies with a government official in Beijing tracking Chinese dissent on the Internet, they would have much practical knowledge in common. What is worrying is that Anti Fog isn’t just an American endeavor---its job number one for any institution, government or private, on Earth. All institutions are fighting the same war against disruption, and are developing the tools necessary to preserve status quo. We live in an era where those tools will be incredibly potent and effective.

The war on terror operates on the same rules as the MIS war. To fight terror, one must adopt the goals of the MIS warriors. It is nearly inevitable that the most innovative technologies and resources of all institutions---ranging from governments that are democracies to totalitarian to corporations---will be put towards building Anti Fog. A robust Anti Fog system will not only curtail some of our civil liberties, but a redefine them. What we could lose in the bargain is ingenuity, change, progress and legitimate dissent. The status quo naturally resists evolutionary pressures that effect change, and in turn growth.

We should ask ourselves these questions with respect to winning this great war:

Will the fight for freedom devolve into a fight for safety? Does the ordered world have only a choice between destruction or a Faustian bargain to preserve what we are at the expense of what we could be?