Listen Print Discuss

LazyWeb and RSS: Given Enough Eyeballs, Are Features Shallow Too?

by Clay Shirky
01/07/2003

A persistent criticism of open source software is that it is more about copying existing features than creating new ones. While this criticism is overblown, the literature of open source is clearer on debugging than on design. This note concerns an attempt to apply debugging techniques to feature requests and concludes by describing Ben Hammersley's attempt to create such a system, implemented as an RSS feed.

A key observation in Eric Raymond's The Cathedral and the Bazaar is: "Given enough eyeballs, all bugs are shallow." Raymond suggests that Brook's Law--"Adding more programmers to a late software project makes it later"--doesn't apply here, because debugging is less collaborative than most other software development. He quotes Linus on the nature of debugging: "Somebody finds the problem, and somebody else understands it. And I'll go on record as saying that finding it is the bigger challenge."

Finding a bug doesn't mean simply pointing out broken behavior, though. Finding a bug means both locating it and describing it clearly. The difference between "It doesn't work" and "Whenever I resize the login window, the Submit button disappears" is the difference between useless mail and useful feedback. Both the description and the fix are vital, and the description precedes the fix.

Enter the LazyWeb

There is evidence that this two-step process applies to features as well, in a pattern Matt Jones has dubbed the LazyWeb. The original formulation was "If you wait long enough, someone will write/build/design what you were thinking about." But it is coming to mean "I describe a feature I think should exist in hopes that someone else will code it." Like debugging, the success of the LazyWeb is related at least in part to the quality of the descriptions. A feature, schema, or application described in enough detail can give the right developer (usually someone thinking about the same problem) a clear idea of how to code it quickly.

Examples of the LazyWeb in action are Stephen Johnson's URL catcher as built by Andre Torrez, and Ben Trott's "More Like This From Others" feature after Ben Hammersley's initial characterization.

LazyWeb seems to work for at least three reasons:

  1. Developers have blind spots

    Because a developer knows a piece of software in a far more detailed way than their users, they can have blind spots. A good LazyWeb description provides a developer with a alternate perspective on the problem. And sometimes the triviality of the coding involved keeps developers from understanding how valuable a feature would be to its users, as with Yoz Grahame's "get, search and replace, display" script for revising the text of web pages. Sometimes four lines of code can make all the difference.

  2. Developers have social itches

    The canonical motivation for open source developers is that they want to "scratch an itch." In this view, most open source software is written with the developer as the primary user, with any additional use seen as a valuable but secondary side-effect.

    Sometimes, though, the itch a developer has is social: they want to write software other people will adopt. In this case, the advantage of the LazyWeb is not just that a new application or feature is described clearly, but that it is guaranteed to have at least one grateful user. Furthermore, LazyWeb etiquette involves publicizing any solution that does arise, meaning that the developer gets free public attention, even if only to a select group. If writing software that gets used in the wild is a motivation, acting on a LazyWeb description is in many ways a karmically optimal move.

  3. Many eyes make features shallow

    This is really a meta-advantage. The above advantages would apply even to a conversation between a single describer and a single developer, if they were the right people. Expanding the conversation to include more describers and developers increases the possibility that at least one such pairing will occur.

Transaction Costs and Coordination Costs

The transaction costs of the LazyWeb are extremely low. Someone describes; someone else codes. The describer can write sketchily or in great detail. No developers are required to read the description; and those who do read it can ignore or modify the proposed design. The interface between the parties is lightweight, one-way, and optional.

However, the coordination costs of the LazyWeb as a whole are very high, and they will grow as more people try it. More people can describe features than write software, just as more people can characterize bugs than fix them. Unlike debugging, however, a LazyWeb description does not necessarily have a target application or a target group of developers. This creates significant interface problems, since maximal LazyWeb awareness would have every developer reading every description, an obvious impossibility. (Shades of Brook's Law.)

This would be true even if the LazyWeb were confined to skilled programmers. The ability of system architects, say, to describe new visual layout tools, or graphics programmers to characterize their filesharing needs, ensures that there will always be more capable describers than suitable developers.

Thus the LazyWeb is currently limited to those environments that maximize the likelihood that a developer with a social itch and a good grasp of the problem space will happen to read a particular LazyWeb description. In practice, this means that successful LazyWeb requests work best when posted on a few blogs read by many developers. Far from being a "More describers than developers" scenario, in other words, the current LazyWeb has many fewer describers than developers, with the developers fragmented across several sites.

Sounds Like a Job for RSS

Related Reading

Content Syndication with RSS
By Ben Hammersley

One common answer to this kind of problem is to launch a portal for all LazyWeb requests. (There have been earlier experiments in this domain, like http://www.halfbakery.com, http://www.shouldexist.org, and http://www.creativitypool.com, and Magnetbox has launched a Lazyweb-specific site.) These sites are meant to be brokers between describers and developers.

However, nearly a decade of experimentation with single-purpose portals shows that most of them fail. As an alternative to making a LazyWeb portal, creating an RSS feed of LazyWeb descriptions has several potential advantages, including letting anyone anywhere add to the feed, letting sites that serve developers present the feed in an existing context, and letting the developers themselves fold, spindle, and mutilate the feed in any way they choose.

Ben Hammersley has designed a version of a LazyWeb feed. It has three moving parts.

The first part is the collection of descriptions. Hammersley assumes that a growing number of people will be writing LazyWeb descriptions, and that most of these descriptions will be posted to blogs.

The second part is aggregation. He has created a trackback address, http://blog.mediacooperative.com/mt-tb.cgi/1080, for LazyWeb posts. Blog posts that point to this address are aggregated and presented at http://www.benhammersley.com/lazyweb/.

The third part is the RSS feed itself, at http://www.benhammersley.com/lazyweb/index.rdf, which is simply the XML version of http://www.benhammersley.com/lazyweb/. However, because it is a feed, third parties can subscribe to it, filter it, present it as a sidebar on their own sites, and so on.

It's easy to see new features that could be added to this system. A LazyWeb item in RDF has only four elements, set by the Trackback spec -- title, link, description, and date. Thus almost all the onus on filtering the feed is on the subscriber, not the producer. An RDF format with optional but recommended tags (type: feature, schema, application, etc; domain: chat, blog, email, etc.) might allow for higher-quality syndication, but would be hard with the current version of Trackback. Alternatively, community consensus about how to use title tags to characterize feature requests could help.

And not everyone with a LazyWeb idea runs a Trackback-enabled weblog, so having some way for those people to register their ideas could be useful. Hooks for automated translation could make the feed more useful to developers working in languages other than English, and so on.

But for all the possible new features, this is a good start, having achieved a kind of bootstrap phase analogous to compiler development. The original work came out of a LazyWeb characterization made during a larger conversation Jones, Hammersley, and I have been having about social software, and some of the early LazyWeb requests are themselves feature descriptions for the system.

Will It Work?

Will it work? Who knows. Like any experiment, it could die from inactivity. It could also be swamped by a flood of low-quality submissions. It may be that the membrane that a weblog forms around its readers is better for matching describers and developers than an open feed would be. And Paul Hammond has suggested that "Any attempt to invoke the LazyWeb directly will cause the whole thing to stop working."

It's worth trying, though, because the potential win is so large. If the benefits open source development offers for fixing bugs can be applied to creating features as well, it could confer a huge advantage on the development of Mob Software.

Stefano Mazzocchi of the Cocoon project has said "Anyway, it's a design pattern: 'good ideas and bad code build communities, the other three combinations do not." This is extremely hard to understand, it's probably the most counter-intuitive thing about open source dynamics." If Mazzocchi is right, then a high-quality stream of feature requests could be a powerful tool for building communities of developers and users, as well as providing a significant advantage to open over closed source development.

The closed source shops could subscribe to such a feed as well, of course, but their advantage on the feature front isn't speed, it's secrecy. If a small group of closed source developers is working on a feature list that only they know, they will often ship it first. But for good ideas in the public domain, the open and closed development teams will have the same starting gun. And releasing early and often is where open development has always excelled.

It's too early to tell if LazyWeb is just the flavor of the month or points to something profound about the way ideas can spread. And it's much too early to know if an RSS feed is the right way to spread LazyWeb ideas to the developers best able to take advantage of them. But it's not too early to know that it's worth trying.

Clay Shirky writes about the Internet and teaches at NYU's Interactive Telecommunications Program. He publishes a mailing list on Networks, Economics, and Culture at shirky.com/nec.html.

Return to the OpenP2P.com.