Metaphorical Web

In a recent interview with Rohit Khare, Director of CommerceNet Labs, Jon Udell may have been responsible for introducing a new meme into the noosphere that will be as important in its time as AJAX was in 2004.

by kurt.cagle

I was asked, recently, why I settled on the name Metaporical Web for my site and blog. This is probably as good a statement of purpose as any I can think of, so if you are curious as to what exactly motivated me to name it this, please read on … (warning, this may take a while) …

I have, over the years, made frequent use of the term Metaphorical Web, a term which become more laden with significance and meaning as the web itself has evolved, for me if no one else. I am not a classically trained programmer – I had no courses in computing in college (when I was working on my Bachelors degree, ultimately in Physics), though I have been working with computers and computer programming since I was in my early teens, and have been a professional software developer and writer pretty much continuously for the last thirty years.

Yet this lack of formalism in my early years has had an interesting consequence upon the way that I approach programming. Most programmers today, at least those that have been collegiately trained, have had some exposure to object oriented programming, either in the form of C++ or increasingly in the form of Java over the course of their matriculation, and like many college students, they tend to take these structures for granted – this is the way that you program, because this is the way that you were taught to program. To question the fundamentals of what we mean by objects, by classes, even design patterns and the like is to embrace a certain heresy that generally does not sit well for many developers.

In recent years, a chasm has opened up in the programming community between those that worked with these concepts as given and those that came to programming later, when languages such as Python, Ruby, JavaScript, and the like were becoming more readily adopted, which in turn has also fragmented further given the number of XML practitioners out there. These new programmers recognized that while type, the set of properties and methods that acted on a given object, remained an important aspect of development, type did not necessarily have to be built into these new fangled objects, but could instead be assigned as a some more of a cloak over a bundle of raw data. Of such seemingly obscure things are schisms made.

The XML revolution that has taken place in the last decade has been hailed by many, derided by as many, and its significance even now is debated daily. XML emerged originally as a way of providing a more consistent way of describing the seemingly innocuous HTML format, though its creators (most of whom had come from the SGML side of things) understood one of the first rules of programming; if you have a generic method of describing a given object, then it is worth looking at this language for describing other objects. Given XML’s bias as a declarative language – a language that provided a formal declaration of either a state or a process (what the model is supposed to be) rather than giving an intentional expression of the actions that the language should take (what the model is supposed to do) – it soon became evident that you could use XML as an abstraction language, one that could readily be used to create a metaphorical abstraction of a process that by itself functioned in much the same way that a score of music does.

A score is not a performance. It is a description of the states that a given piece of music should take over time. It does not tell a musician, except in the very broadest sense, how to move a bow across a string while holding that string down at a fret in order to produce a tone on a violin or cello. These are assumed abstractions; the score indicates only what notes or pauses should be played at what point in the score, and it defines this at all points in time. The score is an abstraction of the music being produced, not the music itself.

This declaration, this abstraction, of the musical harmonies (or cacophonies) of the resulting performance is a metaphor for the music, a model of that music. The specific implementations may vary in subtle and profound ways – a high school orchestra’s implementation will likely be rougher than that of professional symphonic players, for instance – but in the main, the music so produced by either group should be roughly recognizable as being the same piece. This is both the strength and weakness of both metaphor and declarative programming.

In the late 1990s, as HTML became more widely accepted, there was a movement afoot to encode two other domains of human endeavor using similar tools. The first, the creation of a standard for encoding mathematics, was proposed both because of its general utility and because mathematical encoding generally assumes clearly defined predicates of symbolic manipulation, and as such, the belief that math could be converted into a new notation. This effort very quickly collided with one of those wonderful hidden complications that often spell the deepening of knowledge – the fact that there were in fact two forms of mathematical coding, the first encoding notation, the second encoding concept, and the two were not necessarily completely in accord. Our assumptions about the metaphors, the models being used, broke down here, and the result of that was a richer understanding that mathematical notation itself was not completely devoid of semantic interconnections. Thus, if you look closely at MathML you realize that you are in fact examining two fairly distinct languages where such a distinction was initially far from obvious.

The second effort, HyTime, was intended as a way of encoding musical scores, which, like mathematics, seemed to have a fairly discretely defined symbolic set and which, consequently, should have been easy to turn into some formal SGML standard. However, just as with mathematics, the realization began to be made by the participants that the formal notation developed over the last several centuries was considerably more sophisticated and subtle than it appeared on the surface. Perhaps it was because HyTime was being articulated by people who dealt with documents on a daily basis but the effort soon began to intersect such boundaries as the articulation of semantics, the importance of transformation mechanisms and the nature of definition.

In other words, in order to develop HyTime, the developers working in this domain had to define models for relationship mapping (which would in time morph into the Resource Description Framework, or RDF), tools for articulating these concepts in a semantically neutral technology (XML) and mechanisms for addressing this information in a set-wise manner (XPath, and the languages that is spawned). Like panning for gold, the working groups had to mix together the mathematics, the music, and the words and then had to shake vigorously in order to cause that which was common to all three to fall out of this matrix.

I think that it is easy to impute both too much significance to XML and too little, sometimes at the same time. As a document encoding format it is barely adequate (which it should, nearly by definition, be) to it’s purpose. It has been used as the basis for building messaging services and object descriptions and even the internal guts of imperative programs, and in some cases (many cases, I would contend) it has generally proven inefficient, expensive to parse and serialize, conceptually difficult to work with, and fragile to the demands of developers. Moreover, it breaks every paradigm that nearly every programmer working more than a few years has learned, inverting the object-oriented model that was so fundamental to Stroustrup, et al., and forcing the redevelopment and deployment of both new tools and new methodologies for handling this information.

It’s my belief that this effort did not occur simply as an empty academic exercise. One of the earmarks of software development in the late 1990s was the fact that the cost of writing software was rising faster than the attendant cost savings from using the software once it was developed. This was not always obvious, of course; in part because there were many case studies where such software was developed in time, under budget, and well suited to task. Yet these case studies often tended to mask the fact that what was being completed were closed system jobs – those software projects that in general controlled all aspects of the environment, that had clearly articulated boundaries and communication protocols, and that usually involved applications that largely ran within the domain of a single processor space (or between two highly coupled processors).

The ones that failed, either by running over budget or by collapsing from over-complexity, were the ones that fell more into being open systems that needed to work in heterogeneous environments, and typically employed highly distributed networks of nodes rather than clearly defined client/server relationship or strongly articulated hierarchies. Now, some software developers (especially those that were clearly seeking to develop “end-to-end” solutions) attempted to make the argument that the best way to not fail in that domain was simply not to go there, to stay within the well defined proprietary worlds established by the tools (to the extent that, for many developers, the outer domain simply ceased to exist).

The problem with this approach, however, is one that is common in teaching, especially at the university level. You can continue to treat the edge cases as being anomalous for quite some time and be able to function quite efficiently, but especially as the domain itself continues to evolve, the anomalies become more and more frequent, and the closed domain software has increasing trouble solving the needs of the customers.

This is what I believe has happened with the Internet, which is the ultimate heterogeneous environment. The original version of the web, that of a connected radio transmitter/receiver model where each node had both capabilities, very quickly collapsed into hub and spoke models in which the preponderance of state remained within the hub, and the “clients” at the other end of the spoke were basically the software equivalent of dumb terminals. However, due to competition largely between Microsoft and Netscape, the clients continued to gain intelligence (and consequently the ability to manage more state) until Netscape collapsed, at which point Microsoft shifted gears towards developing technology that would map more closely to the client-server paradigm which was intrinsic to the low latency, largely encapsulated applications which they had specialized in earlier.

This was I believe the original mandate for SOAP and WSDL, which, because it involved communication going over port 80 (used by HTTP) bypassed many of the headaches that the company had encountered a few years before in attempting to get Distributed COM (DCOM) integrated into the majority of servers that were not running Windows software. However, the model involved was still fundamentally client-server, with the assumption that the server continued to handle the lion’s share of state management.

Yet the metaphor that worked well for low latency networks doesn’t work especially well for high latency ones, which the Internet, even with considerable improvements in performance, still is in comparison to the nanosecond clock cycles of in-processor computing. Moreover, the Internet is still a remarkably heterogeneous environment, and as such the protocols that seem to work best are the ones that are associated with the twin metaphors of publishing and search rather than the closed-system client/server transactions more typical when Windows (and networking) first arose.

In the view of some (even many), this move away from a server-component orientation is a step back, but over the last few years in particular I’ve come to question that wisdom. All metaphors are ultimately leaky – there are places where the analogy between processes breaks down and you have to readily admit that the web is no more a publishing office than it is an ATM. However, I’d contend as well that certain metaphors are coercive on the web; the system was built in such a way that they are more successful in capturing critical aspects of the process than other metaphors.

The web was built around the notion of high latency, low-bandwidth computing within a heterogeneous environment. It is fundamentally unreliable — indeed, this was in fact a deliberate design decision at the TCP/IP level because of the recognition that it was better to build redundancy into the system than to optimize it for high performance.

It is for this reason that RSS feeds have continued to dominate as data transport mechanisms even when SOAP and WSDL were being splashed across the front page of every tech magazine worldwide – syndication is a strong metaphor on the web, one that works consistently with the publishing model embedded in HTTP. It’s highly indicative that nearly eight years after SOAP/WSDL based web services were first introduced, they remain largely viable only within the immediate confines of department-to-department implementations, behind closed firewalls over homogenous system implementations. The metaphor doesn’t scale.

With the introduction of XMLHttpRequest (ironically by Microsoft), the gateway existed for enabling more sophisticated communication between client and server. When Mozilla Firefox implemented in 2003, however, it opened up a breach in the dam of browser dominance (and either a deliberate or an ill-informed decision to let Internet Explorer wither on the vine), and other companies (most notably Google) came pouring in.

Beyond the immediate effects of cementing the role of DOM-based processing on the client (making that client considerably more malleable than most traditional desktop applications), it also served to push JavaScript to the forefront as theclient language, in turn finally providing in user-controlled software what had only been available to the browser developers initially – a way to turn a client application into a full-featured communication node on the Internet. That piece in turn has revolutionized the web by making it possible for the client to fully manage its own relevant state – indeed, with the advent of built-in databases and database access APIs (such as Google Gears), the next logical move in this process is happening, in which the clients are able to maintain full autonomy independent of a remote server, including the ability to complete and validate data to a certain degree before transmitting it outside of the bounds of a single session.

This brings me to the Metaphorical Web. Tim Berners-Lee first introduced the notion of the Semantic Web in the mid-1990s, spawned ironically by the experience with HyTime and MathML. While there are several concepts that lurk within the umbra of semantics, in general, the role of semantics is to provide mechanisms for the classification of content. Classification is critical in this respect, because until system nodes can effectively (and efficiently) classify content transmitted to them on their own, they must rely upon human classifiers to do the job for them. Object-oriented programming was built around the need to provide a classification scheme upon data-blocks to provide them with a certain inherent intelligence about their own environment, but such intelligence is a proxy – some human programmer somewhere had to write that classification mechanism into the DNA of the object itself.

Metaphors are the precursor to Semantics. To determine what an object is, you have to have a map – a standard of comparison – in order to determine what the object is not. REST based publishing systems and Open Search mechanisms are important not because they simplify the interfaces that humans use but because they provide a semantically neutral mechanism for interactions between nodes in the network. Whether I am publishing a blog, a stock bid, a weather report, or a block of music, the REST model treats all of these things in the same way. Because each of those objects is different, however, what lies behind that REST model must be able to determine both the semantics of the transmitted model and the intent of the transmission (though this latter can usually be determined by the destination of the data being sent).

This abstraction of transmission goes hand-in-hand with the abstraction of data access; provide a data abstraction layer, such as XQuery, between your semantically neutral inbound requests and your data store, and you have a system that is basically transparent – the hardware becomes commoditized. Build a semantically neutral front-end (such as XForms) within your client, and you furthermore have a similar abstraction between your inbound data and your presentation layer. This is one of the reasons that I manage both XForms.org and will soon manage XQuery.org as part of the Metaphorical Web network – I believe that both technologies will become far more important in the next few years as more people understand the benefits inherent in such technologies.

A couple final notes here – First, I believe ultimately that both JSON and E4X will have a significant long term role to play in this as well; JSON solves one of the more immediate problems that DOM-centric XML has – it is considerably lighter-weight as a messaging protocol, and can hold other content (including XML). Similarly, ECMAScript for XML (E4X) provides a way to do many key XML-oriented operations using a lighterweight structure, and it can help buttress the limitations that JSON has (most notably as a document carrier). These will be covered fairly extensively within the mandate of the Metaphorical Web.

The second note – classification, taxonomy, and the abstraction of intentional programming are all key aspects of the Semantic Web. My personal feeling about the Semantic Web is that it is a necessary, perhaps vital technology, that makes it possible to build web systems that are able to go from semantically neutral transmission mechanisms to the realization of intent and classification. However, for this to happen, the Semantic Web is going to need to start communicating more effectively with the rest of the web. I suspect that technologies such as RDFa will play a huge part in that, as will the incorporation of pipelined systems (such as the W3C’s XProc) that will make it possible to orchestrate filters for layered annotation (or decomposition) of metadata from contextual searches.

This particular vision of the web may not please all players. While it will undoubtedly spawn its own markets, the RESTful web won’t sell any more server boxes, operating systems, or “end-to-end” turnkey solutions, nor will it employ armies of contract temp programmers on massive projects that go nowhere. Much of this vision is coming from the open source and open standards communities that recognize that the boutique era of software has been ending for some time as we’re literally drowning in reasonably good software that’s no longer economically worth it to hawk as products. It’s SOA taken out of the box and stripped of its fancy wrappings, and it is ultimately only following the natural grain of the web itself. That seems as good a reason to want to promote it as any.

Efforts have been underway recently to develop a schema language for JSON, analogous to the XML Schema Definition Language (XSD) or RelaxNG languages in the XML arena. Similarly, a JSON transformation language is being proposed and bandied about in various AJAX circles as web2 developers attempt to take the best of what XML has to offer and recast it from the angle-bracket modality to the braced modality.

Quick! Do you use the Compound Document Format?! You, know, CDF … surely you use CDF, right?

Chances are pretty good that you have no idea about what I’m talking about. Everyone knows Microsoft’s word document format and Adobe’s PDF, chances are pretty good that if you’re reading this on XML.com you’ve heard of ODF and OOXML, especially after the fairly rancorous discussions about ISO status for these two formats. Yet CDF, hmmmm … that’s a rough one. Didn’t it belong to Corel, once upon a time?