May 2010 Archives

Part of my continued series of reflections on the WS-REST workshop.

It may continue to amaze, but the most basic aspects of interoperability continue to have a common problem: the implementation almost always violates the protocol specs that it supports. And that violation often gets encoded into practice, and back into the spec. Sometimes this is for good reasons, sometimes not.

Sam drew from his experience with early SOAP interoperability to illustrate the problem. Apache SOAP, for example, a descendent of IBM SOAP4J, and the predecessor to Apache Axis, had a history of supporting the early SOAP spec, when it was a vendor-led W3C note. It had encoded data with the SOAP encoding section of the SOAP specification, because the XML Schema data type system was not due to be out until mid 2001.

Of course, early specifications often under-specify, and that leads to implementation leaking through. A classic borderline cases is representing "Infinity". XML Schema suggests that infinity should be represented as the characters "INF". Apache SOAP, predating this, used the java.util.Number class, whose toString() method generated "Infinity".

The fix is simple enough, but there were deployed systems out there already. The solution was to continue to produce "Infinity", but to accept both "INF" and "Infinity". Future SOAP stacks, like Axis, which were built around XSD in mind, both produced and consumed INF.

My first reaction to a lot of this was, "conform to the (damn) specification". If it were only so simple. Sometimes the spec is wrong or incomplete. Sometimes a boundary condition is missed by implementers. And organizational inertia makes it hard to upgrade. So, documenting what is implemented becomes the preference.

None of the above has really any difference with REST. One of the takeways: contracts matter.

I know that there's trepidation about the term "contract" among those in the REST community, in that it may envision a rigid interface. I think it's a price worth paying -- you cannot eliminate the notion of "contract" when interoperability is a concern. Contracts are not about how rigid or flexible the interface is. They are about ensuring complete, understood, and compliant expectations. And, frustratingly, they cannot be well-specified if there isn't any implementation experience.

One might argue that the Web has proven very good at composing contracts together -- transfer protocol, media types, identifiers, etc. It didn't eliminate the painstaking need for agreement, it depends on it. Those interested in philosophy might find this an interesting paradox, that the explosion of freedom & diversity of web content and applications requires strong control at the lower layers -- an idea Alexander Galloway explores in his book, Protocol: how control exists after decentralization. Practically speaking, the areas of the Web where there was lax specification and/or complex implementation compromises are the areas where there's not a lot of diversity - such as browsers, which have to deal with HTML, CSS, JS, etc. Servers haven't been immune from this, but they have somewhat more understandable specs to follow, with HTTP, MIME, URI, etc., though ambiguity with HTTP has led to a dearth of complete HTTP 1.1 deployments (most use a subset). This has led to the work on HTTPbis to clean up and clarify the spec based on over a decade of experience.

Considering today's new generation of RESTful APIs on social media & cloud platforms, there are some curious trends. JSON-based RESTful services, for example, do not specify dates, or formal decimals. XML (schemas) do have standardized simple types, but the syntax and infoset is off-putting due to its lack of fidelity with popular programming languages. Which is better for interoperability? Maybe the answer is because JSON is so easy to parse, it's ok that each endpoint needs to adapt to a particular set of quirks, like differing date/time formats. But that sounds like a very developer-centric argument, something that serves the lazy side of the brain. We can do better.

Part of my continuing series of reflections on the REST workshop at WWW 2010...

Sam Ruby's keynote began with a call for technologists to back off the extreme positions often taken in technical arguments (likely because he's knee deep in the middle of the mother of all technical standards debates). Using Mark Pilgrim's parable of the web services spec developer fighting to the death over Unicode Normalization Forms, he illustrates that technical advocacy can lead to knee jerk, us vs. them reactions on very minute details, especially when different mindsets collide and have to work out a path to interoperability.

Relating this to the "REST community" (whatever that means), I think we seem to have a built-in defensive posture, and aren't quite used to being in the driver's seat for innovation. There were so many attempts to "fix" the Web into some other architecture: HTTP-ng tried with a distributed object model, the Web Services crowd tried again. The slow counter-revolution of "using the Web" was largely grown out of Mark Baker's health and savings account. And now, REST is now a technical pop culture buzzword, being perverted in meaning as all buzzwords are prone to. Roy's gotta defend the meaning of the word if it's to mean anything. So, in a way, it's completely understandable why RESTafarians honed a defence mechanism against misleading arguments that they fought ad nauseam.

Unfortunately, today, this defence mechanism seems to have limited free discussion of the limitations of REST, the areas of useful extension, the design of media types, and the ability to relate or objectively discuss extensions to REST that may evaluate the benefits of decades-old ideas like transactions, reliable messaging, or content-level security, without being patronizing ("you don't need transactions", or "reliable messaging isn't really reliable", "use SSL, it's easier", etc.). Researchers and standards working groups try to follow the style, or publish extensions, but anticipate the dreaded Roy bitchslap, perhaps because there seems to be a lack of resources (tools, documentation, frameworks, examples in different use cases) to learn how to apply the style, or perhaps because people are being lazy in applying the style (glomming onto a buzzword). Hard to tell.

In short, this is a hard problem to resolve, as there are good reasons for people's strong opinions. Compromise can only be achieved if there's shared understanding of the facts & shared values. Mostly I see shared values in terms of the properties that people find desirable in a networked system (interoperability, extensibility, visibility, etc.), we still seem to be missing a shared understanding of the facts (such as the basic idea of "emergent properties" in software architecture, and the idea that they can be induced through constraints -- which I think many still reject.)

To be fair, there have been plenty of extensions out there, include the ARRESTED thesis from Rohit Khare, various public RESTful APIs and frameworks out there, Henrik Nielsen's use of RESTful SOAP services in Microsoft Robotics studio, etc.

Look at the reference to the latter, for example. There are plenty of "specific methods" being defined while fitting under the POST semantics. Some would have a problem with that, but I'm not sure what the alternative would be. There are "state replacement" methods under the PUT semantics if you prefer that. But some clients may prefer the more specific, precision state transfer of a POST method. Would you consider this heresy? Or would you stop and realize that Henrik was in the front row of the WS-REST workshop, and co-authored too many W3C notes and IETF RFC's to enumerate here?

I'll speak more about this problem with the "write side of the web" soon, but in short, we don't really have a good way of describing the pre-conditions, post-conditions and side-effects of a POST in a media type, which would be probably the more "RESTful" thing to do. Mark Baker took a stab at this 8 years ago, but there's hasn't been much pickup. In lieu of that, the simplest thing might be to make an engineering decision, slap a method name & documentation on the URI+POST interaction, and move on. You'd be no worse than a WSDL-based Web Service, with the added benefit of GET and URIs. Or, depending on your use case, it may mean crafting a full media type for your system, though I'm not sure that's "simple".

This dilemma is why I tend not to have a problem with description languages like WSDL 2.0 or WADL -- they're not complete, certainly, but they're a start to wrapping our heads around how to describe actions on the Web without breaking uniformity.

Most of those I spoke to at WS-REST felt was that media types, tools, and frameworks for REST have been growing slowly and steadily since 2008, but lately there's been a major uptick in activity: HTML5 is giving browsers and web applications a makeover, OAuth is an RFC, several Atom extensions have grown, Facebook has adopted RDFa, Jersey is looking at Hypermedia support, etc.

But, coming out of the WS-* vs. REST debates, it is interesting in noting what hasn't happened the past two years, since the W3C Workshop on the Web of Services, which was a call to bridge-building. As expected, SOAP web services haven't really grown on the public-facing Internet. And, also as expected, RESTful services have proliferated on the Internet, though with varying degrees of quality. What hasn't happened is any real investment in REST on the part of traditional middleware vendors in their servers and toolsets. Enterprises continue to mostly build and consume WSDL-based services, with REST being the exception. There has been, at best, "checkbox" level features for REST in toolkits ("Oh, ok, we'll enable PUT and DELETE methods"). Most enterprise developers still, wrongly, view REST as "a way of doing CRUD operations over HTTP". The desire to apply REST to traditional integration and enterprise use- cases has remained somewhat elusive and limited in scope.

Why was this?

Some might say that REST is a fraud and can't be applied to enterprise integration use cases. I've seen the counter-evidence, and obviously disagree with that assessment, though I think that REST-in-practice (tools, documentation, frameworks) could be improved on quite a lot for the mindset and skills of enterprise developers & architects. In particular, the data integration side of the Web seems confused, with the common practice being "make it really easy to perform the required neurosurgery" with JSON. I still hold out hope for reducing the total amount of point-to-point data mapping & transformation through SemWeb technologies like OWL or SKOS.

Perhaps it was the bad taste left from SOA, or a consequence of the recession. Independently of the REST vs. WS debate, SOA was oversold and under-performed in the market place. Money was made, certainly, but no where near as much as BEA, IBM, TIBCO, or Oracle wanted, and certainly not enough for most Web Services-era middleware startups to thrive independently. It's kind of hard to spend money chasing a "new technology" after spending so much money on the SOAP stack (and seeing many of the advanced specs remain largely unimplemented and/or unused, similar to CORBA services in the 90's).

Or, maybe it was just the consequence of the REST community being a bunch of disjointed mailing list gadflys, standards committee wonks, and bloggers, who all decided to get back to "building stuff" after the AtomPub release, and haven't really had the time or inclination to build a "stack", as vendors are prone to.

Regardless the reason, the recent uptick in activity hasn't come from the enterprise or the enterprise software vendors offering a new stack. The nascent industries that have invested in REST are social computing, exemplified by Twitter, Facebook, etc. and cloud computing, with vCloud, OCCI and the Amazon S3 or EC2 APIs leading the way.

The result has been a number of uneven media types, use of generic formats (application/xml or application/json), mixed success in actually building hypermedia instead of "JSON CRUD over HTTP", and a proliferation of toolkits in JavaScript, Java, Ruby, Python, etc.

We're going to be living with the results of this first generation of HTTP Data APIs for quite some time. It's time to apply our lessons learned into a new generation of tools and libraries.

WS-REST Workshop Themes

| No Comments | 1 TrackBack

The First Workshop on RESTful Design (WS-REST, hur hur) was held at WWW 2010 in Raleigh last week. I attended and was also on the program committee. It was great to finally meet a number of folks from rest-discuss and the #rest IRC channel, along with seeing again folks working to improve REST in the trenches.

This turned into the mother of all blog posts, so I've chunked it up into a series. The tl;dr version of my workshop takeaways are the following:

- Investment in RESTful technology is growing, particularly with cloud and social computing. Enterprise and vendor investment has been low.

- Technical disagreements and arguments from authority are counter-productive, and seem to be ubiquitous. Now is the time to build stuff, and make stuff interoperate, not split hairs. On the other hand....

- In building interoperable systems, sometimes domain experts are wrong, sometimes they're right. Sometimes it's better to document and standardize what works, sometimes that's not good enough.

In short, interoperability is hard, REST doesn't change that, better get a good helmet if you're in a group specification or standards body these days. The most important thing is if you're going to willfully violate someone else's idea of "the Right Thing", then be explicit and honest about it, so that it can be discussed. (This is, in part, the saga of the HTML5 standards effort.)

- REST has become a buzzword among those that aren't architects and don't want to be architects. That means there needs to be better documentation, tools, and frameworks; most developers can't derive the right design from first principles. Most published RESTful APIs published today are not as interoperable as they could be; there will be a price to pay to maintain these.

- Having said that, for architects, progress has been slow in driving a true discipline for architecture, though there is some great work afoot to make this easier

- Hypermedia tends to be a somewhat jarring user experience to those that still love their desktop applications, especially when we build small protocols like OAuth , but fall back on HTML to do other pieces like authentication and authorization policy selection.

- Developers tend to love dynamic binding nature of REST so long as they're not the ones that have to maintain the client. We need a better way of handling dynamic binding (my suggestion is a work in progress) to hypermedia.

- It's high time to work on the "write side" of the web. We know that HTTP GET works well for interoperability. We have had less success with write actions, and need to think through interoperability there to a much greater degree than we have. Enterprise
integration use cases, for example, could really benefit from describing POST interfaces for domain application protocols. The best we have is WSDL, WADL, and generic forms like XForms, HTML Forms, and generic publishing containers like AtomPub. We can do better.

- You can use REST in the enterprise, for legacy integration, successfully.

- Peer to Peer systems quite achievable in RESTful systems. The nature of hypermedia itself is peer-to-peer. And minor extensions to HTTP can lead to full data propagation style systems.

- Progress is being made to formally describe a RESTful Semantic Web, one that leverages the formal descriptions of tuple spaces with the polyadic pi-calculus and extends them to RESTful architectures.

- While HTML 5 is progressing to be a rich platform for the next generation web, there's experimentation happening for alternatives. Some drive traditional desktop UIs through RESTful resources, others are describing mixed reality worlds.

I have some forthcoming blog posts musing on these themes.

Reblog this post [with Zemanta]

About this Archive

This page is an archive of entries from May 2010 listed from newest to oldest.

March 2010 is the previous archive.

October 2010 is the next archive.

Find recent content on the main index or look in the archives to find all content.

About Me
(C) 2003-2011 Stuart Charlton

Blogroll on Bloglines

Disclaimer: All opinions expressed in this blog are my own, and are not necessarily shared by my employer or any other organization I am affiliated with.