Results tagged “rest”

Part of my continued series of reflections on the WS-REST workshop.

It may continue to amaze, but the most basic aspects of interoperability continue to have a common problem: the implementation almost always violates the protocol specs that it supports. And that violation often gets encoded into practice, and back into the spec. Sometimes this is for good reasons, sometimes not.

Sam drew from his experience with early SOAP interoperability to illustrate the problem. Apache SOAP, for example, a descendent of IBM SOAP4J, and the predecessor to Apache Axis, had a history of supporting the early SOAP spec, when it was a vendor-led W3C note. It had encoded data with the SOAP encoding section of the SOAP specification, because the XML Schema data type system was not due to be out until mid 2001.

Of course, early specifications often under-specify, and that leads to implementation leaking through. A classic borderline cases is representing "Infinity". XML Schema suggests that infinity should be represented as the characters "INF". Apache SOAP, predating this, used the java.util.Number class, whose toString() method generated "Infinity".

The fix is simple enough, but there were deployed systems out there already. The solution was to continue to produce "Infinity", but to accept both "INF" and "Infinity". Future SOAP stacks, like Axis, which were built around XSD in mind, both produced and consumed INF.

My first reaction to a lot of this was, "conform to the (damn) specification". If it were only so simple. Sometimes the spec is wrong or incomplete. Sometimes a boundary condition is missed by implementers. And organizational inertia makes it hard to upgrade. So, documenting what is implemented becomes the preference.

None of the above has really any difference with REST. One of the takeways: contracts matter.

I know that there's trepidation about the term "contract" among those in the REST community, in that it may envision a rigid interface. I think it's a price worth paying -- you cannot eliminate the notion of "contract" when interoperability is a concern. Contracts are not about how rigid or flexible the interface is. They are about ensuring complete, understood, and compliant expectations. And, frustratingly, they cannot be well-specified if there isn't any implementation experience.

One might argue that the Web has proven very good at composing contracts together -- transfer protocol, media types, identifiers, etc. It didn't eliminate the painstaking need for agreement, it depends on it. Those interested in philosophy might find this an interesting paradox, that the explosion of freedom & diversity of web content and applications requires strong control at the lower layers -- an idea Alexander Galloway explores in his book, Protocol: how control exists after decentralization. Practically speaking, the areas of the Web where there was lax specification and/or complex implementation compromises are the areas where there's not a lot of diversity - such as browsers, which have to deal with HTML, CSS, JS, etc. Servers haven't been immune from this, but they have somewhat more understandable specs to follow, with HTTP, MIME, URI, etc., though ambiguity with HTTP has led to a dearth of complete HTTP 1.1 deployments (most use a subset). This has led to the work on HTTPbis to clean up and clarify the spec based on over a decade of experience.

Considering today's new generation of RESTful APIs on social media & cloud platforms, there are some curious trends. JSON-based RESTful services, for example, do not specify dates, or formal decimals. XML (schemas) do have standardized simple types, but the syntax and infoset is off-putting due to its lack of fidelity with popular programming languages. Which is better for interoperability? Maybe the answer is because JSON is so easy to parse, it's ok that each endpoint needs to adapt to a particular set of quirks, like differing date/time formats. But that sounds like a very developer-centric argument, something that serves the lazy side of the brain. We can do better.

Part of my continuing series of reflections on the REST workshop at WWW 2010...

Sam Ruby's keynote began with a call for technologists to back off the extreme positions often taken in technical arguments (likely because he's knee deep in the middle of the mother of all technical standards debates). Using Mark Pilgrim's parable of the web services spec developer fighting to the death over Unicode Normalization Forms, he illustrates that technical advocacy can lead to knee jerk, us vs. them reactions on very minute details, especially when different mindsets collide and have to work out a path to interoperability.

Relating this to the "REST community" (whatever that means), I think we seem to have a built-in defensive posture, and aren't quite used to being in the driver's seat for innovation. There were so many attempts to "fix" the Web into some other architecture: HTTP-ng tried with a distributed object model, the Web Services crowd tried again. The slow counter-revolution of "using the Web" was largely grown out of Mark Baker's health and savings account. And now, REST is now a technical pop culture buzzword, being perverted in meaning as all buzzwords are prone to. Roy's gotta defend the meaning of the word if it's to mean anything. So, in a way, it's completely understandable why RESTafarians honed a defence mechanism against misleading arguments that they fought ad nauseam.

Unfortunately, today, this defence mechanism seems to have limited free discussion of the limitations of REST, the areas of useful extension, the design of media types, and the ability to relate or objectively discuss extensions to REST that may evaluate the benefits of decades-old ideas like transactions, reliable messaging, or content-level security, without being patronizing ("you don't need transactions", or "reliable messaging isn't really reliable", "use SSL, it's easier", etc.). Researchers and standards working groups try to follow the style, or publish extensions, but anticipate the dreaded Roy bitchslap, perhaps because there seems to be a lack of resources (tools, documentation, frameworks, examples in different use cases) to learn how to apply the style, or perhaps because people are being lazy in applying the style (glomming onto a buzzword). Hard to tell.

In short, this is a hard problem to resolve, as there are good reasons for people's strong opinions. Compromise can only be achieved if there's shared understanding of the facts & shared values. Mostly I see shared values in terms of the properties that people find desirable in a networked system (interoperability, extensibility, visibility, etc.), we still seem to be missing a shared understanding of the facts (such as the basic idea of "emergent properties" in software architecture, and the idea that they can be induced through constraints -- which I think many still reject.)

To be fair, there have been plenty of extensions out there, include the ARRESTED thesis from Rohit Khare, various public RESTful APIs and frameworks out there, Henrik Nielsen's use of RESTful SOAP services in Microsoft Robotics studio, etc.

Look at the reference to the latter, for example. There are plenty of "specific methods" being defined while fitting under the POST semantics. Some would have a problem with that, but I'm not sure what the alternative would be. There are "state replacement" methods under the PUT semantics if you prefer that. But some clients may prefer the more specific, precision state transfer of a POST method. Would you consider this heresy? Or would you stop and realize that Henrik was in the front row of the WS-REST workshop, and co-authored too many W3C notes and IETF RFC's to enumerate here?

I'll speak more about this problem with the "write side of the web" soon, but in short, we don't really have a good way of describing the pre-conditions, post-conditions and side-effects of a POST in a media type, which would be probably the more "RESTful" thing to do. Mark Baker took a stab at this 8 years ago, but there's hasn't been much pickup. In lieu of that, the simplest thing might be to make an engineering decision, slap a method name & documentation on the URI+POST interaction, and move on. You'd be no worse than a WSDL-based Web Service, with the added benefit of GET and URIs. Or, depending on your use case, it may mean crafting a full media type for your system, though I'm not sure that's "simple".

This dilemma is why I tend not to have a problem with description languages like WSDL 2.0 or WADL -- they're not complete, certainly, but they're a start to wrapping our heads around how to describe actions on the Web without breaking uniformity.

Most of those I spoke to at WS-REST felt was that media types, tools, and frameworks for REST have been growing slowly and steadily since 2008, but lately there's been a major uptick in activity: HTML5 is giving browsers and web applications a makeover, OAuth is an RFC, several Atom extensions have grown, Facebook has adopted RDFa, Jersey is looking at Hypermedia support, etc.

But, coming out of the WS-* vs. REST debates, it is interesting in noting what hasn't happened the past two years, since the W3C Workshop on the Web of Services, which was a call to bridge-building. As expected, SOAP web services haven't really grown on the public-facing Internet. And, also as expected, RESTful services have proliferated on the Internet, though with varying degrees of quality. What hasn't happened is any real investment in REST on the part of traditional middleware vendors in their servers and toolsets. Enterprises continue to mostly build and consume WSDL-based services, with REST being the exception. There has been, at best, "checkbox" level features for REST in toolkits ("Oh, ok, we'll enable PUT and DELETE methods"). Most enterprise developers still, wrongly, view REST as "a way of doing CRUD operations over HTTP". The desire to apply REST to traditional integration and enterprise use- cases has remained somewhat elusive and limited in scope.

Why was this?

Some might say that REST is a fraud and can't be applied to enterprise integration use cases. I've seen the counter-evidence, and obviously disagree with that assessment, though I think that REST-in-practice (tools, documentation, frameworks) could be improved on quite a lot for the mindset and skills of enterprise developers & architects. In particular, the data integration side of the Web seems confused, with the common practice being "make it really easy to perform the required neurosurgery" with JSON. I still hold out hope for reducing the total amount of point-to-point data mapping & transformation through SemWeb technologies like OWL or SKOS.

Perhaps it was the bad taste left from SOA, or a consequence of the recession. Independently of the REST vs. WS debate, SOA was oversold and under-performed in the market place. Money was made, certainly, but no where near as much as BEA, IBM, TIBCO, or Oracle wanted, and certainly not enough for most Web Services-era middleware startups to thrive independently. It's kind of hard to spend money chasing a "new technology" after spending so much money on the SOAP stack (and seeing many of the advanced specs remain largely unimplemented and/or unused, similar to CORBA services in the 90's).

Or, maybe it was just the consequence of the REST community being a bunch of disjointed mailing list gadflys, standards committee wonks, and bloggers, who all decided to get back to "building stuff" after the AtomPub release, and haven't really had the time or inclination to build a "stack", as vendors are prone to.

Regardless the reason, the recent uptick in activity hasn't come from the enterprise or the enterprise software vendors offering a new stack. The nascent industries that have invested in REST are social computing, exemplified by Twitter, Facebook, etc. and cloud computing, with vCloud, OCCI and the Amazon S3 or EC2 APIs leading the way.

The result has been a number of uneven media types, use of generic formats (application/xml or application/json), mixed success in actually building hypermedia instead of "JSON CRUD over HTTP", and a proliferation of toolkits in JavaScript, Java, Ruby, Python, etc.

We're going to be living with the results of this first generation of HTTP Data APIs for quite some time. It's time to apply our lessons learned into a new generation of tools and libraries.

The First Workshop on RESTful Design (WS-REST, hur hur) was held at WWW 2010 in Raleigh last week. I attended and was also on the program committee. It was great to finally meet a number of folks from rest-discuss and the #rest IRC channel, along with seeing again folks working to improve REST in the trenches.

This turned into the mother of all blog posts, so I've chunked it up into a series. The tl;dr version of my workshop takeaways are the following:

- Investment in RESTful technology is growing, particularly with cloud and social computing. Enterprise and vendor investment has been low.

- Technical disagreements and arguments from authority are counter-productive, and seem to be ubiquitous. Now is the time to build stuff, and make stuff interoperate, not split hairs. On the other hand....

- In building interoperable systems, sometimes domain experts are wrong, sometimes they're right. Sometimes it's better to document and standardize what works, sometimes that's not good enough.

In short, interoperability is hard, REST doesn't change that, better get a good helmet if you're in a group specification or standards body these days. The most important thing is if you're going to willfully violate someone else's idea of "the Right Thing", then be explicit and honest about it, so that it can be discussed. (This is, in part, the saga of the HTML5 standards effort.)

- REST has become a buzzword among those that aren't architects and don't want to be architects. That means there needs to be better documentation, tools, and frameworks; most developers can't derive the right design from first principles. Most published RESTful APIs published today are not as interoperable as they could be; there will be a price to pay to maintain these.

- Having said that, for architects, progress has been slow in driving a true discipline for architecture, though there is some great work afoot to make this easier

- Hypermedia tends to be a somewhat jarring user experience to those that still love their desktop applications, especially when we build small protocols like OAuth , but fall back on HTML to do other pieces like authentication and authorization policy selection.

- Developers tend to love dynamic binding nature of REST so long as they're not the ones that have to maintain the client. We need a better way of handling dynamic binding (my suggestion is a work in progress) to hypermedia.

- It's high time to work on the "write side" of the web. We know that HTTP GET works well for interoperability. We have had less success with write actions, and need to think through interoperability there to a much greater degree than we have. Enterprise
integration use cases, for example, could really benefit from describing POST interfaces for domain application protocols. The best we have is WSDL, WADL, and generic forms like XForms, HTML Forms, and generic publishing containers like AtomPub. We can do better.

- You can use REST in the enterprise, for legacy integration, successfully.

- Peer to Peer systems quite achievable in RESTful systems. The nature of hypermedia itself is peer-to-peer. And minor extensions to HTTP can lead to full data propagation style systems.

- Progress is being made to formally describe a RESTful Semantic Web, one that leverages the formal descriptions of tuple spaces with the polyadic pi-calculus and extends them to RESTful architectures.

- While HTML 5 is progressing to be a rich platform for the next generation web, there's experimentation happening for alternatives. Some drive traditional desktop UIs through RESTful resources, others are describing mixed reality worlds.

I have some forthcoming blog posts musing on these themes.

Building a hypermedia-aware client is rather different from building a typical client in a client/server system. It may not be immediately intuitive. But, I believe the notions are rooted in (quite literally) decades of experience in other computing domains that are agent-oriented. Game behaviour engines, control systems, reactive or event-driven systems all have been developed with this programming approach in mind.

The normal way we build clients, in a client/server architecture, looks something like this:

The logic of the application - its objectives and how it wants to achieve them through one or more remove services, is often procedural. A rich OO domain model is sometimes preferred to procedural logic, but this isn't usually used in conjunction with remote services because of the latency involved; a service facade coalesces communication into coarse grained interactions.

This idea of a service facade culminates in SOA, where interfaces, along with all their possible message exchange patterns, are registered for others to lookup:

A agent-oriented client, on the other hand, looks something this (which I've adapted from Russell & Norvig's diagram):

The application agent has several pools of pre-defined logic:

a) Application Logic: some logic for the application itself (e.g. the basic states of a hypermedia application, the goals of the application if it has any. A browser has no goals other than rendering; whereas a product ordering & payment agent would have the goal of completing e-commerce transactions on behalf of a user)

b) Action Logic: some logic for the implications of actions (e.g. how does a payment & product ordering agent know, interpret, or infer that PUT/POST/DELETEing to a particular sequence of URIs will result in a paid product order)?

c) Protocol Logic: some built-in logic for handling protocols & media types (e.g. URI, MIME, HTTP, and maybe some mix of HTML, Atom, Atompub, etc.).

The problem of bridging together application and action logic together is known as Action Selection. Action selection doesn't require fancy algorithms. Its study has often dealt with complex subject matter, which has often lead to complex solutions. But in most agents, the bread and butter for action selection is simple: the Finite State Machine (FSM). An agent responds to changes in the environment based on its current state and a set of known transitions. There are other approaches to agent programming that are growing in popularity, like planners, but let's start with FSMs.

Firstly, an agent's application logic requires a state machine to describe the relationship between sensing ("safe") actions and changing ("unsafe") actions. In a hypermedia application it looks something like this:

This basic hypermedia application state machine is sandwiched hierarchically between the super-state machine for the application's goals and the sub-state machines for the protocols:

The trick with building a RESTful agent based on FSMs would be to figure out a way such that

a) The application's goals can be expressed in terms of hypermedia agent logic (e.g. sensing & effecting)

b) The hypermedia types and link relations themselves contains enough interpretable action logic that can be mapped to the application's domain

c) The action and protocol state machines are modular. RESTful applications tend to have standardized and relatively small number of generic protocols, so they need to be repurposed for different applications and/or contexts.

Two ways of accomplishing this include hierarchical FSMs and behaviour trees.

Hierarchical FSMs are popular in control systems and game engines. They are great for reactive systems, where the correct interpretation & response to input and events is the intent of the application. Managing call control, or a climate control system are examples. There are powerful generic Hierarchical FSM standards out there like SCXML that provide a code-on-demand approach to interpreting and managing states across a set of resources (though it probably could use some RESTful-friendly polish).

Behaviour trees have the same power as hierarchical FSMs, but tend to be more oriented towards goal-based applications, where the purpose of the application is to transition a bunch of resource state to some new state. For example, a calendar scheduling agent, or a payment & ordering agent, are examples of goal-oriented agents.

In future, I'm going to explore how to build a behaviour tree-based agent; probably for the Restbucks domain that Jim Webber, Savas Parastatidis, and Ian Robinson have been using for the past year or so and including in their "REST in Practice" book.

Related articles by Zemanta

• REST in Practice: Rough Cuts Version (oreilly.com)

Update: Comments should be working now.

This is my attempt to summarize an overview of my thinking on RESTful versioning. It's a follow up to Square Peg, REST hole. These concepts can be tricky concepts to describe, and I don't really want to write a small book on this topic, so I may get some of this wrong. Thus, expect updates to this entry to improve it in the future.

Data Versioning vs. Language Versioning

Extensibility and versioning in RESTful services can be viewed in terms of two domains of agreement. The two domains are: resource and representation, which could also be thought of as the "data" vs. "language" domains.

First, let's recall what a resource is: a time varying membership function, where the members are instances of a representation at various points in time. The resource can return different values at different times. BUT resources can be narrowed down into very specific semantics, if resource owner wishes. A resource might be "the most recent version" of a record, whose state might change often, or it might be a "specific version" of a record, and thus unchanging in state. These are two different resources, even though they may have the same representation for a period of time. A resource may even contain format metadata and constrain the language emitted, though content negotiation may be preferred.

Regardless of how often the values change, the semantics of the resource should not change. "Revision 3 of purchase order 123" should retain that meaning. If they do change the meaning, it hurts consumers that relied on the old meaning.

When we think of URI versioning, this is a design choice when resources are immutable across time and we create new resources for state changes (similar to how we manage time-series data in a database).

With language extension or versioning, on the other hand, the state is unchanged, but the way that data is represented has changed.

On Language Versioning

Rule #1: Prefer to extend a language in a forwards and/or backwards compatibile manner. Version indicators are a last resort, to denote incompatible changes.

Extension, of course, requires thought. It implies well-specified interpretation policies for language consumers, and in the case of a machine-readable schema, well-specified extension points. But the range of choices aren't too hard to understand.

This table summarizes the current techniques in practice for extensible or versioned languages, using the terminology from the W3C TAG's draft versioning compatibility strategies document, by David Orchard, which I'm going to butcher through my own brief summaries.

	Consumer	Producer
Backwards-Compatible	Lookup version notifications	Replacement or Side-by-side Version notification via out-of-band channel or links
Forwards-Compatible	Must accept unknowns Must preserve unknowns if persisting state Version identifier substitution model	Media type specification clearly defines consumer forward compatibility expectations (and/or uses a machine-readable schema to denote forward-compatibility extension areas)
Incompatible	Check for version identifier	Side-by-side or Breaking Replacement

Some explanations...

Version Notifications

Agents should be notified of new versions. This can be done out-of-band (email, physical letter, carrier pigeon), but it helps to complement this with links. These links could be an extended, and agreed to link relation, and/or as part of the media type specification. The links may point to a description of the version change, or, in the case of a Side-by-Side, the URI that emits the resource in the new language version.

Replacement

This implies that origin server is replaced by a new backwards-compatible version that is able to accept both old versions and new versions of representations sent by a client (usually via a POST link). This is useful in combination with a forward compatible change -- none of the links need to change.

Side by Side

This implies that the origin server provides a new MIME type or URI-space for resources using the new language, along side old resources using the old language. In either case, you are impyling "this language changes everything". In the case of changing URIs to reflect the new language verison, in effect, you're using "resource versioning", something usually relegated to storing time series data , as a means to work around your language compatibility problems.

To make this RESTful, your media type must include a link from the old resources to their new version, along with metadata indicating the version of the language used at the URI, possibly including a link to a machine-readable schema of the new version (if your media type has such a thing, like XML with Relax NG or XSD). In the case of a new MIME type, you would want a link relation that notes an alternate format is available.

Let me underscore this: You cannot expect clients to understand your URI format and swap out all occurrences of "v1" with "v2"., if you do, you're placing a heavy burden of coupling on your client, that YOUR SERVER is so special, that they need to understand YOUR URI format. This is completely antithetical to why we would want to use REST in the first place, unless you're really just tunnelling XML over HTTP for the heck of it. I note that many "REST APIs" out there actually are built this way, which means they're just as point-to-point coupled as other interface styles.

Must Accept Unknowns

If the consumer sees elements in the data it doesn't recognize, it still accepts the representation. Generally, it ignores these elements for processing.

Must preserve unknowns if persisting state

This is an optional follow-on from "Must accept unknowns", and is often forgotten. If representation state is being persisted (i.e. cached) in the consumer's domain for later use, the unrecognized elements should be preserved, and not stripped. This could greatly assist forward compatibility when the client is upgraded to handle the previously unrecognized elements.

Version identifier substitution model

I defer to Section 5.3 of the compatibility strategies document.

Where do you place the version identifier?

In order of preference:

1. In the media type content


2. In the MIME type itself, or as a MIME type parameters


3. In the URI

Version identifier inside the media type content

This has many examples in the wild, such as HTML DOCTYPE, some uses of XMLNS, a version identifier inside your PDF document.

This requires the replacement model for backwards-compatibility, and encourages the greater use of forwards compatibility. It's the way that most web media types have long worked, with varying degrees of success, but note that those formats were long designed with forward compatibility in mind.

It's still possible to combine this approach into side-by-side versioning if need be, especially if you are changing the semantics of your resources.

Version identifier in the MIME type

e.g. application/vnd.mytype;version=2

This is currently a non-standard and debatable technique. The benefit here is that this enables side-by-side versioning without impacting the URI-space. On the other hand, this reeks of avoiding hypermedia and trying to push things to the other layers of the Web Architecture (HTTP and/or URIs). But in many cases this is preferable to a new URI space.

Version identifier in the URI

e.g. http://example.com/data/v1/po/123

I described the primary problem here earlier: you can't assume you are a special snowflake and the client will know that 'v1' is your magic crystal. You must provide a link or a URI template in the media itself (and/or in a service resource) to denote new versions.

The secondary problem is bookmarks, or inbound hyperlinks. In a database system these are known as "foreign keys". Anyone with a relational data background knows that their primary keys really shouldn't change, as it's expensive to propagate that change to foreign keys.

There is, however, one case, where this approach is preferred over the others. This ties back to the beginning of this entry, when I discussed "Resource Versioning". It's clear we mint URIs when the semantics of the resource itself changes. So, if they change with the language, then mint new URIs -- using hypermedia, if possible, to link old concepts to new ones, as this requires a side-by-side compatibility approach.

For example, if we have an Account resource, and a new version of our resources and language we are deprecating the notion of account, and adding two new resources, "Customer" and "Agreement". It makes no sense to preserve the Account URIs for new Customer resources in this case, as the changed meaning would be confusing to clients expecting an Account.

Some Q&A

Aren't bookmarks the problem? Wouldn't life be better if we rejected bookmarked URIs?

Well, yes, they're a problem, but no, life would suck if we rejected bookmarks, because there's no different between a hyperlink and a bookmark. It would be like saying "no one can hyperlink to me", which is absurd.

Wouldn't versioning be simpler if we separated access from identification, like with WSDL services?

If my data identifiers become opaque primary keys like 123 instead of http://example.org/po/123, then they're tightly coupled to the service that produced the document, as it would be the only context in which I could resolve details for that identifier. Now clearly one benefit is, if I create a new incompatible side-by-side service version, technically (assuming I don't need to re-key my database), the stored foreign keys don't change.

In a RESTful approach, URIs are your "foreign keys", and if you embed a version identifier in them, they need to change when you upgrade to the next version if you embed those versions in the URI. Assuming you can't convince your resource owners to use languages with version identifiers as a MIME parameter or inside the language itself, how is that done?

In a word, lazily.

As I've discussed above, your media type should have an extensibility section or link relation(s) that points to the new version. And upon retiring a language at a particular URI, you would use a permanent redirect (301) to tell all consumers to update their bookmarks / foreign keys. In either case, the agent would have the ability to update their persistent reference.

Again, this is a special case -- there really shouldn't be that many incompatible versions, they should be forward-compatible changes that dont' require new URIs unless you're completely mucking with the resource semantics.

In Summary

1. Prefer extensible, forwards & backwards compatible languages and the replacement approach to compatibility. Note the W3C TAG's position on version identifiers
2. Be judicious when you use version identifiers in URIs, as cool URIs don't change
3. For side-by-side deployments, always include a section in your media, or link relation(s), to point to new/old versions, and update references lazily as the consumer refreshes its cached value. Use permanent redirects to retire URIs bound to old language versions.
4. Version URIs if the semantics of the resource changed, but be courteous to consumers by ensuring links are available to denote the old vs. new alternates
5. Chapter 13 of Subbu's wonderful new book RESTful Web Services Cookbook provides more detailed illustrations of several versioning techniques.

Time to dust off my microphone and bring up a couple of topics on REST and the Web Architecture
- Versioning, or "Cool URIs don't change -- but my data format does!"
- Why the Web Architecture could use a Programming Model for the Enterprise

Which I'll try to get to this weekend.

For now, I leave you with two things:

First, reflecting on Wiliams' recent Square Peg, REST Hole, I draw from the archives, What are the benefits of WS-* or proprietary services?

Let's keep our eye on the prize. REST is a style aimed at extensible, low entry-barrier, multi-organizational, confederate information sharing and communication. I note that most IT organizations are confederacies, adopting a federal or feudal governance model.

The Web Architecture itself (MIME types, HTTP, URIs) provides a much-needed stable intermediate form for interoperability among many different systems and applications -- something that an general-purpose orientation, like SOA, doesn't really provide. Or, fitting for RESTafarians, it is a shared hallucination ;-)

Not every system, or layer of an enterprise's architecture, has the same requirements for scalability or interoperability. The post from 2007 highlights such examples.

Secondly, the song, which ruled my college years in Canada....

Related articles by Zemanta

• Why REST security doesn't exist (computerworld.com)
• Where REST may not always be a good fit (blogs.zdnet.com)

Previously on the hit game show, "What's a Cloud?"

• John Willis: Is Everyone an aaS?
  
• Greg Pfister: Is Cloud Computing Just Everything aaS?
  
• (my interpretation of) IBM's definition: a style of IT delivery combining virtualization + soa + service management, for apps, data, and IT resources
  

Over on RedMonk, I heard a very intriguing quote from James Governor that was buried in video:

"If you think of the post-SOA term, from Nick Gall... Web Oriented Architecture, clearly this is somewhat different from SOA, although there are some patterns common to both of them..... Is the cloud Web-Oriented Operations, or WOO? (We have WOA and WOO)... and what IBM is saying is definitely not WOO, it's business as usual, it's just about flexible delivery of application -- all the stuff that is goodness, all the stuff that Tivoli has been talking about since 1995. That stuff all has value, but it's not Cloud. Cloud involves difference. Business as usual, that's just provisioning service, and automation and virtualization, which is all good, but... if I hear a another person tell me that CLOUD = SOA + VIRTUALIZATION + AUTOMATION, I'm going to ignore them and rubbish the idea as much as I can."

Preaching to the choir here; I left BEA almost a year ago to build out a WOA platform for clouds.

But I'm curious -- as far as I can tell, most clouds really ARE some combination of:

1. Service Interface (e.g. Amazon Web Services aren't really WOA up close)
   
2. Provisioning and Automation of some sort (e.g. images, web applications, multi-tier designs)
   
3. Virtualization is admittedly optional, though increasingly common
   

So, if cloud is different, is the difference really a trend towards WOA, or is this really going to happen? I see two patterns:

One pattern is emerging from the IBM and HPs of the world that have collected a number of shiny baubles in their ERP4IT stacks (Tivoli and OpenView) and invested heavily in SOA and WS-* to painstakingly integrate them (and the pile of IT that has been built on this technology over the past 5+ years)> This pattern indicates that the IT world is cleaving in two, with web architecture on one side, to build the new class of end-user services, and boring old SOA+VIRTUALIZATION+PROVISIONING for the back end.

The other pattern is that the cloud is about Web Architecture end-to-end, using WOA to enable linked data and mashups for the development lifecycle, architecture & operations lifecycle , and end-user-services.

I wouldn't bet on the latter being a fait accompli, as most haven't wrapped their heads around how to make this work. And of course, there's a lot of inertia. There are bright spots: notice one of those links comes from IBM Rational's Jazz / OSLC initiative - they seem to "get the Web" for enabling interoperable software delivery lifecycle tools. But the problem is end-to-end. At some point the industry has to recognize that IT is becoming complicated enough that planning for product-line-style reuse is of isolated value, and designing for serendipity and applying knowledge representation principles at global scale are legitimate ways out of this mess.

A lot will depend how this changes the cost and user experience of the ball-of-IT-mud (and whether that can be effectively communicated to those who don't follow the latest architecture acronyms).

My final presentation for the week is up on SlideShare.

At first I was worried this might be a bit too "out there" , but feedback has indicated the presentation was well received and provocative in the right ways, particularly in getting people working on REST to talk more about areas that have been usually punted, particularly the design of "interaction-oriented" interfaces in hypermedia, using POST.

Great seeing Jim, Steve again, and meeting Leonard and Ian for the first time. Though they left the conference before my session, I also briefly got to meet Tim, and Mark Nottingham, two leaders in this space whom I've long respected.

Coming up for air, briefly. I have a busy month, but will be soon giving a preview of what I've been up to:

Today I'm recording a podcast with John Willis, master of all things ESM and Cloud. Should be up in the next few days.

I'm presenting the Cloud Computing tutorial at QCon San Francisco on Monday, November 17th. Also at QCon, I'll be speaking Thursday afternoon at the REST track on Designing Enterprise IT Systems with REST, wherein I try to explain in my own words what it means for a REST API to be hypertext-driven. As a case study, I'll give a sneak peek of some of the design decisions we've made with Elastra's ECML and EDML, such as our approach to linked data in some parts of the architecture, SPARQL in other parts, and the balancing act of modeling & managing data (aka "Using the Semantic Web like Tabasco Sauce ... a little goes a long way").

That same week, Wednesday November, 19th, I will be at the San Jose Fairmont speaking at the Cloud Computing Expo, on Cloud Computing and the Next Generation of Enterprise Architecture. This discusses Elastra's novel approach to architecture-aware, decentralized cloud computing & data centre automation. We enable composable systems management designs that can listen to & provision changes throughout a system's lifecycle.

Updated: Instead of reading my specific nits here, you can see a brief summary of my suggested guideline to designing a cloud interface.

Ian Foster, Savas Parastatidis, Paul Watson, and Mark Mckeown have an article in the latest Communications of the ACM comparing WS-ResourceFramework (WS-RF), WS-Transfer (W-T), RESTful HTTP, and "no convention" as approaches to modeling state.

Look Inside >> September 2008

This article was particularly timely, as it is very similar in approach to my "Managing Data in an SOA" talk at Jazoon 2008. (A better formatted PDF copy of that presentation is available here.) Jazoon is primarily a Java conference, so I got the impression that about less than half of the room had even heard of the Web Services specifications I was talking about, let alone the tension among them, due to how much they overlap. Roy was in the crowd, offering the one audience comment at the end: "Given all of this, which approach would you recommend, or are you using, for your own work?". Much of Elastra's work in managing and provisioning clouds is based around a hypermedia architecture, with RESTful HTTP being our chosen framework. I'll be talking more about this at QCon San Francisco in November.

With regards to this article, I first want to commend the authors at trying (noticeably) hard to keep the discussion technical and respectful, and I hope my comments here will be also taken in that vein, even if I'm being negative. I've long respected their various blogs . The article is certainly a step up from entries such as this one, on "Web Fundamentalism".

My first comment is that the article might be misnamed. In a paper on "modeling state", I didn't see the word "transition" mentioned once. This seems to be a major limitation of this discussion. The authors clarify:

First, a few observations about what we mean by modeling state. The systems with which we want to interact may have simple or complex internal state. Various aspects of this state may be exposed so that external clients can engage in "management" operations....We are not suggesting these mechanisms provide direct access to the underlying state in its entirety. Rather, we are assuming the principles of encapsulation and data integrity/ownership are maintained.... It is unwieldy to keep talking about "modeling a projection of underlying system state," so we use the short- hand "modeling state." It is important to bear in mind, however, the reality of what could be going on behind the boundaries of a system with which an interaction takes place.

That helps to some degree, but it still seems to miss the major point of why it's useful to project state independently from the underlying system: to loosen the coupling of interactions in a distributed system. Interactions lead to state transitions. Besides the state data itself, the available state transitions also can be projected. REST's approach to this is "hypermedia". Object-oriented designs use the state pattern to project such interactions without requiring client coupling. As far as I can tell, the other approaches don't have a clear answer to this. The "No Convention" approach is the worst culprit: potential transitions are tightly intertwined in whatever domain-specific data identifiers, operations or model is being accessed. On the other hand, both WS-RF and WS-Transfer could enable a form of hypermedia with embedded WS-Addressing Endpoint References (EPRs), but this has a variety of problems that are discussed below (some have called this approach "REST on Crack").

I note that if all you want (or all you see) when contrasting these approaches are a system with consistent CRUD operations with an independent, interoperable data format, a lot of the debate between these approaches seems rather obtuse. But complex systems are better modeled as "state machines" than as a state space that can be "managed" with CRUD. This seems to be the major missing piece of the debate - a misunderstanding, underappreciation, or disinterest on modeling state transitions or lifecycles when accessing or manipulating state. Ian Foster's most recent blog entry claims that (in the context of grid vs. cloud):

No evidence is provided for this assertion that complex interfaces are the reason for the difficulties people have with grids. I argue that the issues are more complex. "We can argue whether we prefer REST or Web Services, or say Nimbus (a grid virtualization interface) or EC2 (a cloud virtualization interface), but the differences among these alternatives are not great."

This blog entry seems to demonstrate Mr. Foster really doesn't understand the differences between these interface types, and seems to be referencing this CACM article as a way of brushing this observation under the rug.

Speaking as someone who's designing and building cloud servers, I've made a snide comment or two on this topic this before (e.g. in one of Greg Pfister's presentations, see slide 2). I think there's a lot of thought value in Globus Grid specs, but the totality has a feeling of scientific computing that most IT people generally can't wrap their heads around. The WS-RF interfaces encourage early-adoption developers to give up because they can't easily call the APIs. Amazon EC2's success is largely due to its "retail experience", but also due to the ecosystem of tools and providers that has developed around it. This ecosystem would not have happened had its interfaces not been very, very simple. I'll admit that EC2's REST API doesn't quite use hypermedia the way that I am discussing here -- a future post (around the time of QConSF) will explain a more hypermedia-oriented approach to representing cloud resources.

The rest of this post gets into the gritty details of my issues with the article.

A couple of areas of agreement that I have with the article:

1. The "no conventions" approach to state management has major drawbacks.
   

   
   While the article doesn't say this outright, it certainly hints at this position. In my opinion, if you MUST use WS-*, and you need a consistent CRUD model, then you probably want to look at WS-Management.

2. The value of lifetime features, such as WS-ResourceLifetime, or subscription features.
   

   
   While I don't think lifetime management necessarily warrants new methods, I do think there's value in having a consistent way of specifying a resource's lifetime. This can be mapped on existing methods & response codes, if a media type were created for it. An expired HTTP resource could return 410 Gone, for example. HTTP-based subscriptions happen all the time, though I admit a more precise hypermedia model would be useful.
   

Unfortunately, there are many points of confusion or contention with the article:

1. Misunderstanding what a uniform interface means.

   "According to REST, a small set of verbs/ operations with uniform semantics should be used to build hypermedia applications, with the Web being an example of such an application."

   "REST = verbs" is a red herring. We cleared this up recently in another episode of minor blog drama. Uniformity has to do with URIs, Resources vs. Representations, self-description, the general applicability of all methods to potentially all resources, and the big one, hypermedia. Or, in implementation terms, one has to look at more than HTTP to analyze REST -- there's URI, MIME, and media types like HTML and Atom.

2. Conflating POST with "Create".

   POST means whatever the hypermedia specification that calls for its use says it should mean. In AtomPub, it means "create". In HTML, it means "process this form". It might be emulating GET. It might be appending something. You don't know unless you know the hypermedia context it was linked in.

   HTTP is not CRUD. REST is not just file storage.

3. The utility of accessing and manipulating partial resource state.

   This is less of a point of confusion, and more of an additional critique against something like WS-ResourceProperties.

   Accessing partial state, like one can do with WS-ResourceProperties or (while it's not mentioned in the article) WS-ResourceTransfer with XPath fragments, makes the data model a lot less uniform, and a lot harder to work with. William explains this quite well.

   As an aside, I note that HTTP has the Range header for accessing partial state of a large representation; this wasn't discussed in the article.

4. Suggesting that RESTful HTTP requires "convention over specification". (Aka. Ignoring the hypermedia constraint)

   "Note that whereas HTTP defines all the verbs used, the structure of the URIs and the format and semantics of the documents exchanged in order to implement the job service's operations are application specific. Thus, while the URIs appear to convey some semantic information based on their structure (for example, a /status at the end of a particular job URI may be interpreted by a human as the identifier of the status resource), this is an application-specific convention. "

   And again in the summary:

   Thus, when defining state management operations, the WS-RF and WS- Transfer approaches both use EPRs to refer to state components and to adopt conventions defined in the WS- RF and related specifications and in the WS-Transfer and related specifications, respectively. In contrast, the no-conventions and REST approaches adopt domain-specific encodings of operations, on top of SOAP and HTTP, respectively.

   This is a bad practice, and not indicative of robust REST approaches. Normallly one would use specifications like:
   

   
   
   • HTML form submission
     
   • XForms submission options
     
   • Atompub service documents
     
   • URI templates
     
   
   to describe how a URI (or, very commonly, an entire representation!) is constructed. Arguably none of these are "application specific", or even "domain specific". They're "binding environment specific" (e.g. they each have a different underlying object model), because RESTful doesn't assume an XML Infoset binding model the way WS-RF or WS-RT do.

   One should think of form hypermedia as "just in time interface description". Instead of coding against an operation like in RPC, you're coding against a message destination (the POST target), whose intent is hopefully described by the surrounding metadata. In practice, this is normally no different than most reflection-assisted RPC (e.g. CORBA DII , Java RMI with reflection, dynamically invoked WSDL, etc.), though eventually if RDF takes off one might see even richer forms of reflection than just "symbol lookup".

5. Confusing URIs with EPRs (and Skipping over WS-Management, which offers a resolution)

   One of the co-authors of this CACM article, Mark McKeown, describes in a 2007 blog entry the problem with EPRs:

   ...given an EPR that has ReferenceParameters you should NEVER share it with anyone else. You cannot know what those ReferenceParameters are for. They could be there for some identification purpose, in which case it would be OK to share them, but you cannot know that for sure. They could actually be for identifying a particular session, or client. Sharing EPRs with ReferenceParameters would be like sharing your HTTP cookies; you simply wouldn't do it. Now, imagine a Web were you were not able to share URIs.

   I wasn't involved with the standards work for WS-Addressing, WS-Resource, or WS-Management. But, as a potential consumer of these specfiications, here's what I understand of the mental journey:

   
   
   1. URIs are identifiers.
      
   2. EPRs are not identifiers. Quoth section 2.6 of the spec:
      

      
      The Architecture of the World Wide Web, Volume One [AoWWW] recommends [AoWWW, Section 2] the use of URIs to identify resources. Using abstract properties of an EPR other than [destination] to identify resources is contrary to this recommendation. In certain circumstances, such a use of additional properties may be convenient or beneficial; however, when building systems, the benefits or convenience of identifying a resource using reference parameters should be carefully weighed against the benefits of identifying a resource solely by URI as explained in [AoWWW, Section 2.1] of the Web Architecture.
      

      
      
   3. WS-Resource claims EPRs are identifiers, ignoring the advice of the WS-Addressing specification,
      
   4. WS-Management, the major spec that adopts WS-Transfer, introduced the wsman:ResourceURI reference parameter to remain consistent with the WS-Addressing recommendation, at the expense of admitting that they now have TWO URI's in the header, one for the "endpoint", and one for the "resource".
      

   The moral of the story: Most technical decisions are a reflection of the economic and political context they were made in.

6. Confusing the relative sizes of deployment base

   Proponents of the HTTP/REST approach empha- size that it provides for more concise requests and permits the use of sim- pler client tooling than approaches based on Web services. Critics point out that the use of HTTP/REST means that users cannot leverage the signifi- cant investment in Web services tech- nologies and platforms for message- based interactions.

   While I agree there has been significant investment in Web services technologies,
   

   
   
   1. the vast majority of it does not use WS-Addressing,
      
   2. an even smaller fraction uses WS-RF or WS-Man (which leverages WS-Transfer)
      
   3. even without this, it is probably several orders of magnitude less than the amount of deployed HTTP-based hypermedia technologies.
      
   
   Now, granted, there is not much in the way of HTTP/REST tooling that's specifically targeted at advanced enterprise use of that technology (such as interaction or business process management), but given the context of this article is on "state management", this counter-argument doesn't really apply.

   WS-RF is a world onto itself, with no bridges to the World Wide Web. WS-Management at least tries to remain consistent with the web architecture by exposing a visible EPR Reference Parameter with wsman:ResourceURI that could be dereferenced via HTTP.

   In this debate, the real question is how "general" distributed hypermedia is. Can it be used as a general-purpose systems architecture? Is it, in many cases, a superior set of constraints to just "contracted message exchange"? That's the core of the debate, in my mind.
   

In summary, while I appreciate its tone of this article, I think the content was very selective in its omissions, and as such should not be seen as a definitive analysis of "modeling state" with these specifications. I really hope it wasn't written to drive a political debate underground, but its primary author seems to be using it in that way based on what I read of his recent blog entry. The result would lead me to interpret this article as well written sophistry. It should, rather, serve as a useful starting point for continued discussion and improvement of our technologies.

There's a debate regarding decentralized media types.

Here's a case, I think, where the scale of solutions in a corporate Intranet are different from the solutions at Internet-scale.

Say you're in an IT department, want to use RESTful web services for your SOA, but have your own canonical XML schemas for representing data in many of your business domains. How do you register those media types?

You could

1. use the plain application/xml media type and hope people will sniff the XML namespace and hope it accurately describes what's in the document (most common, not very RESTful)
   
2. use my own media type with my own private registry (pretty common but not necessarily interoperable + consumers require a priori knowledge of where the registries are)
   
3. use the most general media type you can for the representation and a URI as a media type parameter that points to a registry with more metadata (which could lead to some interoperability, cacheability, etc.)
   
4. go back to using SOAP and UDDI. (....)
   

Obviously #3 seems to make the most sense, with caveats. I echo other commenters when I say that "application/data-format" is too general, that the metadata shouldn't just be RDDL (an HTML page may be more useful in practice!), and that the number of registries should be minimal.

Media type proliferation is a governance problem. On the Internet, the IANA is the governing body. In an Intranet, .... it depends on your governance model. What's clear is that having everyone's IT department register their own vnd media type seems both silly and untenable because those media types will not likely be general. So they'll have their own corporate&partners registry.

As for mixed vocabulary semantics, we do have a problem -- but RDF/OWL is a non-starter for most IT departments. I agree this should change some day, but, baby-steps are needed. So, what can an IT department that wants to use RESTful media types for its SOA do to indicate representation meaning *today*, without adopting the Semantic Web?

For this I imagine a registry that points to a model, whether written text, UML, ERD, or something more formal, that shows an architect or developer how the mixed elements relate to one another. In other words, use configuration management as a palliative. This does not solve the problem in general, but it arguably makes for a workable solution in a smaller scale.

So, coming back to decentralized media types, here's what I see:

• There are many that feel a need to introduce a standardized "more information on this representation" hook , beyond just the IANA media type.
  
• A URI likely is the best candidate format for this hook.
  
• Other media types are already offering this feature inside the representation body (e.g. XMLNS declarations, GRDDL declarations in HTML) ....
  
• ... But to work best with the deployed web, and to be most general-purpose, it seems this URI should be somewhere in the HTTP header.
  
• The debate is mostly matter of whether a) there is such a thing as a general purpose "more info on this media type" resource , and b) if so, where to place the link, so that it fits well with the deployed Web and doesn't necessarily cause problems for a future Semantic Web.