1. REST APIs are a case of good intentions run amok. The point of REST was to *eliminate* specific APIs. Not cause their proliferation.

2. When someone says you shouldn't document a REST API, they're saying you shouldn't have an API. Not that you shouldn't document.

I mean by this, Network APIs. I also mean APIs that aren't re-implementable across sites. The point is that the uniform interface isn't just HTTP - it's a small number of evolving media types, and the goal is to enable sharing of information at a global scale. Your pet JSON format may not be the way to do this.

3. That said, RESTians saying you shouldn't have an API are somewhat vacuous because they offer no good alternative. Custom media types ain't.

4. Thus, IMO, until we have a workable generic media type or two for the 80% case of "REST APIs", we're just growing a frustrating mess.

In other words, I completely recognize that crafting a media type is a bit heady, and even still a custom media type for generic use is quite hard. With a REST API at least you get the benefits of HTTP GET (to some degree). It does the job, if all you want is a trendy vaguely hopeful way of building something for the long term without getting too bogged down in philosophy. I get it.

What I would like to see is a generic media type (above & beyond a "format" like JSON) to cover the 80/20 cases. This is seemingly contrary to where many RESTheads are at - they prefer often prefer opinionated media types, media types that "say something".

I'm actually asking for that too, I think -- say something about the programming model that breaks people's habits away from RPCs. Find a programming model that doesn't orient all "write-side" communication in terms of a flat list of actions, and into something that can be extended & shared across the web via hyperlinks and URIs. Hierarchical statecharts is my stab at this. There may be others.

5. From William's blog (with a couple tweaks)....

I've sort of half given up on REST advocacy because it's pretty impossible these days. People want to ship APIs that will be valid for the next next 6 months, not build systems that will evolve over a decade or more.

Jan Algermissen's comment about how when you've documented an API, you're not really doing REST, is actually spot on, fundamentally, but is met with ridicule. I can completely understand how vacuous the comment sounds if you just want to ship an API nownownow, are being short-term practically minded, or are just playing buzzword bingo with architectural styles. But if we actually want to leverage the benefits of the style, we'd work on the core issue of having a generic media type or two for these sorts of use cases.

If one is thinking of "how to methods map to my URI structures", we've basically lost the plot as to why we have HTTP, URIs and media types in the first place. The goal is to *eliminate* the API as we normally think of it. (It's not about a library of calls over a network).

Five points. The consequences of REST changes some rules:

1. The assumption can't be that the end user of your information service is going to be a programmer. Building an API bakes that assumption in how you expose your information or controls to manipulate it. The assumption rather should be that "this can be any agent of any sort". If your goal really is "I want a programmer to learn my API and program against this", then there's really no reason to use REST other than that it's trendy (and HTTP GET).

2. The assumption can't be that you will be the only site implementing your interface. The whole point of the Web is sharing information - publishing an API bakes in the assumption that YOU, the publisher, are in control. It is the consumer, on the other hand - the agent itself, that's in control of how/what the information is consumed or manipulated.

3. The semantics of the interface that need to be documented are the media type that you use to describe your data and links. The structure of your URIs really shouldn't be a part of said semantics.

4. But doing this in a custom media type implies #2, that you're the only person implementing your interface. Of course, every media type has to start somewhere, but it really doesn't help matters unless you understand that others may be *implementing* your interface, not just *talking* to it. Facebook's Open Graph, for example, is an example of a custom overlay media type that takes great pains to be generic in either direction.

5. All of the above is really heady stuff that is above most developer's pay grades, so it won't ever happen. The best we can hope for is some person/organization to step up with a generic media type or two + guidelines that help describe and document a Web of Data interface for accessibility by programmers. Call it a description language, if you'd like. Some RESTians don't like the idea of a description language. I think it's an essential 80/20 piece if developers are going to actually *use* the architecture the way it was meant to be used.

Currently (as I mentioned in my WS-REST keynote) I think some kind of hierarchical lifecycle/state machine might be the right abstraction for such an interface. I'd like to demonstrate this if I ever got past my day job sucking up most of my time. (But I may have an opportunity to work on this there soon.)

Recording (75 minutes) is now available at this link. (It's an M4A / AAC file, sorry).

I'll eventually post my thoughts on the workshop along with feedback & discussion from this keynote.

Yes, I do wonder if I have Brain Damage some days ;)

Here's the abstract:

While global-scale digital information sharing has become synonymous with REST,
state manipulation remains an inconsistent experience. Many so-called "RESTful
APIs" are specified out of band, through documentation, rather than in
hypermedia. Developers struggle with constructing their own custom
hypermedia-driven media types, falling back on lightweight site-specific data
structures. And the few standardized publishing media types, such as the Atom
Publishing Protocol, have had limited adoption. All of this serves to undermine
the serendipitous reuse of "writes", a quality we take for granted when dealing
with "reads". Of course, state-changing actions are semantically much more
difficult to describe generically than a simple HTTP GET, and it is an open
question as to whether it's even possible to do so practically.

This talk will explore whether there's an opportunity to provide generalized
media types and programming frameworks to assist with resource state
manipulation, and what they might look like, drawing inspiration from other
computing domains with dynamic environments, such as video games, robotics, and
embedded systems controllers.

Slides will be posted around the same time as the keynote.

Last year had a high quality set of attendees and submissions, and was a great opportunity to meet those involved in this community.

I'm on the program committee again this year. We are accepting four-page position papers, or eight-page full research papers on your favourite REST topic. Send your papers in by Jan 31st! Even if you can't make it to India, it will be a great forum to disseminate your research!

The problem is that few web services or RESTful hypermedia describe a data model. Representations themselves don't often describe a data model, they describe a format. This is the same problem as WS-RT, in that they're trying to describe updates to a format (XML), not a data model.

In Atom, this doesn't matter, as it kicks the responsibility to the entry.
In HTML, this doesn't matter, as the machinery of browser & server only needs the format for interoperability.

In systems-to-systems communication, this matters a whole lot, as you're coupling your expectations differently with every media type.

A data model is foundational, it describes the mathematical basis for how data is managed. Examples (for better or worse) include:

1. Key/Value


2. Arrays


3. Relations


4. Logical Graph


5. Inverted List

OData describes an implicit data model, based on relations. RDF and OWL describe graph data models with an explicit theoretic foundation that makes them more powerful than Relations.

"Your New JSON Service" probably has a data model, but I bet it's unpublished, and in your head. And even if you did publish it, it probably is different from "this other guy's JSON Service".

What does all of this mean? I think it means that there's two big pieces of work to be done to help evolve RESTful web services to better interoperability:

(a) a data model that covers 80% of common use cases and can be formatted with JSON and XML. This needs to be a closed-world model, like most databases, and thus I don't think RDF qualifies.

(b) a media type that covers 80% of common use cases to describe a resource's lifecycle and state transitions -- in other words, to make POSTs self-descriptive in hypermedia. Because the world of computing isn't just about updating data, it's about abstractions.

It is passe' to perform SQL updates directly from your GUI. It's just as passe' to expect every RESTful hypermedia API to just behave like a database entity. So, we need something in between, something that gives POST meaning to systems. POST is the method which powers most of what we DO (vs. read) on the Web anyway. This "roll your own media type" trend works for simple situations, but really isn't sustainable in the long run, in my opinion.

p.s. yes, that "Part 2" of building a hypermedia agent article is coming ;-)

1. Definitions Matter. Or, are CA/CP are the same thing?

Gilbert & Lynch's paper defines the following (I'm not going to use their exact terms, so as to keep this accessible)

• Consistency (big-C)- aka Atomicity, or a total order on operations (as visible to any user); aka. by database practitioners as serializable isolation. Makes distributed systems look like a single system.
• Eventual Consistency (little-C)- aka Delayed-t Consistency, or a bounded amount of time before a data object is consistent, also implies a minimum latency between writes on a single piece of data (without specialized conflict resolution)
• Weak Availability (little-A)- every request received from a non-failing node must result in a response, but this response may take forever to come (i.e. latency is not a part of this definition)
• Strong Availability (big-A)- every request received from a non-failing node must result in a response even when network partitions occur, but this response may take forever to come (i.e. latency is not a part of this definition)
• Strong Network Partition Tolerance (big-P)- the network will be allowed to lose arbitrarily many messages sent from one node to another (not counting total network outage). When a network is partitioned, all messages sent across the partition are lost.

• Weak Network Partition Tolerance (little-P)- a system that can tolerate some network partitions, but not all combinations of them.

I add this due to my reading of their definition of Big-P and sections 3.2.1 and 3.2.2: "If there are no partitions, it is clearly possible to provide atomic, available data. In fact the centralized algorithm described in Section 3.2.1 meets these requirements. Systems that run on intranets and LANs are an example of these type of algorithms". Section 3.2.1 defines an algorithm for Consistent & Partition Tolerant systems. Hang on to this one, I'll finish my point in a moment.

So, by these definitions, you basically can have (assuming mixed reads & writes):
- AP: eventual consistency, strong availability, strong partition tolerance
- CP: strong consistency, weak availability, strong partition tolerance
- CA: strong consistency, strong availability, weak partition tolerance

Clearly an AP system is different from CP/CA. But the crux is whether CA and CP are the same kind of system. If you consider the idea of weak vs. strong partition tolerance, the difference becomes clear (to me, anyway, but I'm admittedly blonde).

A CA system might be able to handle some partitions, but not all. Simply put, it has a single point of failure, somewhere. The example I like to give is a shared-disk cluster database with a SAN. Partition the SAN, and you're hosed. Partition the nodes, or even the interconnect, and you will still be available, albeit with higher latency. Redundancy is the common way to reduce this risk.

Whereas a CP system is designed to handle all partitions, i.e. it has no single point of failure. But some non-failed nodes may not be able to service clients at all during a partition. This can suck. Anyone who's managed a synchronous log shipping setup or EMC SRDF probably knows about this one.

2. Scope matters

The definition of "node" and "network" can vary depending on your granularity and scope.

The above definitions only make sense if the scope of the system is the server-side of a client/server relationship. (i.e. How can non-failed nodes experiencing a partition still receive and respond to a request? Only if the client isn't the one being partitioned.)

A database that's normally a CA system can be considered CP if you zoom out to notice that it's in a multi-node synchronous replication ring. Or AP if it's in an asynchronous multi-node replication ring. But notice that, probabilistically, it's behaving like a CA system for 99.9% of the time (or however long your MTBF is).

An AP system on the other hand has one big drawback, one that's not spoken about often. It's about the scope of the partition: is it recoverable or not?

An unrecoverable partition in a CP or CA system is "no data loss", even if you can't get at the system. That's not true in an AP system, if there's an unrecoverable error in a certain set of nodes during the "delayed consistency" time window. This occurs during catastrophic media failures, like a bug in the SAN controller corrupting all the disks, trucks driving through a data center, or floods, or bombs, hurricanes, etc.

Even with backups, during this "replication latency", you have to re-enter those transactions, or hope that a CP or CA system somewhere has a copy.

3. Definitions are too easy to get hung up on.

One of the main reasons I'm talking about this is because I see bolded, underlined claims that can be easily misconstrued, like, "you can't choose consistency and availability together in a distributed system". This gives me The Fear.

This point of the above quote, is not to say you can't have some consistency. You just can't have fully serializable consistency and isolation.

But in practice, when you think about it, this claim is actually rather banal. Practitioners have long understood that there are a spectrum of consistency & isolation tradeoffs at scale, even at a single node. Why?

Because the definition of availability arguably needs to also include latency, which is something Daniel Abadi brought up. Serializability means locks, in practice, and that means increased latency, and thus reduced availability. This is why we don't like Two Phased-Commit very much these days.

But, for kicks, go back to the ANSI SQL isolation level debate that Jim Gray stirred up. It was because databases like Oracle (gasp) don't provide serializable consistency, and haven't since the late 1980s! They provide snapshot consistency, which is a different beast where readers don't block writers, and arguably was the primary technical reason for Oracle's success in the market place through the 1990s. Amusingly, this is the same argument that Jim Starkey keeps bringing up when discussing CAP, having invented the idea of multi-version concurrency control, when he's talking about his new baby, Nimbus DB.

So the idea that you can't have fully serializable consistency at scale -- even in a single node database -- is completely uncontroversial. It's when the followup feels like "...and therefore you need to throw out your RDBMS", that cognitive dissonance and tempests in a tweetpot stirs up.

• You Can't Sacrifice Partition Tolerance by coda hale


• Problems with CAP by Daniel Abadi


• Clarifications on the CAP Theorem by Mike Stonebraker


• Reactions to Coda's CAP Post by Jeff Darcy


• Someone is Wrong on the Internet by Jeff Darcy

I am confused. I've narrowed it down to 7 matters.

Confusion #1: Changing system scope from underneath our feet
The biggest confusion I have is that the scope of "distributed system" under the CAP theorem seems to have changed, silently.

If you look at the original papers:

• Lessons from Giant-Scale Services by Brewer


• Harvest, Yield, and Scalable Tolerant Services by Fox & Brewer


• Toward Robust Distributed Systems presentation by Brewer


• Brewer's Conjecture and the Feasibility of Consistent Available Partition-Tolerant Web Services
by Gilbert & Lynch

All of them seem to scope the CAP theorem to the server side of a client/server relationship. That is, it's assumed that the distributed system under consideration is a client/server architecture, but the server-side is distributed in some way. This is a reasonable assumption, since the Web is a client/server architecture variation, and CAP was originally about the tradeoffs in huge scale web sites.

But practically speaking, this is particularly important when considering Partition Tolerance. In the original papers, CA (Consistent, Available) was absolutely common, and in fact, the most common scenario, with Brewer giving examples like cluster databases (on a LAN), single-site databases, LDAP, etc. The most popular commercial cluster databases today arguably retain this definition of CA, such as Oracle's RAC.

But now, AP (availability, partition tolerance) advocates are questioning if CA distributed systems are even possible, or merely just something that insane people do. It's a very strange, extreme view, and I don't think it offers any enlightenment to this debate. It would be good to clarify or at least offer two scopes of CAP, one within a server scope, and another in a system-wide scope.

Confusion #2: What is Partition Intolerance?

This is never well defined. Like lactose intolerance, the results, can be, err, unpredictable. It seems to imply one of: complete system unavailability, data inconsistency, or both. It's also unclear if this is recoverable or not. Generally it seems to depend on the implementation of the specific system.

But because of this confusion, it's easy to let one's mind wander and say that a CA system "acts like" an AP or CP system when experiencing a partition. That leads to more confusion as to whether a CA system really was just a CP system in drag all along.

Confusion #3: Referring to Failures Uniformly

It seems to be a simplifying assumption to equate failures with network partitions, but the challenge is that there are many kinds of failures: node failures, network failures, media failures, and each are handled differently in practice.

Secondly, there are systems with many different kinds of nodes, where they tolerate partitions in one kind of node, but not another kind, or just maybe just not a "full" partition" across two distinct sets of nodes. Surely this is an interesting property, worthy of study?

Taking Oracle RAC or IBM DB2 pureScale, as an example. It actually tolerates partitions of many sorts - cluster interconnect failures, node failures, media failures, etc. What it doesn't tolerate is a total network partition of the SAN. That's a very specific kind of partition. Even given such a partition, the failure is generally predictable in that it still prefers consistency over availability.

Confusion #4: Rejecting Weak vs. Strong Properties

Coda hale sez: "You cannot, however, choose both consistency and availability in a distributed system [that also tolerates partitions]".

Yet this seems very misleading. Brewer's original papers and Gilbert/Lynch talk about Strong vs. Weak consistency and availability. The above statement makes it sound as if there is no such thing.

Furthermore, most distributed systems are layered or modular. One part of that system may be CA-focused, the other may be AP-focused. How does this translate to a global system?

Confusion #5: Conflating Reads & Writes

Consider the Web Architecture. The Web is arguably an AP system for reads, due to caching proxies, but is a CP system for writes.

Is it useful to categorize a system flatly as binary one or the other?

Confusion #6: Whether AP systems can be a generic or the "default" database for people to choose

AP systems to my understanding are application-specific, as weak consistency is difficult to correct without deep application knowledge or business involvement.

The question and debate is, what's the default database for most IT systems... should it be a traditional RDBMS that's still a CA system, like MySQL or Oracle? Or should it be an AP system like a hacked-MySQL setup with Memcached (as AP), or Cassandra?

When I see claims that CA advocates as "not getting it", or "not in their right mind", I have a severe case of cognitive dissonance. Most IT systems don't require the scale of an AP system. There are plenty of cases of rather scalable, highly available CA systems, if one looks at the TPC benchmarks. They're not Google or Amazon scale, and I completely agree that an AP system is necessary under those circumstances. I even agree that consistency needs to be sacrificed, especially as you move to multi-site continuous availability. But it's very debatable that the basic database itself should be CA by default or AP by default.

For example, many IT shops have built their mission-critical applications in a CA manner within a single LAN/rack/data-centre, and an AP manner across racks or data-centres via asynchronous log shipping. Some can't tolerate any data loss and maintain a CP system with synchronous log shipping (using things like WAN accelerators to keep latency down). And not just for RDBMS - this is generally how I've seen Elastic Data Cache systems (Gigaspaces or Coherence or IBM WebSphere ExtremeScale) deployed. They deal with the lack of partition tolerance at a LAN level through redundancy.

It's useful to critique the CA single-site / AP multi-site architecture, since it's so common, but it's still not clear this should be replaced with AP-everywhere as a useful default.

Confusion #7: Why people are being a mixture of dismissive and douchey in their arguments with Stonebraker

Stonebraker's arguments make a lot of sense to IT practitioners (I count myself as one). He speaks their language. Yes, he can be condescending, but he's a rich entrepreneur - they get that way. Yes, he gets distracted and talks about somewhat off-topic matters. Neither of those things implies he's wrong overall. Neither of these things imply that you're going to get your point across more by being dismissive or douchey, however good it feels. (I'm not referring to anyone specifically here, just a general tone of twitter and some of the snark in the blog posts. )

One particular complaint I find rather naive - that he's characterizing CAP theorem (more AP system) proponents as only using CAP theorem to evaluate distributed system availability. Clearly that's not true, but on the other hand, it is kind of true.

Here's the problem: when I hear venture capitalists discussing the CAP theorem over cocktails because of their NoSQL investments, or young newly-graduated devs bugging their IT managers to use Cassandra, you can bet that plenty of people are making decisions based on a twisted version of the new fad, rather than reality. But that's how every technology fad plays out. I think Stonebraker has seen his share of them, so he's coming from that angle, from my view. Or he could just be an out of touch dinosaur like many seem fond of snarking. I personally doubt it.

I'm no longer involved full-time with Elastra; I do remain a part time advisor. One day the story will be told. Until then, I highly recommend reading Steve Blank's blog & books for an apropos understanding of how startups often struggle and remain stubborn in the face of said struggle. On the bright side, the current team has their head screwed on straight and is still doing great work.

What am I up to? Consulting, for now at least! Jetstream Cirrus is my company, and yes, there are intentionally many ways of reading into that name. I'm offering independent advice & implementation on integration architecture, REST, agile operations, scalable systems, and cloud computing consulting in the U.S. and Canada.

It may continue to amaze, but the most basic aspects of interoperability continue to have a common problem: the implementation almost always violates the protocol specs that it supports. And that violation often gets encoded into practice, and back into the spec. Sometimes this is for good reasons, sometimes not.

Sam drew from his experience with early SOAP interoperability to illustrate the problem. Apache SOAP, for example, a descendent of IBM SOAP4J, and the predecessor to Apache Axis, had a history of supporting the early SOAP spec, when it was a vendor-led W3C note. It had encoded data with the SOAP encoding section of the SOAP specification, because the XML Schema data type system was not due to be out until mid 2001.

Of course, early specifications often under-specify, and that leads to implementation leaking through. A classic borderline cases is representing "Infinity". XML Schema suggests that infinity should be represented as the characters "INF". Apache SOAP, predating this, used the java.util.Number class, whose toString() method generated "Infinity".

The fix is simple enough, but there were deployed systems out there already. The solution was to continue to produce "Infinity", but to accept both "INF" and "Infinity". Future SOAP stacks, like Axis, which were built around XSD in mind, both produced and consumed INF.

My first reaction to a lot of this was, "conform to the (damn) specification". If it were only so simple. Sometimes the spec is wrong or incomplete. Sometimes a boundary condition is missed by implementers. And organizational inertia makes it hard to upgrade. So, documenting what is implemented becomes the preference.

None of the above has really any difference with REST. One of the takeways: contracts matter.

I know that there's trepidation about the term "contract" among those in the REST community, in that it may envision a rigid interface. I think it's a price worth paying -- you cannot eliminate the notion of "contract" when interoperability is a concern. Contracts are not about how rigid or flexible the interface is. They are about ensuring complete, understood, and compliant expectations. And, frustratingly, they cannot be well-specified if there isn't any implementation experience.

One might argue that the Web has proven very good at composing contracts together -- transfer protocol, media types, identifiers, etc. It didn't eliminate the painstaking need for agreement, it depends on it. Those interested in philosophy might find this an interesting paradox, that the explosion of freedom & diversity of web content and applications requires strong control at the lower layers -- an idea Alexander Galloway explores in his book, Protocol: how control exists after decentralization. Practically speaking, the areas of the Web where there was lax specification and/or complex implementation compromises are the areas where there's not a lot of diversity - such as browsers, which have to deal with HTML, CSS, JS, etc. Servers haven't been immune from this, but they have somewhat more understandable specs to follow, with HTTP, MIME, URI, etc., though ambiguity with HTTP has led to a dearth of complete HTTP 1.1 deployments (most use a subset). This has led to the work on HTTPbis to clean up and clarify the spec based on over a decade of experience.

Considering today's new generation of RESTful APIs on social media & cloud platforms, there are some curious trends. JSON-based RESTful services, for example, do not specify dates, or formal decimals. XML (schemas) do have standardized simple types, but the syntax and infoset is off-putting due to its lack of fidelity with popular programming languages. Which is better for interoperability? Maybe the answer is because JSON is so easy to parse, it's ok that each endpoint needs to adapt to a particular set of quirks, like differing date/time formats. But that sounds like a very developer-centric argument, something that serves the lazy side of the brain. We can do better.

Sam Ruby's keynote began with a call for technologists to back off the extreme positions often taken in technical arguments (likely because he's knee deep in the middle of the mother of all technical standards debates). Using Mark Pilgrim's parable of the web services spec developer fighting to the death over Unicode Normalization Forms, he illustrates that technical advocacy can lead to knee jerk, us vs. them reactions on very minute details, especially when different mindsets collide and have to work out a path to interoperability.

Relating this to the "REST community" (whatever that means), I think we seem to have a built-in defensive posture, and aren't quite used to being in the driver's seat for innovation. There were so many attempts to "fix" the Web into some other architecture: HTTP-ng tried with a distributed object model, the Web Services crowd tried again. The slow counter-revolution of "using the Web" was largely grown out of Mark Baker's health and savings account. And now, REST is now a technical pop culture buzzword, being perverted in meaning as all buzzwords are prone to. Roy's gotta defend the meaning of the word if it's to mean anything. So, in a way, it's completely understandable why RESTafarians honed a defence mechanism against misleading arguments that they fought ad nauseam.

Unfortunately, today, this defence mechanism seems to have limited free discussion of the limitations of REST, the areas of useful extension, the design of media types, and the ability to relate or objectively discuss extensions to REST that may evaluate the benefits of decades-old ideas like transactions, reliable messaging, or content-level security, without being patronizing ("you don't need transactions", or "reliable messaging isn't really reliable", "use SSL, it's easier", etc.). Researchers and standards working groups try to follow the style, or publish extensions, but anticipate the dreaded Roy bitchslap, perhaps because there seems to be a lack of resources (tools, documentation, frameworks, examples in different use cases) to learn how to apply the style, or perhaps because people are being lazy in applying the style (glomming onto a buzzword). Hard to tell.

In short, this is a hard problem to resolve, as there are good reasons for people's strong opinions. Compromise can only be achieved if there's shared understanding of the facts & shared values. Mostly I see shared values in terms of the properties that people find desirable in a networked system (interoperability, extensibility, visibility, etc.), we still seem to be missing a shared understanding of the facts (such as the basic idea of "emergent properties" in software architecture, and the idea that they can be induced through constraints -- which I think many still reject.)

To be fair, there have been plenty of extensions out there, include the ARRESTED thesis from Rohit Khare, various public RESTful APIs and frameworks out there, Henrik Nielsen's use of RESTful SOAP services in Microsoft Robotics studio, etc.

Look at the reference to the latter, for example. There are plenty of "specific methods" being defined while fitting under the POST semantics. Some would have a problem with that, but I'm not sure what the alternative would be. There are "state replacement" methods under the PUT semantics if you prefer that. But some clients may prefer the more specific, precision state transfer of a POST method. Would you consider this heresy? Or would you stop and realize that Henrik was in the front row of the WS-REST workshop, and co-authored too many W3C notes and IETF RFC's to enumerate here?

I'll speak more about this problem with the "write side of the web" soon, but in short, we don't really have a good way of describing the pre-conditions, post-conditions and side-effects of a POST in a media type, which would be probably the more "RESTful" thing to do. Mark Baker took a stab at this 8 years ago, but there's hasn't been much pickup. In lieu of that, the simplest thing might be to make an engineering decision, slap a method name & documentation on the URI+POST interaction, and move on. You'd be no worse than a WSDL-based Web Service, with the added benefit of GET and URIs. Or, depending on your use case, it may mean crafting a full media type for your system, though I'm not sure that's "simple".

This dilemma is why I tend not to have a problem with description languages like WSDL 2.0 or WADL -- they're not complete, certainly, but they're a start to wrapping our heads around how to describe actions on the Web without breaking uniformity.

But, coming out of the WS-* vs. REST debates, it is interesting in noting what hasn't happened the past two years, since the W3C Workshop on the Web of Services, which was a call to bridge-building. As expected, SOAP web services haven't really grown on the public-facing Internet. And, also as expected, RESTful services have proliferated on the Internet, though with varying degrees of quality. What hasn't happened is any real investment in REST on the part of traditional middleware vendors in their servers and toolsets. Enterprises continue to mostly build and consume WSDL-based services, with REST being the exception. There has been, at best, "checkbox" level features for REST in toolkits ("Oh, ok, we'll enable PUT and DELETE methods"). Most enterprise developers still, wrongly, view REST as "a way of doing CRUD operations over HTTP". The desire to apply REST to traditional integration and enterprise use- cases has remained somewhat elusive and limited in scope.

Why was this?

Some might say that REST is a fraud and can't be applied to enterprise integration use cases. I've seen the counter-evidence, and obviously disagree with that assessment, though I think that REST-in-practice (tools, documentation, frameworks) could be improved on quite a lot for the mindset and skills of enterprise developers & architects. In particular, the data integration side of the Web seems confused, with the common practice being "make it really easy to perform the required neurosurgery" with JSON. I still hold out hope for reducing the total amount of point-to-point data mapping & transformation through SemWeb technologies like OWL or SKOS.

Perhaps it was the bad taste left from SOA, or a consequence of the recession. Independently of the REST vs. WS debate, SOA was oversold and under-performed in the market place. Money was made, certainly, but no where near as much as BEA, IBM, TIBCO, or Oracle wanted, and certainly not enough for most Web Services-era middleware startups to thrive independently. It's kind of hard to spend money chasing a "new technology" after spending so much money on the SOAP stack (and seeing many of the advanced specs remain largely unimplemented and/or unused, similar to CORBA services in the 90's).

Or, maybe it was just the consequence of the REST community being a bunch of disjointed mailing list gadflys, standards committee wonks, and bloggers, who all decided to get back to "building stuff" after the AtomPub release, and haven't really had the time or inclination to build a "stack", as vendors are prone to.

Regardless the reason, the recent uptick in activity hasn't come from the enterprise or the enterprise software vendors offering a new stack. The nascent industries that have invested in REST are social computing, exemplified by Twitter, Facebook, etc. and cloud computing, with vCloud, OCCI and the Amazon S3 or EC2 APIs leading the way.

The result has been a number of uneven media types, use of generic formats (application/xml or application/json), mixed success in actually building hypermedia instead of "JSON CRUD over HTTP", and a proliferation of toolkits in JavaScript, Java, Ruby, Python, etc.

We're going to be living with the results of this first generation of HTTP Data APIs for quite some time. It's time to apply our lessons learned into a new generation of tools and libraries.

This turned into the mother of all blog posts, so I've chunked it up into a series. The tl;dr version of my workshop takeaways are the following:

- Investment in RESTful technology is growing, particularly with cloud and social computing. Enterprise and vendor investment has been low.

- Technical disagreements and arguments from authority are counter-productive, and seem to be ubiquitous. Now is the time to build stuff, and make stuff interoperate, not split hairs. On the other hand....

- In building interoperable systems, sometimes domain experts are wrong, sometimes they're right. Sometimes it's better to document and standardize what works, sometimes that's not good enough.

In short, interoperability is hard, REST doesn't change that, better get a good helmet if you're in a group specification or standards body these days. The most important thing is if you're going to willfully violate someone else's idea of "the Right Thing", then be explicit and honest about it, so that it can be discussed. (This is, in part, the saga of the HTML5 standards effort.)

- REST has become a buzzword among those that aren't architects and don't want to be architects. That means there needs to be better documentation, tools, and frameworks; most developers can't derive the right design from first principles. Most published RESTful APIs published today are not as interoperable as they could be; there will be a price to pay to maintain these.

- Having said that, for architects, progress has been slow in driving a true discipline for architecture, though there is some great work afoot to make this easier

- Hypermedia tends to be a somewhat jarring user experience to those that still love their desktop applications, especially when we build small protocols like OAuth , but fall back on HTML to do other pieces like authentication and authorization policy selection.

- Developers tend to love dynamic binding nature of REST so long as they're not the ones that have to maintain the client. We need a better way of handling dynamic binding (my suggestion is a work in progress) to hypermedia.

- It's high time to work on the "write side" of the web. We know that HTTP GET works well for interoperability. We have had less success with write actions, and need to think through interoperability there to a much greater degree than we have. Enterprise
integration use cases, for example, could really benefit from describing POST interfaces for domain application protocols. The best we have is WSDL, WADL, and generic forms like XForms, HTML Forms, and generic publishing containers like AtomPub. We can do better.

- You can use REST in the enterprise, for legacy integration, successfully.

- Peer to Peer systems quite achievable in RESTful systems. The nature of hypermedia itself is peer-to-peer. And minor extensions to HTTP can lead to full data propagation style systems.

- Progress is being made to formally describe a RESTful Semantic Web, one that leverages the formal descriptions of tuple spaces with the polyadic pi-calculus and extends them to RESTful architectures.

- While HTML 5 is progressing to be a rich platform for the next generation web, there's experimentation happening for alternatives. Some drive traditional desktop UIs through RESTful resources, others are describing mixed reality worlds.

I have some forthcoming blog posts musing on these themes.

The normal way we build clients, in a client/server architecture, looks something like this:

The logic of the application - its objectives and how it wants to achieve them through one or more remove services, is often procedural. A rich OO domain model is sometimes preferred to procedural logic, but this isn't usually used in conjunction with remote services because of the latency involved; a service facade coalesces communication into coarse grained interactions.

This idea of a service facade culminates in SOA, where interfaces, along with all their possible message exchange patterns, are registered for others to lookup:

A agent-oriented client, on the other hand, looks something this (which I've adapted from Russell & Norvig's diagram):

The application agent has several pools of pre-defined logic:

a) Application Logic: some logic for the application itself (e.g. the basic states of a hypermedia application, the goals of the application if it has any. A browser has no goals other than rendering; whereas a product ordering & payment agent would have the goal of completing e-commerce transactions on behalf of a user)

b) Action Logic: some logic for the implications of actions (e.g. how does a payment & product ordering agent know, interpret, or infer that PUT/POST/DELETEing to a particular sequence of URIs will result in a paid product order)?

c) Protocol Logic: some built-in logic for handling protocols & media types (e.g. URI, MIME, HTTP, and maybe some mix of HTML, Atom, Atompub, etc.).

The problem of bridging together application and action logic together is known as Action Selection. Action selection doesn't require fancy algorithms. Its study has often dealt with complex subject matter, which has often lead to complex solutions. But in most agents, the bread and butter for action selection is simple: the Finite State Machine (FSM). An agent responds to changes in the environment based on its current state and a set of known transitions. There are other approaches to agent programming that are growing in popularity, like planners, but let's start with FSMs.

Firstly, an agent's application logic requires a state machine to describe the relationship between sensing ("safe") actions and changing ("unsafe") actions. In a hypermedia application it looks something like this:

This basic hypermedia application state machine is sandwiched hierarchically between the super-state machine for the application's goals and the sub-state machines for the protocols:

The trick with building a RESTful agent based on FSMs would be to figure out a way such that

a) The application's goals can be expressed in terms of hypermedia agent logic (e.g. sensing & effecting)

b) The hypermedia types and link relations themselves contains enough interpretable action logic that can be mapped to the application's domain

c) The action and protocol state machines are modular. RESTful applications tend to have standardized and relatively small number of generic protocols, so they need to be repurposed for different applications and/or contexts.

Two ways of accomplishing this include hierarchical FSMs and behaviour trees.

Hierarchical FSMs are popular in control systems and game engines. They are great for reactive systems, where the correct interpretation & response to input and events is the intent of the application. Managing call control, or a climate control system are examples. There are powerful generic Hierarchical FSM standards out there like SCXML that provide a code-on-demand approach to interpreting and managing states across a set of resources (though it probably could use some RESTful-friendly polish).

Behaviour trees have the same power as hierarchical FSMs, but tend to be more oriented towards goal-based applications, where the purpose of the application is to transition a bunch of resource state to some new state. For example, a calendar scheduling agent, or a payment & ordering agent, are examples of goal-oriented agents.

In future, I'm going to explore how to build a behaviour tree-based agent; probably for the Restbucks domain that Jim Webber, Savas Parastatidis, and Ian Robinson have been using for the past year or so and including in their "REST in Practice" book.

This is my attempt to summarize an overview of my thinking on RESTful versioning. It's a follow up to Square Peg, REST hole. These concepts can be tricky concepts to describe, and I don't really want to write a small book on this topic, so I may get some of this wrong. Thus, expect updates to this entry to improve it in the future.

Data Versioning vs. Language Versioning

Extensibility and versioning in RESTful services can be viewed in terms of two domains of agreement. The two domains are: resource and representation, which could also be thought of as the "data" vs. "language" domains.

First, let's recall what a resource is: a time varying membership function, where the members are instances of a representation at various points in time. The resource can return different values at different times. BUT resources can be narrowed down into very specific semantics, if resource owner wishes. A resource might be "the most recent version" of a record, whose state might change often, or it might be a "specific version" of a record, and thus unchanging in state. These are two different resources, even though they may have the same representation for a period of time. A resource may even contain format metadata and constrain the language emitted, though content negotiation may be preferred.

Regardless of how often the values change, the semantics of the resource should not change. "Revision 3 of purchase order 123" should retain that meaning. If they do change the meaning, it hurts consumers that relied on the old meaning.

When we think of URI versioning, this is a design choice when resources are immutable across time and we create new resources for state changes (similar to how we manage time-series data in a database).

With language extension or versioning, on the other hand, the state is unchanged, but the way that data is represented has changed.

On Language Versioning

Rule #1: Prefer to extend a language in a forwards and/or backwards compatibile manner. Version indicators are a last resort, to denote incompatible changes.

Extension, of course, requires thought. It implies well-specified interpretation policies for language consumers, and in the case of a machine-readable schema, well-specified extension points. But the range of choices aren't too hard to understand.

This table summarizes the current techniques in practice for extensible or versioned languages, using the terminology from the W3C TAG's draft versioning compatibility strategies document, by David Orchard, which I'm going to butcher through my own brief summaries.

Some explanations...

Version Notifications

Agents should be notified of new versions. This can be done out-of-band (email, physical letter, carrier pigeon), but it helps to complement this with links. These links could be an extended, and agreed to link relation, and/or as part of the media type specification. The links may point to a description of the version change, or, in the case of a Side-by-Side, the URI that emits the resource in the new language version.

Replacement

This implies that origin server is replaced by a new backwards-compatible version that is able to accept both old versions and new versions of representations sent by a client (usually via a POST link). This is useful in combination with a forward compatible change -- none of the links need to change.

Side by Side

This implies that the origin server provides a new MIME type or URI-space for resources using the new language, along side old resources using the old language. In either case, you are impyling "this language changes everything". In the case of changing URIs to reflect the new language verison, in effect, you're using "resource versioning", something usually relegated to storing time series data , as a means to work around your language compatibility problems.

To make this RESTful, your media type must include a link from the old resources to their new version, along with metadata indicating the version of the language used at the URI, possibly including a link to a machine-readable schema of the new version (if your media type has such a thing, like XML with Relax NG or XSD). In the case of a new MIME type, you would want a link relation that notes an alternate format is available.

Let me underscore this: You cannot expect clients to understand your URI format and swap out all occurrences of "v1" with "v2"., if you do, you're placing a heavy burden of coupling on your client, that YOUR SERVER is so special, that they need to understand YOUR URI format. This is completely antithetical to why we would want to use REST in the first place, unless you're really just tunnelling XML over HTTP for the heck of it. I note that many "REST APIs" out there actually are built this way, which means they're just as point-to-point coupled as other interface styles.

Must Accept Unknowns

If the consumer sees elements in the data it doesn't recognize, it still accepts the representation. Generally, it ignores these elements for processing.

Must preserve unknowns if persisting state

This is an optional follow-on from "Must accept unknowns", and is often forgotten. If representation state is being persisted (i.e. cached) in the consumer's domain for later use, the unrecognized elements should be preserved, and not stripped. This could greatly assist forward compatibility when the client is upgraded to handle the previously unrecognized elements.

Version identifier substitution model

I defer to Section 5.3 of the compatibility strategies document.

Where do you place the version identifier?

In order of preference:

1. In the media type content


2. In the MIME type itself, or as a MIME type parameters


3. In the URI

Version identifier inside the media type content

This has many examples in the wild, such as HTML DOCTYPE, some uses of XMLNS, a version identifier inside your PDF document.

This requires the replacement model for backwards-compatibility, and encourages the greater use of forwards compatibility. It's the way that most web media types have long worked, with varying degrees of success, but note that those formats were long designed with forward compatibility in mind.

It's still possible to combine this approach into side-by-side versioning if need be, especially if you are changing the semantics of your resources.

Version identifier in the MIME type

e.g. application/vnd.mytype;version=2

This is currently a non-standard and debatable technique. The benefit here is that this enables side-by-side versioning without impacting the URI-space. On the other hand, this reeks of avoiding hypermedia and trying to push things to the other layers of the Web Architecture (HTTP and/or URIs). But in many cases this is preferable to a new URI space.

Version identifier in the URI

e.g. http://example.com/data/v1/po/123

I described the primary problem here earlier: you can't assume you are a special snowflake and the client will know that 'v1' is your magic crystal. You must provide a link or a URI template in the media itself (and/or in a service resource) to denote new versions.

The secondary problem is bookmarks, or inbound hyperlinks. In a database system these are known as "foreign keys". Anyone with a relational data background knows that their primary keys really shouldn't change, as it's expensive to propagate that change to foreign keys.

There is, however, one case, where this approach is preferred over the others. This ties back to the beginning of this entry, when I discussed "Resource Versioning". It's clear we mint URIs when the semantics of the resource itself changes. So, if they change with the language, then mint new URIs -- using hypermedia, if possible, to link old concepts to new ones, as this requires a side-by-side compatibility approach.

For example, if we have an Account resource, and a new version of our resources and language we are deprecating the notion of account, and adding two new resources, "Customer" and "Agreement". It makes no sense to preserve the Account URIs for new Customer resources in this case, as the changed meaning would be confusing to clients expecting an Account.

Some Q&A

Aren't bookmarks the problem? Wouldn't life be better if we rejected bookmarked URIs?

Well, yes, they're a problem, but no, life would suck if we rejected bookmarks, because there's no different between a hyperlink and a bookmark. It would be like saying "no one can hyperlink to me", which is absurd.

Wouldn't versioning be simpler if we separated access from identification, like with WSDL services?

If my data identifiers become opaque primary keys like 123 instead of http://example.org/po/123, then they're tightly coupled to the service that produced the document, as it would be the only context in which I could resolve details for that identifier. Now clearly one benefit is, if I create a new incompatible side-by-side service version, technically (assuming I don't need to re-key my database), the stored foreign keys don't change.

In a RESTful approach, URIs are your "foreign keys", and if you embed a version identifier in them, they need to change when you upgrade to the next version if you embed those versions in the URI. Assuming you can't convince your resource owners to use languages with version identifiers as a MIME parameter or inside the language itself, how is that done?

In a word, lazily.

As I've discussed above, your media type should have an extensibility section or link relation(s) that points to the new version. And upon retiring a language at a particular URI, you would use a permanent redirect (301) to tell all consumers to update their bookmarks / foreign keys. In either case, the agent would have the ability to update their persistent reference.

Again, this is a special case -- there really shouldn't be that many incompatible versions, they should be forward-compatible changes that dont' require new URIs unless you're completely mucking with the resource semantics.

In Summary

1. Prefer extensible, forwards & backwards compatible languages and the replacement approach to compatibility. Note the W3C TAG's position on version identifiers
2. Be judicious when you use version identifiers in URIs, as cool URIs don't change
3. For side-by-side deployments, always include a section in your media, or link relation(s), to point to new/old versions, and update references lazily as the consumer refreshes its cached value. Use permanent redirects to retire URIs bound to old language versions.
4. Version URIs if the semantics of the resource changed, but be courteous to consumers by ensuring links are available to denote the old vs. new alternates
5. Chapter 13 of Subbu's wonderful new book RESTful Web Services Cookbook provides more detailed illustrations of several versioning techniques.

Which I'll try to get to this weekend.

For now, I leave you with two things:

First, reflecting on Wiliams' recent Square Peg, REST Hole, I draw from the archives, What are the benefits of WS-* or proprietary services?

Let's keep our eye on the prize. REST is a style aimed at extensible, low entry-barrier, multi-organizational, confederate information sharing and communication. I note that most IT organizations are confederacies, adopting a federal or feudal governance model.

The Web Architecture itself (MIME types, HTTP, URIs) provides a much-needed stable intermediate form for interoperability among many different systems and applications -- something that an general-purpose orientation, like SOA, doesn't really provide. Or, fitting for RESTafarians, it is a shared hallucination ;-)

Not every system, or layer of an enterprise's architecture, has the same requirements for scalability or interoperability. The post from 2007 highlights such examples.

Secondly, the song, which ruled my college years in Canada....