June 2008 Archives

RESTful Design Guidelines

| No Comments | No TrackBacks

General musings about RESTful design that I'd like to write down. Feel free to comment.

For the love of ponies, stop trying to change HTTP to suit your world view.

The uniform request semantics of HTTP are generic for a reason -- it's hard to be uniform if you're not generic. If you want to describe or interpret something more specific, you'll need a media type to help you out.

The biggest problem with applying the Web Architecture for enterprise systems is that we have no broad community (to my knowledge) that's experimenting in media types to help solve problems that are common inside enterprises. That, I think, is a cultural and psychological limitation, of both the REST community and enterprise developers -- and not due to the Web Architecture being inappropriate for enterprise systems-of-systems interaction.

The main problem when you jump into the enterprise with the Web Architecture is the semantic interoperability problem. This is the same problem with other architectures, but with REST's benefits of easier interoperability, it becomes more important. We might be able to specify media types for infrastructure-level topics, like identity, authentication, or syndication, or content publishing, but we have much less success with, say, "Job Applications" or "Order Management", because those domains change faster than a media type should, and the semantics themselves are subject to disagreement across trust boundaries.

The takeway is not "and thus REST is useless in the enterprise". The point is that we have a lot of work to build media types and tools to bridge the gap between the communication actions that are uniform and the business actions that are non-uniform. See below for more on this.

REST is not CRUD.

PUT means "replace" but doesn't imply a byte-for-byte guarantee that what you send will be what you GET later. It means "here is my desired state for this URI, please replace it". It cannot be used to replace POST which means "process this please, impacting whatever resources you think make sense".

Furthermore, DELETE does not guarantee you will get a 404 Not Found or 410 Gone at the URI afterwards. There may still be a representation at that URI later!

HTTP is an abstract interface about describing agent intent. That's it. All of the intended specific side effects and preconditions need to be described in hypermedia. You can't infer too much from the methods.

State transition are a useful way to describe the lifecycle of both resources and representiations; but note that they're not necessarily the same thing

A little known fact about UML 2 statecharts: there are two types. The "protocol statechart", describes the lifecycle from the perspective of interaction, but says nothing about internals. An "object statechart", on the other hand. describes the internal states of an object.

A hypermedia representation type describes the former -- it's how an agent knows what hyperlinks are supplements to the current representation, which links are a sensing action to a separate resource, and which links are an state-transitioning action. Perhaps these categories are wrong-headed; maybe there are other sorts of links. But those three came to mind based on what I know of HTML inline links vs. anchors vs. forms. If we're to build more RESTful agents, we need to work on coming up with a general sense of how different sorts of links impact the knowledge of the agent.

BTW, only the origin server knows the internal object lifecycle, and how it corresponds to the protocol state transitions. They may be very dissimilar in practice. That's the whole point of abstraction.

Requesting domain-specific action on a resource requires the use of POST.

State transitions in the Web are modeled with PUT if you modify the representation of the state directly, or POST if you're specifying a more domain-specific (aka "business") action. Anyone who says POST is anathema to the Web is making shit up. POST is essential to the Web Architecture. If you want to remove it in your world, fine, but it's not the Web, and won't help the Web.

The problem with POST is when we abuse it by having it perform things that are more expressive in one of the other methods. GET being the obvious one that needs no hypermedia description. For the other methods, a good design guideline is that you MUST not break the general contract of the HTTP method you choose -- but you SHOULD describe the specific intent of that method in hypermedia.

For example, POST in HTML means "process this form". POST in AtomPub means "create this entry". They're very different intents. The meaning of these actions aren't inherent in POST, they are inherent in the media types (specifically the hyperlinks in an HTML form tag or an AtomPub collection tag). This is no different if you wanted to design an "order" action for an Order Processing media type.

I note that the RESTful Web Services book doesn't like POST, but they did invent their own REST-influenced architecture, ROA, which you can feel free to agree or disagree with. Much of their advice is well intentioned and right, but this one case is, in my opinion, very wrong.

Encoding a business domain in a media type

If you're using a media type to describe domain-specific data, then you have two choices:

a) use a domain-specific media type, and deal with the versioning issues,
b) use a more generic media type that describes actions and their consequences.

(a) is not aligned with the goals of REST because media types aren't meant to change very often. Contracts that change are supposed to be described (and discovered) through hypermedia. Unfortunately, we don't have a media type that does this. (On the other hand, most other approaches don't describe contracts all that well either).

But wait -- one may say, "what about RDF?" Indeed, RDF (and RDFa) does cover a good chunk of this problem. But besides issues of RDF usability, we don't really have a well-understood way of using RDF to describe *contracts* of action
successor-state axioms (i.e. state transition post conditions). It probably could do this, we just haven't really used it that way, to my knowledge. If Linked Data is to be a success, we need RDF (or some other description framework) to help describe what a POST or a PUT means for a particular hyperlink. I have my own ideas on how to do it, but I doubt I'm the only one who has thought about the problem; I just haven't seen too many solutions yet.

On using colloquial XML to describe business domains

One could use colloquial XML (i.e. XML elements that correspond to domain entities and relationships) as your media type and build forward compatibility into the squishy areas; this isn't much different to how people do things in the enterprise today with XML messaging. Just make sure you get your change management straight. Plan for forward compatibility in the XML schema, backward compability in the origin server, and versioning either in the URIspace or the schema, depending on how incompatible the version is. Contrary to rumor, RESTful versioning really isn't all that different than versioning with plain XML messaging, and it's arguably much smoother for forward compatibility than alternatives.

When using colloquial XML in this way, you're not really getting all of the benefits of REST, since, by definition, your media type is way more specific than it should be. The main benefit of building such interactions on the Web architecture is the pervasive use of URIs instead of just embedding primary keys into your documents, a practice that's very common and a big problem with data management in SOA. URIs give the benefit of at least some well-understood semantics (e.g. GET and response codes) without requiring a whole lot of media type description.

That's enough for now. I have more to come...

A belated response of sorts to JJ's post back in May...

Stu, I work for a BEA shop and my management had asked me to elaborate on all the SOA domains that you guys have defined. I can assure you that you guys were millions of miles away from providing guidance that would deliver successful SOAs. You simply did not get it.

Compared to your vision, perhaps. Though I'll note that not very many people understand your vision. This isn't to say you're wrong -- I happen to agree with some of what you say, particularly around interaction, though I don't certainly agree with everything. But you continue to denigrate pretty much everyone in this industry as clueless, whereas you seem to have "the answer", and not only is that hard to believe, it's hard to figure out what your problem with them is other than that they're "wrong". I suggest a more effective approach is to cite your sources and those others that agree with you; perhaps it will lead to better communication. Because, as much as you claim the REST community is providing no solutions, you're not really providing them either, if people are unable to communicate with you.

Regarding BEA: compared to other competitors, and even most system integrators, I think the thought leaders within BEA really did get it, at least for a while. When getting into deep discussions or consulting engagements, however, BEA's performance often was a regional affair; the U.S. West field region had a different style and approach than U.S. East, or my area, Canada.

Today, your analysis is that returning to the principles of the Web is somehow going to give us this magical programming model that everyone is looking for. What's the evidence? I am desperately seeking that evidence. Even though some people don't like my style, I am genuinely open to anything that works. I have no technology or product interest, I work for an IT organization. I do expect that people would be just as open as I am, but that's not always true.

Dude, look around. Look at how we're communicating. There's lots of evidence that the Web's architecture is something to embrace, not something to throw away or minimize. If you want more evidence of that, all I can say is "stay tuned".

Could there be a possibility that one more time you would be going in the wrong direction? Have you ever considered that your experience and training conditioned you to pretty much create the same programming model over and over? from EJB to Web Services to REST. Behind the bytes, isn't it the same thing?

Of course there's a possibility that this is the wrong direction. Point me to either a sound theory (with references) or working alternative software that should be a better direction. Though note, I'm pretty sure that the WS-* stack is not it.

Behind the bytes, from EJB to Web Services, they absolutely are similar things, at least in the session bean sense of EJB. With REST, it's pretty different. I do think it has some similarities to object-orientation (identity, reference vs. value, message-orientation, shared protocols as a basis of interchange) . I do think it can be abused to defeat the properties that it fosters. I don't think it's the be-all end-all architectural style, but basically, I like that it forces us to revisit the role of data vs. behaviour in a scalable system.

I would argue that data concerns (identity, reference, base semantics of access, provenance, the role of time & concurrency) should be the foundation of a scalable networked system-of-systems; only after we have that foundation can we address issues of behaviour, interaction and choreography. That's the main reason why I'm a REST proponent; if you want to convince me otherwise, you'll probably have to start with that assumption (e.g. should behavioural interactions be the basis? how would we ever agree on the semantics?)

Now, isn't it clear that the Web, even though it is the largest information system by far, has a very limited programming model unsuitable for building enterprise class information systems?

It's clear to me is that the Web isn't an architecture for building an localized information system. You need several styles needed for any of such things.

The Web's benefit is as a "system of systems" architecture; it's a way of dealing with issues of governance when you span systems and agencies. I don't think that it's the final approach -- there will be successors. But I think aspects of it, such as the URI, probably will endure for decades. And I think enterprises would benefit greatly if they adopted, at the very least, URIs more prevalently.

So please note, I don't think 'the answer to IT woes' is to rebuild your enterprise systems as "web applications" with RESTful resources, full stop. I think that's hopelessly naive. The point, again, is that there are many architectural styles, and the web's exhibits properties that I think would benefit most enterprises for what they've been usually been using WS-* for, in my experience. But it's far from the only architectural style that the enterprise uses or will use in the future.

Stu, the way I view it, when you say the "Web is enough" you are simply rejecting the all the vendors (small or large, from Redmond to Waldorf) as well as the standards and technologies they produce, rather than convincingly provide a solution to the problem.

I never said the Web is enough. I said that it's probably the way forward - a foundation for a next generation of sharing information at scale. I think any successors in the near future will have to start with the Web and drop or change the parts that don't work (for their domain) rather than inventing something from scratch. For example, turning the Web from client/server to event-based is an obvious one. Though we'd have to be careful not to lose the RESTful benefits; asynchronous messaging tends to lead to more complicated and less scalable systems when designed in an amateurish way.

As for rejecting all the vendors' standards and technologies -- considering the level of RESTful activity going on in Redmond and Armonk these days, I'd say they're (on some levels) doing a good enough job on their own of at least hedging their own efforts. Most standards these days are not driven by good technology, they're driven by power politics. There's only so many times you can compromise for political reasons before you just want to duck the whole system and try something new.

So, how am I providing a solution to the problem of making the Web more suitable for enterprise systems? Firstly, this isn't something one can solve in a blog. It requires working software and specifications. I don't think a Grand Unified Theory of Everything approach, the approach I worry we sometimes were moving towards in our "chats" about BPM and interactions and choreography in early 2008, is productive -- it just leads to frustration and communication breakdowns. All I can do is hope to catalyze the growth of knowledge in this area. To that end:

(a) I would like to dedicate time to frame discussions on some key problems; there's a long-awaited article I've been sitting on to discuss an architecture for business process & interactions on top of REST, which will be useful food for thought (it's a big topic, so I've had to revisit it several times, to the chagrin of my very patient editor, who's likely given up by now [grin]);

(b) There's a long dormant article or blog post on semantics of action that I'd like to get around to (about why REST or ARRESTED is a suitable foundation for enterprise systems-of-systems interactions) ;

(c) What takes most of my time up is that I'm now the technology leader at a startup in a growing industry; I'm going to do what I can to drive forward approaches that I think are productive & open. You can bet that the Web's architecture will be a part (but not anywhere near the only part) of that effort.

Cloud Cafe Podcast

| No Comments | No TrackBacks

John Willis interviewed me yesterday for this week's Cloud Cafe podcast. We mostly talk about Elastra and cloud computing in general. I do insert some REST stuff into it ;-) I hope you enjoy it.

I've long felt that if there's ever a band that will break progressive death metal to the masses, it will be Opeth. They're the closest band I've found to stirring me the way Metallica did back in the 1980's, and after nine (!) albums, they're getting even better.

Some food for thought:

Metallica - Master of Puppets
1986 - Billboard - Position: #29
1986 - UK Albums Chart - Position #41

Opeth - Watershed (praised by the NY Times, Pitchfork, Blender, AMG, etc, reaching 88 on Metacritic)
2008 - Billboard - Position: #23
2008 - UK Album Chart - Position: #34
2008 - Finland - #1
2008 - Sweden, Norway & Australia: #7

Not that I'm predicting anything. I'm just sayin' that Watershed is, I hope, a sign of things to come.

p.s. I'm really looking forward to Death Magnetic too, by what I've heard so far. It's been over 12 years since I've been able to say that about a Metallica record.

For those attending, I will be presenting at Jazoon in Zurich on Wednesday June 25th. The talk is influenced by my work the past year on data in SOA, how REST offers some useful guidance to the problem, and how some WS-* specs are attempting to address this area.

Feel free to email me or come up and say hello!

Against the Canadian DMCA

| No Comments | No TrackBacks

Read the comic book; write to your MP.

CMCC says... "As we feared, this bill represents an American-style approach to copyright. It's all locks and lawsuits," said Safwan Javed, the drummer for the band Wide Mouth Mason and a member of the CMCC. "Rather than building a made-in-Canada proposal to help musicians get paid, the government has chosen to import American-style legislation that says the solution to the music industry's problems is suing our fans. Suing fans won't make it 1992 again. It's a new world for the music business and this is an old approach."
"Who gains from this bill?" asked CMCC member Brendan Canning, co-founder of Broken Social Scene. "It's not musicians. Musicians don't need lawsuits, we don't need DRM protection. These aren't the things that help us or our careers. What we do need is a government that is willing to sit down with all the stakeholders and craft a balanced copyright policy for Canada that will not repeat the mistakes made in the United States."

Specifying a cloud computer

| 1 Comment | No TrackBacks

A response David Young's call to action on cloud computing. This entry is based on a Cloud Computing list.

I think there are a number of assumptions here that are worth
discussing, and I think it's great that David brought them up.

Firstly, as James posted on the list, there's a wide variety of "clouds" out there, from black boxes that support web applications like AppEngine, to CohesiveFT's stack deployers on EC2, to Elastra's scalable database on demand. Not all of them expose instances. Certainly it would help to have a standard for the providers that DO expose instances, I just don't think it covers the market. Wasn't this the point of the Open Virtual Machine Format (OVF) from VMWare, Xen, etc? [ Of course, it's yet not ratified by the DMTF to my knowledge. Oh, and also, DMTF... not exactly a well known standards body, yet. ]

Secondly, on to the specifics:

"Cloud computers must operate on some sort of virtualization
technology for many of the following features to even be feasible."

Really? Why can't it just work on pre-installed capacity? Blade arrays have been doing this for many years. Certainly virtualization is important long term, but, in the open
systems world at least, it's still _very_ new in the data centre. What happens if enterprises just aren't ready to adopt VMWare or Xen for a variety of technical or political reasons? To paraphrase what one person here asked, "will you trust your hypervisor more than your OS?"

"Application Layer Interoperability"

It'll be interesting to see if user-packaging standards start to form here, like WAR files. Or wouldn't a tarball be enough for many of these frameworks?

"State Layer Interoperability"

There are billions of dollars thrown at interoperability, semantics, and integration annually by both vendors and customers, flowing like a river into a tarpit of epic proportions. A large neon sign hangs over this tarpit that says "Abandon All Hope, Ye Who Enter Here.", obfuscated by an even larger fireworks display that spells "A-G-I-L-I-T-Y".

There is some hope, of course - cue talk of the Semantic Web. It's
still too soon to tell what pieces remain to make it work, but a standard XMPP-based information bus is probably low on the list of priorities -- it's too specific a solution. (The technical challenges in financial markets don't apply to every industry). I'd say data model interoperability probably is a more important nut to crack.

"Application Services"

Yup, I agree here. But It'll be hard-fought street fighting to get the standards in place over the next few years.

Automatic Scale (deploy and forget about it)

Funny, I keep looking for the "fast=true" parameter in my infrastructure, and I still haven't found it.

Joking aside, I do agree that 'adding & removing resources on demand' should be a feature of how one provisions cloud infrastructure. (Not coincidentally, that's one of Elastra's features in our preview release). Though I'll go back to our conversation on performance management a couple of weeks ago: many times, the best way to scale
is to fix the application.

Barring that, the cloud would have to detect and rewrite the developer's shitty code for them. This isn't completely out of the question -- recent Oracle releases can rewrite a developers' SQL to perform better, and a DBA can choose to "swizzle" the statement at
runtime without requiring a code change. They added this feature so shops could workaround the crappy code in many packaged applications that somehow never seem to be fixed.

Hardware Load Balancing

This seems very much aimed at web application hosting. While clearly important, that would be pretty limiting, especially if we're trying to shift the enterprise onto the cloud. I'd go further and suggest that the network capacity and (virtual) topology itself needs to be a cloud service.

Storage as a Service

WebDAV?! Why not AtomPub?

On another note, has anyone seriously considered iSCSI? Certainly we can't think of storage just as web resources -- there also need to be mount points as a service, like Amazon EBS. Or is this just a pipe dream?

About this Archive

This page is an archive of entries from June 2008 listed from newest to oldest.

May 2008 is the previous archive.

July 2008 is the next archive.

Find recent content on the main index or look in the archives to find all content.

About Me
(C) 2003-2011 Stuart Charlton

Blogroll on Bloglines

Disclaimer: All opinions expressed in this blog are my own, and are not necessarily shared by my employer or any other organization I am affiliated with.