Convenience, Correctness, and Network Interactions

| 3 Comments | 2 TrackBacks

A response to Steve Jones' post criticizing what I thought was a great article by Steve Vinoski. In this response I will refer to Steve Jones as "Steve" and Steve Vinoski as "Vinoski".

Steve, I normally enjoy your business services architecture discussions, but I think you are flat wrong in this post. You seem to be revising a whole lot of historical context in this entry, and I think you've seriously misread many aspects of the article. Furthermore, I think you've contradicted yourself several times in this rant.

"No decent architect built an RPC system which assumed local = remote"

But, that was *the whole point* about RPC systems to start with, and that tradition was carried on WELL into the late 1990's.

" What a load of crap. Seriously this is an unmitigated pile of tripe in what it means to write distributed systems. It makes two basic errors.... That the architecture and design of a system is focused on a programming language"

And that was entirely his point! The point of the article was that one should never view this. YET -- vast majority of RPC systems, from RMI, to CORBA, have --all-- been focused on mapping the design of a system into programming language constructs that completely gloss over most of the issues inherent in distribution, and forces the semantics of the network interchange to conform to what's comfortable in the programming language!

Look back at Don Box's "XML Manifesto" from 1998, which was basically claiming that the reason for XML was to wrest control of distributed systems interoperability from the programming geeks and use a document geek's standard, because that's the only way we can stop this "my language is better than your language" crap.

At worst you could accuse Vinoski of repeating an insight that's 10 to 15 years old. But, in my opinion, he is going down this road because

  1. we still see RPC systems being built and released (Etch) for some reason
  2. most SOAP toolkits are still RPC-oriented (hello JAX-WS) even though they could be messaging-oriented. I recall when JAXM was rolled back -- no one wanted it!
  3. Most WSDL interfaces are still very RPC-oriented
  4. a common mistake in RESTful design is trying to overlay RPC on it because it's easier for a programmer to work with

The answer to interacting with the network should not be "bend the network to make my life easier", it should be "make my life as easy as the reality of the network will let me".

"Architectures are normally programming language independent and anyone who starts a distributed system by writing C++ is a muppet who should be fired."

Yikes. My Top Gear cockometer has just been broken (you sure you don't drive an M3? ;-)

Unless I'm misreading, "Programming languages don't matter, but if you're using C++, you're a muppet." I understand in an enterprise IT environment using C++ is somewhat anathema (for some good reasons), but not everyone is in an enterprise IT environment.

"Errr apart from muppets who thinks that a distributed RPC call can be treated the same as a remote call? Every single RPC approach has a litany of exceptions that you have to catch because you are doing a remote call, that is baked into the frameworks."

Those muppets you cite seem to be every vendor that's ever built a toolkit around RMI, CORBA, or SOAP.

And no, not every single RPC approach has a litany of exceptions that you have to catch. That was Waldo's "innovation" in RMI -- forcing RemoteException to be checked. Because, as even a cursory examination of history will show you, most distributed developers never checked it. It was brittle because failure was rarely taken into account.

This is getting better, but I've seen plenty lot of cases in the EJB, CORBA, and SOAP world where failure was rarely taken into account other than basic retry logic. But I'd even see retries in non-idempotent, non-transactional invocations (oops, sorry we billed you twice!). "Oh, but this never would happen in my system", you may scoff. Great, I'm glad for you. But that's not a scalable solution to the problem.

Now, here is one large steaming contradiction:

"The writer's position is that everyone should know about the network, this is a false position as it requires everyone to be of the same sophistication as the, very talented, writer."


"The reality is that all decent architects have always known that the two are different and have worked accordingly... This really is the worst kind of argument. Setting up a completely specious argument as the basis for this area really does undermine the whole credibility of the writer. If the writer is saying that when they built RPC systems in the past, which they apparently have, that they treated remote and local as the same then the writer is a muppet."

Which is it? Any decent architect "knows this stuff"-- they need to know the network. Yet, they can't know the network, because they're not that sophisticated?

Why is it that all the toolkits provide dumb interfaces to the network *without really handling any of these issues*?

What about the end-to-end argument, wherein you can't really provide for true reliability and security without the application-layer understanding the implications (i.e. it's not something you can "abstract" away and still have a complete solution)? This argument belies much of the Internet's architecture and has served us well. Shall we dispose it because it doesn't fit into our worldview of "ubermench and muppets"?

The point is that a skilled architect can't just provide an "uber abstraction" to make these problems go away in a transparent fashion. I know in your world of muppets and puppetmasters this is desirable. But the whole point of Vinoski's paper, and Waldo's, and Brewer's conjecture, and even Vogel's writings on the topic, is "Abstraction > Transparency". You can, at best, abstract the requisite variety of a network down to a protocol or API, but you still have to understand the domain that the abstraction is describing, a domain that is not necessarily aligned your programming paradigm, eg. objects, arrays, triples, relations, etc.

You of all people should understand this argument as it's the whole reason why so much technology in SOA is problematic and the source of "business SOA vs. technical SOA". Many of the toolkits developed are written by programmers, for programmers, and not with the reality of the network in mind.

"The suggestion in the text around is that its impossible to build a large scale system using RPC as it lacks these abilities. .... this must come as a big surprise the to the companies doing ESBs, Web Service monitoring or Web Service gateways which in fact allow all of these elements in just such the RPC environment that the writer claims isn't possible. Nothing in RPC could ever indicate whether a result is cacheable for instance."

The point is not that this is impossible. It's that it is very difficult to achieve at scale due to the lack of standardized semantics! Sure, anyone can solve those problems in their little pocket of the world, but how are you going to get multiple organizations in multiple geographies under different governance structures to agree on these things?

As an example, caching at an XML infoset level is hard to do, slow, and brittle. The ESB would need to have coupled knowledge of all of the IDLs or WSDLs that pass through it, and each operation's safeness or idempotency, to be able to understand what is and is not cacheable. We would need a standard representation for a cache key -- would that be a WS-Addressing reference parameter? An XPath into my SOAP body? What? Then, if the body were cacheable, we'd need to deal with XML canonicalization issues. None of this is standardized. So the solution usually is to give up with intermediaries and just do caching at the implementation layer.

The point is that introducing an intermediary in an RPC network is an exercise in neurosurgery. Introducing an interemediary on the Web happens all the time. Does it magically whisk away enterprise-specific needs, like data transformation? Not necessarily. But it has been shown to work well for global-scale general purpose features, like routing (see CDNs), caching, streaming, and redirection.

REST of course address all of these concerns (and more) by having... a field that says you can cache the result something that is trivial to add to an RPC environment but is lauded as hugely different. REST is great because it doesn't fit with normal programming language abstractions.

The point is not that adding a caching header is somehow new or different. The point is that, regardless of the nature of the data passing through the interface, *all of it* is accessible in a uniform semantics, and thus *all of it* is inherently cacheable. Not a bunch of local optima solutions sprouting up that need to be integrated after the fact.

So REST can be made to fit in a "normal" programming language (unless someone can show me an example that couldn't fit) so again this isn't a real argument its just prejudice and an example of the lack of sophistication in current REST frameworks and tools.

I think you've missed the point. REST is an architectural style. Can you make "Client/Server" fit in a normal programming language? Sure, but that doesn't tell me much. I usually need a lot more facilities than a programming language to make it work (file I/O, network I/O, etc.)


a) The purpose of most RPC toolkits is to make the network look, as much as possible, as a regular programming construct (Java object, C call, etc.) This holds for RMI, CORBA, JAX-RPC, etc. It was the whole point of object/relational mappers (which wraps SQL RPCs), and EJB entity and session beans. ON the bright side, RMI did make some things explicit that weren't in the past (e.g Serializable objects to distinguish value vs. reference semantics).

b) The purpose of a messaging toolkit (e.g. JMS or an AMQP kit like RabbitMQ) is to make the network look like an abstraction that covers much of the requisite variety in the network though still enables you build your own custom semantics of interaction. Notice that the point isn't entirely to make everything "convenient for a programmer", as anyone using JMS will attest. It has a different goal in mind than appeasing a programmer who just wants to make a remote call and be done with it.

c) The purpose of a RESTful toolkit (e.g. JAX-RS or HTTPClient or Restlet) is to make the network look like a distributed hypermedia system that covers much of the requisite variety in the network and specifies uniform semantics for all interactions as part of a hypermedia application. It too has a different goal than a programmer that just wants to send a message and be done with it.

REST is trying to raise the bar of abstraction above message passing into one of a hypermedia application, to assist interoperability and scalability at the expense of efficiency in some cases. Is it always appropriate? Heck, no. But it is a legitimate attempt of "raising the bar of abstraction" beyond just message interactions.

The problem with current approaches to RESTful client access is that the facilities are somewhat new and we don't have a lot of experience of what they should look like. There's innovation going on. One example is a client-side representation cache, like what a Web browser has. Good feature, it's an identified component in the architecture, but I don't see it in any of the REST toolkits yet.

I would also highlight that if you think all of this uniformity is somehow useless, then why are the vendors submitting several standards to the W3C that basically define standard semantics for web service interactions? And adopt some of REST's constraints? It's not because these are RESTafarians. It's because in the IT management domain, they realized they couldn't make things work if every possible IT resource had their own WSDL interface!

What frustrates me about interchanges like these is the utter disrespect and hubris thrown around. See, the way I read it, besides the REST stuff (which I think you inherently can't or won't wrap your head around), your position isn't actually that far from Vinoski's position on RPC.

Vinoski is one of the world's most experienced distributed systems architects and developers, but based on your response, you'd think the man was a recent college graduate being upbraded for inexperience. Anyone that's used CORBA in anger has likely used IONA's products in the 1990's and will know what it was capable of. Yet you immediately leap to showing how much better and more experienced you are instead of, oh, I don't know, questioning your bias? Asking Vinoski more precisely what he meant? Non sequiturs and ad hominem arguments work well in enterprise IT power politics, but I don't give them much merit when we're actually trying to solve real problems instead of spouting off how smart we are compared to the 'muppets'.

Update: Steve Vinoski adds his own response to this.

2 TrackBacks

TrackBack URL:

I can’t shake the feeling that if Steve Vinoski, Steve Jones and Stuart Charlton had a drink together they’d actually agree on pretty much any distributed computing question that is worded in specific and unambiguous terms. If you are not s... Read More

Rabbitmq Bookmarks from Post Saver - Website voting and saving system on September 11, 2008 11:17 PM

Bookmarked your page with keywords rabbitmq! Read More


Well defended Stu.

I think Steve should dedicate a good bit of a future article to defining "distributed scale".

From a scale perspective - there is a certain degree to which you can get away with ignoring the shortcomings of RPC-style approaches and still call the operating environment "distributed". A 4-node system over multiple locations is distributed, so is a 1000-node system. However the challenges one encounters @ 1000, 10000 or 100000 nodes are different and exponentially more difficult to identify if proper separation of concern and tenacity are not addressed.

As you rightly stated, people have toughed it out and overcome the shortcomings to build large-scale systems of a distributed nature.

However, it is hard to ignore successful internet-scale paradigms that offer uniform interaction constructs to promote interoperability through simplicity. That's what REST-style techniques offer and that's why it has traction.

The number of distributed applications likely to require "awareness" of aspects outside the programming language & run-time are going to increase drastically. The data deluge is upon enterprises and people are going to want to extricate competitive advantages by implementing internet-scale paradigms inside their walls.

Will the enterprise of the future place a higher premium on resiliency & efficient scalability over consistency? You betcha. Why? Because if they don't - they will sacrifice agility and that could be fatal.

And then the whole thing about async and message passing being a bitch etc. That has NOTHING to do with IQ. Some people are just able to think distributed with a sense of inevitability and some people are not.

I know plenty of bright people - some get the paradigm and some don't. Vinoski, Ippolito, Amstrong, Sadan and Stu get it. So Erlang comes easily to them. And to some it never well. Intelligence has jack to do with it.

But, that was *the whole point* about RPC systems to start with, and that tradition was carried on WELL into the late 1990's.

Where was it? Seriously I just don't know the people who worked like that. I was working on distributed systems throughout the 90s and in NONE of them did we consider local = remote. Even EJBs made a big deal about the remote/local piece which shows how mainstream the difference has become. I've clearly worked with very different people as the coarse-grain interface and "worry about the network" bit base battered into my head at University and through out my formative years.

It is the point that to the average developer that they looked the same. This meant that the architects should have constructed a system that enabled a blending of the talents. This is why JAXM failed, it presented everything to the developers and 99% of developers can't handle that. This is the sad reality of IT. Don Box's article is interesting but again it doesn't take into account Meyfroidt's Law.

This is why there isn't a contradiction in the article. You, Steve, Dan Creswell, Don Box, Jim Waldo, hell even myself, have always taken the network into account and designed systems accordingly. This however does not represent the talent of the majority of IT.

This is my issue. At the moment people are chasing a perceived technical perfection over a practical set of improvements. RPC works sometimes, REST works sometimes, Messaging works sometimes, flying monkeys only work on certain edge cases. The real focus should be how to think about the overall business problem, the continual belief that the next technology will make IT more successful is a big part of the problem.

I did indeed use the IONA ORB in the 90s and pain in the arse though it could be it worked and when you tuned it absolutely flew. This is far from the fundamentally flawed argument that Steve is putting forwards today.

"And that was entirely his point! The point of the article was that one should never view this. YET -- vast majority of RPC systems, from RMI, to CORBA, have --all-- been focused on mapping the design of a system into programming language constructs that completely gloss over most of the issues inherent in distribution"

Can you provide some example issues that RMI and CORBA gloss over please?

I guess I'm asking are we talking latency, failure, something else?

Leave a comment

About this Entry

This page contains a single entry by Stu published on July 7, 2008 3:45 PM.

Elastra is hiring was the previous entry in this blog.

Can technology progress? is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

About Me
(C) 2003-2011 Stuart Charlton

Blogroll on Bloglines

Disclaimer: All opinions expressed in this blog are my own, and are not necessarily shared by my employer or any other organization I am affiliated with.