July 2008 Archives

Hug an SA today

| No Comments | No TrackBacks

If you know a system administrator, today's the day to thank them for keeping stuff up ;-)

The freedom to use crap

| 7 Comments | No TrackBacks

The FSF wants you to avoid the iPhone 3G in favour of OpenMoko Neo FreeRunner.

Some of the FSF's points have merit, but some are outright deceptive - for example that the iPhone exposes your whereabouts without your knowledge. It doesn't; in fact it always pops up with a confirmation before accessing the "location services", whether GPS or cell-tower triangulation. And you can turn it off through a global setting.

The biggest joke is the alternative. Freedom to modify and tinker and extend means little when the basis is barely useable. People will often tradeoff one freedom for another freedom. A more usable device arguably is just another form of freedom -- from hassle, annoyance, and wasted time.

Open source makes economic sense in many contexts, but I think they're at least 10 years away from even denting the mainstream in this market. Which is why, BTW, I think the FSF is targeting the iPhone. The iPhone represents a refutation to the idea that open, free, collaboration will necessarily lead to better products, and that proprietary software makers cannot compete.

The OpenMoko counter-argument is "give it time, in the long run, it will win". And look, in a way, I hope so. Using the iPhone is a great case of following Keynes' adage, in the long run, we are all dead., where we optimize for short term gratification at the expense of our future. By using the iPhone, we're supporting and contributing something that doesn't build something open for our collective future, but instead leases our future over to Apple. On the other hand, the iPhone does represent something that is important to our future -- the triumph of entrepreneurship over bureaucrats and technocrats. More on that in a moment.

I'll note that the economic angle isn't usually the FSF's preferred line of argument. It's more the OSI's tactic, though the FSF has certainly referenced it. On the other hand, the FSF argues for free software from an ethical stance. Without getting into the muck, my opinion basically is that the sort of freedom the FSF advocates is not, IMO, political freedom, and thus I don't consider it sacred. Whether one chooses to be imprisoned by license is an economic tradeoff, not capitulation to evil. I believe the author should have the option to retain certain limited rights, for a limited time, over its users. I believe in entrepreneurship and the Schumpeterian model of the economy. Profits should go to the innovators for a limited time, as profit is the source of tomorrow's jobs and developments.

The iPhone represents a triumph of entrepreneurship -- the Cathedral over the Bazaar. The user experience loosens the telecom bureaucrat's insistence on device control, or the technocrat's desire for infinite options. The rules of bazaar development are flipped: Scratch the user's itch, not the developer's. Release "when it's ready", not early. Users are not co-developers, and developers aren't even co-developers, they're cordoned off into their controlled area for the sake of the user experience.

But wait, several complications to this picture:

- Open source is not incompatible with entrepreneurship. The Mozilla Foundation demonstrates this regularly with Firefox. OTOH, the market dynamics of browsers imply that there's not a lot of money to be made through direct distribution. Moz is funded largely by complementary product placement: redirecting searches to Google.

- The iPhone OS X already uses plenty of open source: its web browser is open source, as is much of its operating system layer (including both BSD and GPLv2 code). They're both complex, mature areas of computing, where it's (again) hard to declare some kind of "secret sauce" that needs protection. So it is a great place for open source collaboration.

- Apple is exceptional in its ability to successfully deliver great software and build a thriving community with a cathedral model. Most aren't. Though, looking through Freshmeat, I'm not sure the ability to build solid software with a thriving community is intrinsic to either model. It's just hard to do.

One perspective is that the architecture of a project likely has more to do with its success, quoting Roy Fielding:


In spite of the hype and hysteria surrounding open source software development, there is very little that can be said of open source in general. Open source projects range in scope from the miniscule, such as the thousands of non-maintained code dumps left behind at the end of class projects, dissertations, and failed commercial ventures, to the truly international, with thousands of developers collaborating, directly or indirectly, on a common platform. One characteristic that is shared by the largest and most successful open source projects, however, is a software architecture designed to promote anarchic collaboration through extensions while at the same time preserving centralized control over the interfaces.

Which may bode well for Apple's approach to applications.

- Apple also has found a "secret sauce" that free software rarely measures up to: usability and aesthetics. "Taste" is a hard thing to replicate, especially when it's delivered as a tight coupling between software and hardware.

A followup on the Convenience vs. Correctness debate. I made a longish comment on SJ's blog about what the nut of our disagreement seems to be: how to improve IT! Please forgive typos and grammatical errors, I wrote it on the iPhone 3G...

One approach assumes that technology will always be a crapfest, thus we should focus on the business problems, create a roadmap, and map backwards on to whatever technology turd pile that stinks least for our business / skillset / culture.

The other approach assumes that business can and will change due to technology progress, thus technology can be made to smell better. We should focus on improving our current tech turds and crapfests to better fit big picture challenges (like those of businesses) so that they stink less, and thus businesses can leverage them to change their competitiveness.

I don't think these approaches target the same audiences, one is clearly the mindset of an IT consultant, the other seems to come from activists at IT vendors.

I also don't think either approach is necessarily wrong, except that both approaches carry biases make communication frustrating. ;-)

Can technology progress?

| 2 Comments | 1 TrackBack

In our last episode, a minor bun fight ensued about the way to think about convenience vs. correctness in distributed interactions.

I made a number of bold claims, such as "the vast majority of RPC systems, from RMI, to CORBA, have --all-- been focused on mapping the design of a system into programming language constructs that completely gloss over most of the issues inherent in distribution".

In the comments section, Dan Creswell asked, "Can you provide some example issues that RMI and CORBA gloss over please?" I guess I'm asking are we talking latency, failure, something else?

Similarly, Steve Jones asked, "Seriously I just don't know the people who worked like that. I was working on distributed systems throughout the 90s and in NONE of them did we consider local = remote."

To which I say, "yes latency, yes failure, but mostly semantics.".

To use latency as an example, I have worked on at least two or three financial trading systems that involved 50 to 500 remote procedure calls to perform a transaction, wreaking havoc on both performance and recoverability. I know that anyone who works in the wireless telecom industry is likely familiar with a certain telecom billing systems' Java wrapper API, which was designed completely around performing approximately 2-3 network round trips for every field you needed to set in a transaction (which was a stateful session bean that talked to entity beans), leading to between 30 and 150 RPC's for every transaction. People wonder why so many business transactions failed, and why so much infrastructure was required to support a small volume of business transactions.

The problem is that, as I believe Steve Jones has pointed out before, there are too many projects without any real experts available on them, so the developers building the systems (which involve billions of dollars of transactions) aren't aware of any innate issues of distribution, so build the network interactions with the approach they know -- local object-oriented programming. This is also why vendors continue to build tooling that doesn't deviate from local object-oriented programming, as it's risky to try to shift the mindset of developers to something different.

The semantics area is one that's been discussed many times with regards to REST's uniform interface, so I won't repeat it here, though we could dig into it more if some would like to.

Then, Steve Jones noted the following:

This is my issue. At the moment people are chasing a perceived technical perfection over a practical set of improvements. RPC works sometimes, REST works sometimes, Messaging works sometimes, flying monkeys only work on certain edge cases. The real focus should be how to think about the overall business problem, the continual belief that the next technology will make IT more successful is a big part of the problem.

I have run into this opinion before from Steve several times in the past. It almost is, to me, a form of technology fatalism. Many IT executives that I know and respect share this opinion. So, I respectfully disagree, and will try to explain why.

I agree that technology alone will not make IT successful.

I do not deny that there are variable levels of skill and productivity among IT workers.

I reject the notion that ensuring that all IT workers understand the business better, though highly desirable, is a realistic solution to the problem of IT failure.

I also reject the notion that we cannot raise the bar of productivity across the board through better technology, for a variety of reasons that I will explain next.

Firstly, there will always be significant numbers of IT workers that do not, and cannot (!) understand business problems to a greater degree than technological problems. This is for a variety of factors (educational, for one; engineers aren't usually schooled in those liberal arts that provides insights into humanity and society). This problem goes both ways: there will always be significant numbers of business people that are clueless with regards to technology, for better or worse (as one's has only so much time in the day and thus must choose to be ignorant of some topics). It may be possible to fix this problem, but it is a problem for entire generations of workers, not something fixable with anything less than a 30 year time window.

So yes, if you don't understand the business, you're dead. But, there's a lingering problem: if your business problem relies on technology, and you're either not aware of the appropriate technology to solve your problem, or you screw the technology up, you're likely still dead.

In short, contrary to the popular saying, it's not just about the business. But it's also not just about the technology. It's about both. They reflexively impact and depend upon one another.

History seems to show this relationship. The development of the steam engine, the telephone, the railroad, the automobile, the television, the computer, the web, etc. have all shaped society in ways it did not expect. Few of these were driven by 'understanding the business better'. This is not to say that all IT projects are on the level of these innovations. It is, however, an recognition that major innovations come with major changes, some of which are hard to interpret for those stuck in the prior model. (e.g. Television impacted U.S. presidential politics as early as 1960, with the Nixon vs. JFK debate).

The moral of this story is that the technology-influenced mindset and architectures we take to our business initiatives will have a direct impact on how the business performs. There's an old saying in software product development: "TEAM = PRODUCT", also known as Conway's Law. Technology is often a necessary, even if not sufficient, enabler for those other, more important business changes. And much of that enablement is emergent behaviour, i.e. not directly traceable to the functional implementation of a business requirement. Rigid, brittle technology, mixed with a dose of fear, can often provide enough inertia to kill the best laid hopes of broader change.

So, tying back to our discussion of network interactions and RPC's. The problems I have with suggestions such as "an expert can make RPC (or EJB, or WS-*, or.. *) work well in practice" are:
a) they're generally correct in their outlook, for a particular audience;
b) they sometimes use the Internet's architecture and principles as an example of the right way to do things;
c) but they expect every IT department to build their own version of the Internet's architecture with the toolboxes that vendors give them.

Yet the vendors themselves are stuck in the approaches and techniques of yesteryear - old wine in new bottles, while they have nearly always worked to actively dissuade adoption of alternatives to their approach, or even destroy those who support new alternatives. Technology debates and standards bodies become proxy wars for market competition, and progress in the technology debates and standards is often more about fixing the political situation, not the actual technology itself. How is an IT department supposed to know any better if industry leaders aren't raising the bar for them?

The problem with approaches like "yet another RPC stack" are that they are not catalyzing changes in technology architecture much beyond where they were in 1984. Similarly, the WS-* world has not been catalyzing changes in technology architecture much beyond where it was in 1998. It's still, basically, component software, just now focused on protocols of runtimes. Much of what we have is improved tooling for Visual COBOL. And there are even signs that the vendors are moving back to emphasizing runtimes.

In summary, I think improving technology is necessary along with (not instead of) an increased understanding of business problems, because I think they help each other out. IT's business value basically comes down to providing tools and catalytic ideas to change the transaction cost structure of a firm. You can't do it without the business, but you often won't have the germ of the thought without innovative technology and associated architectures.

A response to Steve Jones' post criticizing what I thought was a great article by Steve Vinoski. In this response I will refer to Steve Jones as "Steve" and Steve Vinoski as "Vinoski".

Steve, I normally enjoy your business services architecture discussions, but I think you are flat wrong in this post. You seem to be revising a whole lot of historical context in this entry, and I think you've seriously misread many aspects of the article. Furthermore, I think you've contradicted yourself several times in this rant.

"No decent architect built an RPC system which assumed local = remote"

But, that was *the whole point* about RPC systems to start with, and that tradition was carried on WELL into the late 1990's.

" What a load of crap. Seriously this is an unmitigated pile of tripe in what it means to write distributed systems. It makes two basic errors.... That the architecture and design of a system is focused on a programming language"

And that was entirely his point! The point of the article was that one should never view this. YET -- vast majority of RPC systems, from RMI, to CORBA, have --all-- been focused on mapping the design of a system into programming language constructs that completely gloss over most of the issues inherent in distribution, and forces the semantics of the network interchange to conform to what's comfortable in the programming language!

Look back at Don Box's "XML Manifesto" from 1998, which was basically claiming that the reason for XML was to wrest control of distributed systems interoperability from the programming geeks and use a document geek's standard, because that's the only way we can stop this "my language is better than your language" crap.

At worst you could accuse Vinoski of repeating an insight that's 10 to 15 years old. But, in my opinion, he is going down this road because


  1. we still see RPC systems being built and released (Etch) for some reason
  2. most SOAP toolkits are still RPC-oriented (hello JAX-WS) even though they could be messaging-oriented. I recall when JAXM was rolled back -- no one wanted it!
  3. Most WSDL interfaces are still very RPC-oriented
  4. a common mistake in RESTful design is trying to overlay RPC on it because it's easier for a programmer to work with

The answer to interacting with the network should not be "bend the network to make my life easier", it should be "make my life as easy as the reality of the network will let me".

"Architectures are normally programming language independent and anyone who starts a distributed system by writing C++ is a muppet who should be fired."

Yikes. My Top Gear cockometer has just been broken (you sure you don't drive an M3? ;-)

Unless I'm misreading, "Programming languages don't matter, but if you're using C++, you're a muppet." I understand in an enterprise IT environment using C++ is somewhat anathema (for some good reasons), but not everyone is in an enterprise IT environment.

"Errr apart from muppets who thinks that a distributed RPC call can be treated the same as a remote call? Every single RPC approach has a litany of exceptions that you have to catch because you are doing a remote call, that is baked into the frameworks."

Those muppets you cite seem to be every vendor that's ever built a toolkit around RMI, CORBA, or SOAP.

And no, not every single RPC approach has a litany of exceptions that you have to catch. That was Waldo's "innovation" in RMI -- forcing RemoteException to be checked. Because, as even a cursory examination of history will show you, most distributed developers never checked it. It was brittle because failure was rarely taken into account.

This is getting better, but I've seen plenty lot of cases in the EJB, CORBA, and SOAP world where failure was rarely taken into account other than basic retry logic. But I'd even see retries in non-idempotent, non-transactional invocations (oops, sorry we billed you twice!). "Oh, but this never would happen in my system", you may scoff. Great, I'm glad for you. But that's not a scalable solution to the problem.

Now, here is one large steaming contradiction:

"The writer's position is that everyone should know about the network, this is a false position as it requires everyone to be of the same sophistication as the, very talented, writer."

vs.

"The reality is that all decent architects have always known that the two are different and have worked accordingly... This really is the worst kind of argument. Setting up a completely specious argument as the basis for this area really does undermine the whole credibility of the writer. If the writer is saying that when they built RPC systems in the past, which they apparently have, that they treated remote and local as the same then the writer is a muppet."

Which is it? Any decent architect "knows this stuff"-- they need to know the network. Yet, they can't know the network, because they're not that sophisticated?

Why is it that all the toolkits provide dumb interfaces to the network *without really handling any of these issues*?

What about the end-to-end argument, wherein you can't really provide for true reliability and security without the application-layer understanding the implications (i.e. it's not something you can "abstract" away and still have a complete solution)? This argument belies much of the Internet's architecture and has served us well. Shall we dispose it because it doesn't fit into our worldview of "ubermench and muppets"?

The point is that a skilled architect can't just provide an "uber abstraction" to make these problems go away in a transparent fashion. I know in your world of muppets and puppetmasters this is desirable. But the whole point of Vinoski's paper, and Waldo's, and Brewer's conjecture, and even Vogel's writings on the topic, is "Abstraction > Transparency". You can, at best, abstract the requisite variety of a network down to a protocol or API, but you still have to understand the domain that the abstraction is describing, a domain that is not necessarily aligned your programming paradigm, eg. objects, arrays, triples, relations, etc.

You of all people should understand this argument as it's the whole reason why so much technology in SOA is problematic and the source of "business SOA vs. technical SOA". Many of the toolkits developed are written by programmers, for programmers, and not with the reality of the network in mind.

"The suggestion in the text around is that its impossible to build a large scale system using RPC as it lacks these abilities. .... this must come as a big surprise the to the companies doing ESBs, Web Service monitoring or Web Service gateways which in fact allow all of these elements in just such the RPC environment that the writer claims isn't possible. Nothing in RPC could ever indicate whether a result is cacheable for instance."

The point is not that this is impossible. It's that it is very difficult to achieve at scale due to the lack of standardized semantics! Sure, anyone can solve those problems in their little pocket of the world, but how are you going to get multiple organizations in multiple geographies under different governance structures to agree on these things?

As an example, caching at an XML infoset level is hard to do, slow, and brittle. The ESB would need to have coupled knowledge of all of the IDLs or WSDLs that pass through it, and each operation's safeness or idempotency, to be able to understand what is and is not cacheable. We would need a standard representation for a cache key -- would that be a WS-Addressing reference parameter? An XPath into my SOAP body? What? Then, if the body were cacheable, we'd need to deal with XML canonicalization issues. None of this is standardized. So the solution usually is to give up with intermediaries and just do caching at the implementation layer.

The point is that introducing an intermediary in an RPC network is an exercise in neurosurgery. Introducing an interemediary on the Web happens all the time. Does it magically whisk away enterprise-specific needs, like data transformation? Not necessarily. But it has been shown to work well for global-scale general purpose features, like routing (see CDNs), caching, streaming, and redirection.

REST of course address all of these concerns (and more) by having... a field that says you can cache the result something that is trivial to add to an RPC environment but is lauded as hugely different. REST is great because it doesn't fit with normal programming language abstractions.

The point is not that adding a caching header is somehow new or different. The point is that, regardless of the nature of the data passing through the interface, *all of it* is accessible in a uniform semantics, and thus *all of it* is inherently cacheable. Not a bunch of local optima solutions sprouting up that need to be integrated after the fact.



So REST can be made to fit in a "normal" programming language (unless someone can show me an example that couldn't fit) so again this isn't a real argument its just prejudice and an example of the lack of sophistication in current REST frameworks and tools.

I think you've missed the point. REST is an architectural style. Can you make "Client/Server" fit in a normal programming language? Sure, but that doesn't tell me much. I usually need a lot more facilities than a programming language to make it work (file I/O, network I/O, etc.)

Contrast:

a) The purpose of most RPC toolkits is to make the network look, as much as possible, as a regular programming construct (Java object, C call, etc.) This holds for RMI, CORBA, JAX-RPC, etc. It was the whole point of object/relational mappers (which wraps SQL RPCs), and EJB entity and session beans. ON the bright side, RMI did make some things explicit that weren't in the past (e.g Serializable objects to distinguish value vs. reference semantics).

b) The purpose of a messaging toolkit (e.g. JMS or an AMQP kit like RabbitMQ) is to make the network look like an abstraction that covers much of the requisite variety in the network though still enables you build your own custom semantics of interaction. Notice that the point isn't entirely to make everything "convenient for a programmer", as anyone using JMS will attest. It has a different goal in mind than appeasing a programmer who just wants to make a remote call and be done with it.

c) The purpose of a RESTful toolkit (e.g. JAX-RS or HTTPClient or Restlet) is to make the network look like a distributed hypermedia system that covers much of the requisite variety in the network and specifies uniform semantics for all interactions as part of a hypermedia application. It too has a different goal than a programmer that just wants to send a message and be done with it.

REST is trying to raise the bar of abstraction above message passing into one of a hypermedia application, to assist interoperability and scalability at the expense of efficiency in some cases. Is it always appropriate? Heck, no. But it is a legitimate attempt of "raising the bar of abstraction" beyond just message interactions.

The problem with current approaches to RESTful client access is that the facilities are somewhat new and we don't have a lot of experience of what they should look like. There's innovation going on. One example is a client-side representation cache, like what a Web browser has. Good feature, it's an identified component in the architecture, but I don't see it in any of the REST toolkits yet.

I would also highlight that if you think all of this uniformity is somehow useless, then why are the vendors submitting several standards to the W3C that basically define standard semantics for web service interactions? And adopt some of REST's constraints? It's not because these are RESTafarians. It's because in the IT management domain, they realized they couldn't make things work if every possible IT resource had their own WSDL interface!

What frustrates me about interchanges like these is the utter disrespect and hubris thrown around. See, the way I read it, besides the REST stuff (which I think you inherently can't or won't wrap your head around), your position isn't actually that far from Vinoski's position on RPC.

Vinoski is one of the world's most experienced distributed systems architects and developers, but based on your response, you'd think the man was a recent college graduate being upbraded for inexperience. Anyone that's used CORBA in anger has likely used IONA's products in the 1990's and will know what it was capable of. Yet you immediately leap to showing how much better and more experienced you are instead of, oh, I don't know, questioning your bias? Asking Vinoski more precisely what he meant? Non sequiturs and ad hominem arguments work well in enterprise IT power politics, but I don't give them much merit when we're actually trying to solve real problems instead of spouting off how smart we are compared to the 'muppets'.

Update: Steve Vinoski adds his own response to this.

Elastra is hiring

| No Comments | No TrackBacks

Two notes

1. I've recently been promoted - my new title at Elastra is "Chief Software Architect", and I'm now reporting to the CEO, Kirill Sheynkman. I am responsible for Elastra's technology direction across its product lines.

2. Elastra is hiring! If you want to change the world of enterprise service management and provisioning, can contribute to a world-class team, and are located in (or want to relocate to) the San Francisco Bay Area -- send me an email.

About this Archive

This page is an archive of entries from July 2008 listed from newest to oldest.

June 2008 is the previous archive.

August 2008 is the next archive.

Find recent content on the main index or look in the archives to find all content.

About Me
(C) 2003-2011 Stuart Charlton

Blogroll on Bloglines

Disclaimer: All opinions expressed in this blog are my own, and are not necessarily shared by my employer or any other organization I am affiliated with.