Stu says stuff

I will be giving the Friday night keynote at the RESTfest unconference in Greenville SC on September 14. The topic should come as no surprise to those who have read my (admittedly stale) entries over the past few years. There are still a couple of spots open if you'd like a weekend of REST-related discussion in a relaxed environment...

Linking Data and Actions on the Web

Everywhere we look, there's a REST API popping up. For better or worse, we have massive proliferation of Web APIs, and a reasonable bench of developers who know how to design a "good" RESTful API (whatever that means).

The next progression for REST practitioners, in my opinion, is to find ways to ease integration headaches. There's a hole that exists between a site exposing a new API for their startup, and a fully approved media type RFC. This is where a good idea, exposed by an API, transcends its origins but isn't quite at the scale that it can survive 38 draft rounds within the IETF. This is also the area where enterprise integration lies - systems of limited scale and scope that might be able to take advantage of the constraints and emergent properties that the REST style offers.

Two areas of development would be helpful: firstly, a general purpose data model and associated media types for the web. Given the evolution of the semantic web and linked data, this is a hot area of interest, but one fraught with difficulty bringing to the mainstream. Secondly, and less explored, are media types that encourage programming models to shift developer's habits away from remote procedure calls (RPCs). There are Web programming models to explore: today's predominant model being dynamic RPC, but also event-driven, data access (CRUD), and state machine processors. Ideally a Web programming model should avoid the pitfalls of orienting all "write-side" communication in terms of flat actions or generic CRUD actions, but rather should encourage exposing side-effect inducing actions as something that can be extended & shared across the web via hyperlinks and URIs. This talk will explore approaches to linked data and actions on the web, their tradeoffs, and their possible future.

Montreal Smoked Meat (MSM) is a bit of a religion in Canada, particular for those who live in or around Montreal, Quebec. During my recent time off, I've been learning to cook with an outdoor smoker, and one of my projects was to try to recreate some quality MSM like the best Jewish delis in montreal such as Schwartz's are known for, or perhaps even a Pastrami of NYC Katz's Deli quality (a religion onto itself).

My Recipe (as of April 3, 2012)

I didn't invent this recipe, but amalgamated it from several sources (discussed below).

Equipment

1. A kitchen scale. I use one of the smaller, flat digital ones that my girlfriend bought at our local gourmet coffee shop. It measures up to 5kg and has a tare (zeroing) function to remove the weight of containers from the tally. I don't recommend using volume measurements when curing meat, especially with the variability of salt volume available out there.
2. A large sharp knife and fork. For cutting and serving the meat.
3. Ziplock Big Bags XXL. These are mammoth bags, nearly 2 x 3 feet, food safe, and large enough to seal in a full 15 pound packer brisket during its dry cure. They can be found at Canadian Tire stores in Canada, or Wal-Mart in the US. If you are using smaller cuts of meat, you can downward adjust your Ziploc size accordingly. If you have a big enough tupperware container, that's also usable.
4. A steamer or sous vide immersion circulator. This could be a stovetop steamer, rice steamer, roasting pan with a rack for the oven, or a large bamboo steamer. This is for finishing the meat, usually 3 hours. Note the sous vide approach I will describe below, based on the method in Modernist Cuisine. It requires a suitable bag + sealer that holds a 1:1 ratio of water to meat (thus is only appropriate for smaller cuts) and will require up to 72 hours.
5. An outdoor smoker (optional). I use the 18.5" Weber Smokey Mountain Cooker, which in my opinion the best bang for the buck in terms versatility for a charcoal-fired smoker. It also has a great online community behind it, dating back to 1997. Bradley Smoker makes a great electric smoker which is less versatile but dead simple to use. Note this is optional because there is some debate as to whether some establishments actually smoke their meat rather than just roast it. Smokers lead to better flavour, but using the oven will do in a pinch.
6. A digital probe thermometer (optional) For inserting into the meat - it's the reliable way to check doneness reliably. Every cut of meat is different and may takes way longer or way shorter to get to the desired temperature. Some ovens run hot, some smokers run hot. In my case, one brisket took only 3 hours to get to 165 F, another took 5.5 hours. Overcooking mainly means you may have stringier meat at the end.
7. A moderately cold refrigerator with room or a cold room. You don't want the meat to rot, you want it to cure, so you don't want to to be super cold. 38-40F or 3C-4.4C. Try not to go colder than 37F/2.5C.

Ingredient Summary
These are hypothetical quantities based on ratios from the initial cut of meat. I'm using the Modernist Cuisine format due to its ease of reading once you know the recipe; details on these ingredients below.

For the Dry Cure

For the Rub

For Smoking

1. Charcoal - lump or briquettes, depending on your smoker, enough for 4-5 hours of low heat (250F)
2. Smoke wood - about 4 to 6 fist-sized chunks of fruit wood (apple/cherry), pecan or maple. Hickory can be mixed in but sparingly (say 2 pieces out of 6). MSM traditionally used maple but in modern times isn't smoked at all. I like pecan.

For Serving

1. Rye bread
2. Mustard
3. Pickles (optional)
4. A large sharp knife
5. A large fork

Dry Cure Procedure

1. Trim some fat off the brisket, particularly on sides and top. Leave at least 1/4 to 1/2" of the fat cap on the bottom.
2. Rub the garlic powder on the brisket.
3. Combine the kosher salt with the curing salt, being mindful of meat to salt ratios. Rub the brisket with the salt mixture. If there's excess, throw it in the bottom of the ziplock bag.
4. Grind the remaining dry cure ingredients and mix together in a large bowl. Rub the meat with the dry cure spices. There shouldn't be much excess, but it can go in the ziplock.
5. Place the brisket in the bottom of the ziplock back, try to ensure any excess that was in the bag is evenly distributed on the meat.
6. Squeeze the air out of the bag and close the zipper; store the ziplock bag in a cold room or refrigerator, around 38-40F (not super cold).
7. Overhaul (turn over) the brisket every 12 hours or so, for 7 days. Smaller briskets can take less time (roughly, I'd estimate a 6 lb brisket for 5 days, a 15 lb brisket for 9 days).
8. After the cure, take the brisket out of the bag, and rinse off the curing spices.
9. Fill a large sink with water and soak the brisket for 3 hours, changing water every 1/2 hour. Pat the brisket dry with paper towels.

Rub and Smoke Procedure

1. Grind the rub ingredients and mix together in a large bowl. Rub the meat with the dry rub.
2. Optionally, wrap the brisket in ziplock again and let it sit in the dry rub in a fridge or cold room for 6-8 hours. (I skip this sometimes)
3. Light the smoker with the smoke wood, to 225-250F
4. Smoke the brisket until it reaches 165F internal temperature - around 4 to 5 hours, depending on the brisket size and temperature of your smoker
5. Remove from smoker, and either proceed with steaming, or wrap in foil, then ziplock or plastic wrap, and refrigerate until ready to eat

Steaming and Serving

1. Carve the fatty (thicker, point end) of the brisket to separate it from the leaner (flat end) meat, and carve again into smaller chunks to fit in your preferred steamer.
2. If using a stovetop steamer, get the water steaming, and keep on low. Oven roasting pans should be filled with water up to the rack and the oven set to 200F. If using sous-vide, set your immersion circulator for 140F.
3. Steam for 3 hours, until the brisket is fork tender.
4. If using sous vide, you have two options. (a) Vaccuum pack your chunk and cook for up to 72 hours. I found this didn't work as well as straight steaming, hence ... (b) The modernist cuisine way calls for putting an equal weight of the pastrami brine (water would be fine) in the bag with the pastrami, seal it (this may be difficult with non-chamber sealers), and cooking for 72 hours. I have not personally tried this approach yet.
5. Once tender, grab the chunk with a large fork, carve brisket chunks against the grain with a sharp knife, serve on rye bread with mustard.

Ingredient notes

1. Beef cuts. In order of preference: Full packer brisket, beef cheeks, boneless short ribs. Most MSM is from the brisket, and has widest variety of "fatty vs. lean" cuts. I will be discussing brisket in this recipe, but other cuts are great for beginners (and have great flavour). Ensure you get a good cut: USDA Prime or Choice and/or Canada AAA or Prime. Note that brisket in the USA is much cheaper than in Canada, UDSA Choice runs for $2/pound whereas Canada AAA can be $6-7/pound. That means in Canada you'd be dropping a C-note for a full packer brisket.
2. Salt. I prefer Diamond Crystal Kosher Salt's feel and density, but use whatever you can get. Make sure it's just non-iodized sodium chloride, it isn't a mix. A 3/4 full box should be sufficient (i.e. a brisket will use up to 2/3 of a pound). Always measure salt by weight when curing - 2 cups of diamond crystal have the same weight of 1 cup of table salt! It's the same stuff, but the flakes / crystals are of a different size.
3. Curing (pink) Salt .This is usually a 6.25% Sodium Nitrate (cure) to 93.75% Sodium Chloride (salt) mixture, and died pink to ensure you don't mistake it for regular salt. This is available at most Bass Pro Shops as LEM Cure, and one packet will cure up to 100 pounds of meat. For those worried about the health effects of nitrates, please see this blog post from Michael Ruhlman. In short: don't worry about it. Use only the amount required, and it is safe.
4. Sugar. White refined is fine; demerara or turbinado sugar also is fine and you may prefer the more molasses-y taste.

Sources of Recipe Inspiration

This is a direct derivation of what I learned from these sources, and all credit goes to them for pioneering home smoked MSM / Pastrami:

1. This Chowhound thread
2. This eGullet thread
3. The Modernist Cuisine bookset, which has a very similar ingredient list to mine, but uses a brine instead of a dry cure, and has more sugar, since it is Pastrami.
4. The Three Squabbling Asians
5. The Virtual Weber Bullet recipe from Chris Allingham
6. The Virtual Weber Bulletin Board threads on Pastrami and MSM
7. And last but not least, Michael Ruhlman

Photos

1. See my Flickr photoset for a few of photos of the ziplock bag , brisket, and yummy results from a couple of attempts.

I've been off the past few months recovering from illness and just generally relaxing. Blogging to recommence.

I posted a couple of thoughts on Twitter and a few comments elsewhere recently as a results of a discussion on George Reese's blog and William's followup. I thought I'd repost these thoughts here with annotation as it seems to be evolving an interesting discussion.

1. REST APIs are a case of good intentions run amok. The point of REST was to *eliminate* specific APIs. Not cause their proliferation.

2. When someone says you shouldn't document a REST API, they're saying you shouldn't have an API. Not that you shouldn't document.

I mean by this, Network APIs. I also mean APIs that aren't re-implementable across sites. The point is that the uniform interface isn't just HTTP - it's a small number of evolving media types, and the goal is to enable sharing of information at a global scale. Your pet JSON format may not be the way to do this.

3. That said, RESTians saying you shouldn't have an API are somewhat vacuous because they offer no good alternative. Custom media types ain't.

4. Thus, IMO, until we have a workable generic media type or two for the 80% case of "REST APIs", we're just growing a frustrating mess.

In other words, I completely recognize that crafting a media type is a bit heady, and even still a custom media type for generic use is quite hard. With a REST API at least you get the benefits of HTTP GET (to some degree). It does the job, if all you want is a trendy vaguely hopeful way of building something for the long term without getting too bogged down in philosophy. I get it.

What I would like to see is a generic media type (above & beyond a "format" like JSON) to cover the 80/20 cases. This is seemingly contrary to where many RESTheads are at - they prefer often prefer opinionated media types, media types that "say something".

I'm actually asking for that too, I think -- say something about the programming model that breaks people's habits away from RPCs. Find a programming model that doesn't orient all "write-side" communication in terms of a flat list of actions, and into something that can be extended & shared across the web via hyperlinks and URIs. Hierarchical statecharts is my stab at this. There may be others.

5. From William's blog (with a couple tweaks)....

I've sort of half given up on REST advocacy because it's pretty impossible these days. People want to ship APIs that will be valid for the next next 6 months, not build systems that will evolve over a decade or more.

Jan Algermissen's comment about how when you've documented an API, you're not really doing REST, is actually spot on, fundamentally, but is met with ridicule. I can completely understand how vacuous the comment sounds if you just want to ship an API nownownow, are being short-term practically minded, or are just playing buzzword bingo with architectural styles. But if we actually want to leverage the benefits of the style, we'd work on the core issue of having a generic media type or two for these sorts of use cases.

If one is thinking of "how to methods map to my URI structures", we've basically lost the plot as to why we have HTTP, URIs and media types in the first place. The goal is to *eliminate* the API as we normally think of it. (It's not about a library of calls over a network).

Five points. The consequences of REST changes some rules:

1. The assumption can't be that the end user of your information service is going to be a programmer. Building an API bakes that assumption in how you expose your information or controls to manipulate it. The assumption rather should be that "this can be any agent of any sort". If your goal really is "I want a programmer to learn my API and program against this", then there's really no reason to use REST other than that it's trendy (and HTTP GET).

2. The assumption can't be that you will be the only site implementing your interface. The whole point of the Web is sharing information - publishing an API bakes in the assumption that YOU, the publisher, are in control. It is the consumer, on the other hand - the agent itself, that's in control of how/what the information is consumed or manipulated.

3. The semantics of the interface that need to be documented are the media type that you use to describe your data and links. The structure of your URIs really shouldn't be a part of said semantics.

4. But doing this in a custom media type implies #2, that you're the only person implementing your interface. Of course, every media type has to start somewhere, but it really doesn't help matters unless you understand that others may be *implementing* your interface, not just *talking* to it. Facebook's Open Graph, for example, is an example of a custom overlay media type that takes great pains to be generic in either direction.

5. All of the above is really heady stuff that is above most developer's pay grades, so it won't ever happen. The best we can hope for is some person/organization to step up with a generic media type or two + guidelines that help describe and document a Web of Data interface for accessibility by programmers. Call it a description language, if you'd like. Some RESTians don't like the idea of a description language. I think it's an essential 80/20 piece if developers are going to actually *use* the architecture the way it was meant to be used.

Currently (as I mentioned in my WS-REST keynote) I think some kind of hierarchical lifecycle/state machine might be the right abstraction for such an interface. I'd like to demonstrate this if I ever got past my day job sucking up most of my time. (But I may have an opportunity to work on this there soon.)

Slides for my keynote, "I'll See you on the Write Side of the Web" at the 2nd International Workshop for RESTful Design, WWW 2011 in Hyderabad, India are available here.

Recording (75 minutes) is now available at this link. (It's an M4A / AAC file, sorry).

I'll eventually post my thoughts on the workshop along with feedback & discussion from this keynote.

I will be giving the keynote at the 2nd International Workshop on RESTful Design, entitled: I'll See You on the Write Side of the Web.

Yes, I do wonder if I have Brain Damage some days ;)

Here's the abstract:

While global-scale digital information sharing has become synonymous with REST,
state manipulation remains an inconsistent experience. Many so-called "RESTful
APIs" are specified out of band, through documentation, rather than in
hypermedia. Developers struggle with constructing their own custom
hypermedia-driven media types, falling back on lightweight site-specific data
structures. And the few standardized publishing media types, such as the Atom
Publishing Protocol, have had limited adoption. All of this serves to undermine
the serendipitous reuse of "writes", a quality we take for granted when dealing
with "reads". Of course, state-changing actions are semantically much more
difficult to describe generically than a simple HTTP GET, and it is an open
question as to whether it's even possible to do so practically.

This talk will explore whether there's an opportunity to provide generalized
media types and programming frameworks to assist with resource state
manipulation, and what they might look like, drawing inspiration from other
computing domains with dynamic environments, such as video games, robotics, and
embedded systems controllers.

Slides will be posted around the same time as the keynote.

The Second International Workshop on RESTful Design (WS-REST 2011) is a forum for discussion and dissemination of research on the the Web's architectural style, Representational State Transfer (REST). It will be held on March 28, 2011 as part of WWW 2011 in Hyderabad, India.

Last year had a high quality set of attendees and submissions, and was a great opportunity to meet those involved in this community.

I'm on the program committee again this year. We are accepting four-page position papers, or eight-page full research papers on your favourite REST topic. Send your papers in by Jan 31st! Even if you can't make it to India, it will be a great forum to disseminate your research!

Just read William's Partial resource update, one more time, which is another kick at the partial resource update can.

The problem is that few web services or RESTful hypermedia describe a data model. Representations themselves don't often describe a data model, they describe a format. This is the same problem as WS-RT, in that they're trying to describe updates to a format (XML), not a data model.

In Atom, this doesn't matter, as it kicks the responsibility to the entry.
In HTML, this doesn't matter, as the machinery of browser & server only needs the format for interoperability.

In systems-to-systems communication, this matters a whole lot, as you're coupling your expectations differently with every media type.

A data model is foundational, it describes the mathematical basis for how data is managed. Examples (for better or worse) include:

1. Key/Value


2. Arrays


3. Relations


4. Logical Graph


5. Inverted List

OData describes an implicit data model, based on relations. RDF and OWL describe graph data models with an explicit theoretic foundation that makes them more powerful than Relations.

"Your New JSON Service" probably has a data model, but I bet it's unpublished, and in your head. And even if you did publish it, it probably is different from "this other guy's JSON Service".

What does all of this mean? I think it means that there's two big pieces of work to be done to help evolve RESTful web services to better interoperability:

(a) a data model that covers 80% of common use cases and can be formatted with JSON and XML. This needs to be a closed-world model, like most databases, and thus I don't think RDF qualifies.

(b) a media type that covers 80% of common use cases to describe a resource's lifecycle and state transitions -- in other words, to make POSTs self-descriptive in hypermedia. Because the world of computing isn't just about updating data, it's about abstractions.

It is passe' to perform SQL updates directly from your GUI. It's just as passe' to expect every RESTful hypermedia API to just behave like a database entity. So, we need something in between, something that gives POST meaning to systems. POST is the method which powers most of what we DO (vs. read) on the Web anyway. This "roll your own media type" trend works for simple situations, but really isn't sustainable in the long run, in my opinion.

p.s. yes, that "Part 2" of building a hypermedia agent article is coming ;-)

In light of some interesting twitter, blog, and IM discussions over yesterday's post (thanks to Chad Walters, Billy Newport, Ryan Rawson, Bill de hOra, Jeff Darcy, and others), I've been pondering three CAP points. Last post on this topic (for a bit), I promise, before I return to your regularly scheduled RESTful programming.

1. Definitions Matter. Or, are CA/CP are the same thing?

Gilbert & Lynch's paper defines the following (I'm not going to use their exact terms, so as to keep this accessible)

• Consistency (big-C)- aka Atomicity, or a total order on operations (as visible to any user); aka. by database practitioners as serializable isolation. Makes distributed systems look like a single system.
• Eventual Consistency (little-C)- aka Delayed-t Consistency, or a bounded amount of time before a data object is consistent, also implies a minimum latency between writes on a single piece of data (without specialized conflict resolution)
• Weak Availability (little-A)- every request received from a non-failing node must result in a response, but this response may take forever to come (i.e. latency is not a part of this definition)
• Strong Availability (big-A)- every request received from a non-failing node must result in a response even when network partitions occur, but this response may take forever to come (i.e. latency is not a part of this definition)
• Strong Network Partition Tolerance (big-P)- the network will be allowed to lose arbitrarily many messages sent from one node to another (not counting total network outage). When a network is partitioned, all messages sent across the partition are lost.

• Weak Network Partition Tolerance (little-P)- a system that can tolerate some network partitions, but not all combinations of them.

I add this due to my reading of their definition of Big-P and sections 3.2.1 and 3.2.2: "If there are no partitions, it is clearly possible to provide atomic, available data. In fact the centralized algorithm described in Section 3.2.1 meets these requirements. Systems that run on intranets and LANs are an example of these type of algorithms". Section 3.2.1 defines an algorithm for Consistent & Partition Tolerant systems. Hang on to this one, I'll finish my point in a moment.

So, by these definitions, you basically can have (assuming mixed reads & writes):
- AP: eventual consistency, strong availability, strong partition tolerance
- CP: strong consistency, weak availability, strong partition tolerance
- CA: strong consistency, strong availability, weak partition tolerance

Clearly an AP system is different from CP/CA. But the crux is whether CA and CP are the same kind of system. If you consider the idea of weak vs. strong partition tolerance, the difference becomes clear (to me, anyway, but I'm admittedly blonde).

A CA system might be able to handle some partitions, but not all. Simply put, it has a single point of failure, somewhere. The example I like to give is a shared-disk cluster database with a SAN. Partition the SAN, and you're hosed. Partition the nodes, or even the interconnect, and you will still be available, albeit with higher latency. Redundancy is the common way to reduce this risk.

Whereas a CP system is designed to handle all partitions, i.e. it has no single point of failure. But some non-failed nodes may not be able to service clients at all during a partition. This can suck. Anyone who's managed a synchronous log shipping setup or EMC SRDF probably knows about this one.

2. Scope matters

The definition of "node" and "network" can vary depending on your granularity and scope.

The above definitions only make sense if the scope of the system is the server-side of a client/server relationship. (i.e. How can non-failed nodes experiencing a partition still receive and respond to a request? Only if the client isn't the one being partitioned.)

A database that's normally a CA system can be considered CP if you zoom out to notice that it's in a multi-node synchronous replication ring. Or AP if it's in an asynchronous multi-node replication ring. But notice that, probabilistically, it's behaving like a CA system for 99.9% of the time (or however long your MTBF is).

An AP system on the other hand has one big drawback, one that's not spoken about often. It's about the scope of the partition: is it recoverable or not?

An unrecoverable partition in a CP or CA system is "no data loss", even if you can't get at the system. That's not true in an AP system, if there's an unrecoverable error in a certain set of nodes during the "delayed consistency" time window. This occurs during catastrophic media failures, like a bug in the SAN controller corrupting all the disks, trucks driving through a data center, or floods, or bombs, hurricanes, etc.

Even with backups, during this "replication latency", you have to re-enter those transactions, or hope that a CP or CA system somewhere has a copy.

3. Definitions are too easy to get hung up on.

One of the main reasons I'm talking about this is because I see bolded, underlined claims that can be easily misconstrued, like, "you can't choose consistency and availability together in a distributed system". This gives me The Fear.

This point of the above quote, is not to say you can't have some consistency. You just can't have fully serializable consistency and isolation.

But in practice, when you think about it, this claim is actually rather banal. Practitioners have long understood that there are a spectrum of consistency & isolation tradeoffs at scale, even at a single node. Why?

Because the definition of availability arguably needs to also include latency, which is something Daniel Abadi brought up. Serializability means locks, in practice, and that means increased latency, and thus reduced availability. This is why we don't like Two Phased-Commit very much these days.

But, for kicks, go back to the ANSI SQL isolation level debate that Jim Gray stirred up. It was because databases like Oracle (gasp) don't provide serializable consistency, and haven't since the late 1980s! They provide snapshot consistency, which is a different beast where readers don't block writers, and arguably was the primary technical reason for Oracle's success in the market place through the 1990s. Amusingly, this is the same argument that Jim Starkey keeps bringing up when discussing CAP, having invented the idea of multi-version concurrency control, when he's talking about his new baby, Nimbus DB.

So the idea that you can't have fully serializable consistency at scale -- even in a single node database -- is completely uncontroversial. It's when the followup feels like "...and therefore you need to throw out your RDBMS", that cognitive dissonance and tempests in a tweetpot stirs up.

A bit of a Twittergasm lately over the CAP Theorem, with relevant blog posts

• You Can't Sacrifice Partition Tolerance by coda hale


• Problems with CAP by Daniel Abadi


• Clarifications on the CAP Theorem by Mike Stonebraker


• Reactions to Coda's CAP Post by Jeff Darcy


• Someone is Wrong on the Internet by Jeff Darcy

I am confused. I've narrowed it down to 7 matters.

Confusion #1: Changing system scope from underneath our feet
The biggest confusion I have is that the scope of "distributed system" under the CAP theorem seems to have changed, silently.

If you look at the original papers:

• Lessons from Giant-Scale Services by Brewer


• Harvest, Yield, and Scalable Tolerant Services by Fox & Brewer


• Toward Robust Distributed Systems presentation by Brewer


• Brewer's Conjecture and the Feasibility of Consistent Available Partition-Tolerant Web Services
by Gilbert & Lynch

All of them seem to scope the CAP theorem to the server side of a client/server relationship. That is, it's assumed that the distributed system under consideration is a client/server architecture, but the server-side is distributed in some way. This is a reasonable assumption, since the Web is a client/server architecture variation, and CAP was originally about the tradeoffs in huge scale web sites.

But practically speaking, this is particularly important when considering Partition Tolerance. In the original papers, CA (Consistent, Available) was absolutely common, and in fact, the most common scenario, with Brewer giving examples like cluster databases (on a LAN), single-site databases, LDAP, etc. The most popular commercial cluster databases today arguably retain this definition of CA, such as Oracle's RAC.

But now, AP (availability, partition tolerance) advocates are questioning if CA distributed systems are even possible, or merely just something that insane people do. It's a very strange, extreme view, and I don't think it offers any enlightenment to this debate. It would be good to clarify or at least offer two scopes of CAP, one within a server scope, and another in a system-wide scope.

Confusion #2: What is Partition Intolerance?

This is never well defined. Like lactose intolerance, the results, can be, err, unpredictable. It seems to imply one of: complete system unavailability, data inconsistency, or both. It's also unclear if this is recoverable or not. Generally it seems to depend on the implementation of the specific system.

But because of this confusion, it's easy to let one's mind wander and say that a CA system "acts like" an AP or CP system when experiencing a partition. That leads to more confusion as to whether a CA system really was just a CP system in drag all along.

Confusion #3: Referring to Failures Uniformly

It seems to be a simplifying assumption to equate failures with network partitions, but the challenge is that there are many kinds of failures: node failures, network failures, media failures, and each are handled differently in practice.

Secondly, there are systems with many different kinds of nodes, where they tolerate partitions in one kind of node, but not another kind, or just maybe just not a "full" partition" across two distinct sets of nodes. Surely this is an interesting property, worthy of study?

Taking Oracle RAC or IBM DB2 pureScale, as an example. It actually tolerates partitions of many sorts - cluster interconnect failures, node failures, media failures, etc. What it doesn't tolerate is a total network partition of the SAN. That's a very specific kind of partition. Even given such a partition, the failure is generally predictable in that it still prefers consistency over availability.

Confusion #4: Rejecting Weak vs. Strong Properties

Coda hale sez: "You cannot, however, choose both consistency and availability in a distributed system [that also tolerates partitions]".

Yet this seems very misleading. Brewer's original papers and Gilbert/Lynch talk about Strong vs. Weak consistency and availability. The above statement makes it sound as if there is no such thing.

Furthermore, most distributed systems are layered or modular. One part of that system may be CA-focused, the other may be AP-focused. How does this translate to a global system?

Confusion #5: Conflating Reads & Writes

Consider the Web Architecture. The Web is arguably an AP system for reads, due to caching proxies, but is a CP system for writes.

Is it useful to categorize a system flatly as binary one or the other?

Confusion #6: Whether AP systems can be a generic or the "default" database for people to choose

AP systems to my understanding are application-specific, as weak consistency is difficult to correct without deep application knowledge or business involvement.

The question and debate is, what's the default database for most IT systems... should it be a traditional RDBMS that's still a CA system, like MySQL or Oracle? Or should it be an AP system like a hacked-MySQL setup with Memcached (as AP), or Cassandra?

When I see claims that CA advocates as "not getting it", or "not in their right mind", I have a severe case of cognitive dissonance. Most IT systems don't require the scale of an AP system. There are plenty of cases of rather scalable, highly available CA systems, if one looks at the TPC benchmarks. They're not Google or Amazon scale, and I completely agree that an AP system is necessary under those circumstances. I even agree that consistency needs to be sacrificed, especially as you move to multi-site continuous availability. But it's very debatable that the basic database itself should be CA by default or AP by default.

For example, many IT shops have built their mission-critical applications in a CA manner within a single LAN/rack/data-centre, and an AP manner across racks or data-centres via asynchronous log shipping. Some can't tolerate any data loss and maintain a CP system with synchronous log shipping (using things like WAN accelerators to keep latency down). And not just for RDBMS - this is generally how I've seen Elastic Data Cache systems (Gigaspaces or Coherence or IBM WebSphere ExtremeScale) deployed. They deal with the lack of partition tolerance at a LAN level through redundancy.

It's useful to critique the CA single-site / AP multi-site architecture, since it's so common, but it's still not clear this should be replaced with AP-everywhere as a useful default.

Confusion #7: Why people are being a mixture of dismissive and douchey in their arguments with Stonebraker

Stonebraker's arguments make a lot of sense to IT practitioners (I count myself as one). He speaks their language. Yes, he can be condescending, but he's a rich entrepreneur - they get that way. Yes, he gets distracted and talks about somewhat off-topic matters. Neither of those things implies he's wrong overall. Neither of these things imply that you're going to get your point across more by being dismissive or douchey, however good it feels. (I'm not referring to anyone specifically here, just a general tone of twitter and some of the snark in the blog posts. )

One particular complaint I find rather naive - that he's characterizing CAP theorem (more AP system) proponents as only using CAP theorem to evaluate distributed system availability. Clearly that's not true, but on the other hand, it is kind of true.

Here's the problem: when I hear venture capitalists discussing the CAP theorem over cocktails because of their NoSQL investments, or young newly-graduated devs bugging their IT managers to use Cassandra, you can bet that plenty of people are making decisions based on a twisted version of the new fad, rather than reality. But that's how every technology fad plays out. I think Stonebraker has seen his share of them, so he's coming from that angle, from my view. Or he could just be an out of touch dinosaur like many seem fond of snarking. I personally doubt it.