thinking-out-loud

Idempotent Messaging

Ramkumar K. discusses idempotent services, and he later expands on the concept. Very informative. There was an interesting bit at the end:

One reader, by the name of David Taylor, mentioned that in their domain they follow a different approach to implement idempotent behavior. Apparently, in their domain, most of the messages contain some unique message identifier. For instance, they have used timestamps as a means to uniquely identify the message. Here is his message.

In my industry, we use an API called EPP for Domain Name provisioning (an IETF standard). In that case the protocol has been designed to be idempotent purely by API design (rather than sending unique identifiers).

An example being that when you renew a domain, you need to supply the current expiry date, so if you send a command to renew for 1 year (and then repeat the message) you dont end up with a 2 year renewal.

So the API designers have specifically designed each operation as idempotent without requiring a unique client or server identifier. It words quite well because you dont have to cache the unique ids.

I previously recommended the most excellent book Enterprise Integration Patterns. One of the patterns, aptly titled "Idempotent Receiver" deals explicitly with idempotency in the context of messaging, and discusses both the use of unique message identifiers and idempotent interfaces.

Reading that post digs up some [read: dorky] memories, both from college and work related to idempotent functions. The work related one was a much better memory, because it challenged me to dig back into the math that I had left behind when I decided to do software for a living so that I could speak intelligently on idempotent functions. Suffice it to say that the years had not been so good to my memory, but the refresher was good for my soul.

Idempotency has its roots in mathematics, and has a definition something like:

For any A, A is idempotent if A*A = A

This is commonly referenced in computing literature in the following form:

give a function f, f is idempotent if f(f(x)) = f(x)

another way to state that (which is the way a good partner from a good company stated in the aforementioned work-related memory): for each y in the range of f, f(y) = y

All of those definitions make thinking about designing idempotent services very difficult, if not down-right confusing. First, functions do not have side-effects, but services often do. Services that don't are idempotent. Services that insert and update records into databases have side-effects. Are they idempotent? Answer: Depends.

Second, it is difficult to ascertain what exactly most computer scientists, software engineers, hackers, whatever would consider the domain versus the range for a service, and what is an element of the range, and what does it mean to pass that element to the function?

Is it the document passed to the service? Is the entire state of the system? Is it the subset of the entire state of the system to which the caller of the service has access and influence?
Consider the following soap body as message to a service and then you tell me what is domain, what is range, and whether it is idempotent:

<hire>
<employee name="Jef"/>
</hire>

Inquiring minds want to know.

In computer science, idempotency is often simply reduced to "If you do the same thing multiple times, you get the same result" OR "If you do an operation more than once, it only has the effect of a single operation."

Now.

Back to Ram's article. David Taylor mentioned the domain renewal message and that it was idempotent. The specification he was refering to is RFC 3731: Extensible Provisioning Protocol Domain Name Mapping. In particular, he was refering to the 3.2.2, the EPP <renew> command.

Here is an [slightly modified] example from the RFC:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:ietf:params:xml:ns:epp-1.0
epp-1.0.xsd">
<command>
<renew>
<domain:renew
xmlns:domain="urn:ietf:params:xml:ns:domain-1.0"
xsi:schemaLocation="urn:ietf:params:xml:ns:domain-1.0
domain-1.0.xsd">
<domain:name>example.com</domain:name>
<domain:curExpDate>2000-04-03</domain:curExpDate>
<domain:period unit="y">5</domain:period>
</domain:renew>
</renew>
<clTRID>ABC-12345</clTRID>
</command>
</epp>

Note how the request accepts a domain name, a current expiration date, and a renewal period. The "current expiration date" reminds me of implementing optimistic locking to manage concurrency issues in database updates (which reminds me, I need to do another Recommending Reading entry).

This operation is idempotent because the system ignores renewals where the passed in "current expiration date" is invalid.

It would be easy to screw up designing this interface. Some alternative interfaces include:

Renew(domain, amountOfTime) -- resends of the same message result in unintended renewals

RenewUntil(domain, aDateInTheFuture) -- this could arguably be considered idempotent, but the result might not be what was expected, especially if the system doesn't guard against the current expiration date being greater than the new renewal date.

MSDN Architecture Center, Thought Leaders

The Microsoft .NET Architecture Center provides articles, references, and resources for application, service, and enterprise architecture in .NET. I just recently noticed that the site has a list of architecture-related blogs, all of which provide some excellent insight into MS, .NET, architecture, and what Don Box buys his wife for Mother's Day.

I wanted to feature one of the bloggers, Ramkumar Kothandaraman, here, because I think he provides some detailed insight into a lot of the SOA related concepts and issues that I have been talking or wanting to talk about. I find them particularly relevant to my current engagement, so a big hearty thanks for the mind dumps.

Microsoft Application Blocks

One more thing on the application blocks. When I say that they aren't the most sophisticated things in the world, and then I read it in full pixelated glory, I realized that it sounded more negative than I had intended. Maybe it wasn't the best choice of words.

Take this dictionary entry for sophisticate, for instance. Could mean less naive, could mean more complex, could mean impure! Heh. Your assignment, if you choose to accept it, is to download them and figure out which definitions apply.

Regardless, I wanted to correct myself [before anyone actually started reading my blog that loves the application blocks] by saying that I have seen/built frameworks in the past that address the concerns addressed by most of the application blocks, and some of them have been amazingly coherent and beautiful, and some of them have not. The MS app blocks are in the sweet spot of being useful for most applications. They have the additional benefit of being open-source, which is itself a double-edged sword. You can fix any bugs you find, and you can customize at will, but in doing so, you might run the risk of creating a maintenance headache. One of the blocks is community supported, the data access block.

Ron Jacobs, a member of the patterns and practices team, is the product manager for the application blocks. He is also the product manager for Shadowfax. Shadowfax is a reference architecture for service-oriented applications. It is currently in "closed" beta [they were nice enough to let me in the group without having a clue who I am, so they don't seem all that picky about membership], and they have one of those all-inclusive, ultra-restrictive NDAs that you have to agree to to access the code.

Microsoft Application Blocks

Daily.Fact.4: Dave Ranck maintains an index of Microsoft Application Blocks. The authoratative reference is, of course, maintained by Microsoft.

Opinion: Microsoft's application blocks provide a large body of source code that you may use as a starting point to building out your application's architectural frameworks. The code is well-written and open-source. The solutions are not panaceas, and they are not the most sophisticated solutions that exist either, but they are good, and they let you spend more time focusing on your core business, regardless of the fact that framework programming is way more fun.

Thought Leader: Sean McGrath

Sean McGrath opines on Transactions and SOA. In his post, he offers a guideline for scoping services that ties in to some of the earlier posts that I have been writing, and affords me a chance to go off a bit more. He says:

Systems that have ACID semantics make perfectly good services to expose on an SOA. Like any paradigm, one of the skills of service decomposition is knowing when to stop. A temporal coupling boundary (such as a classic ACID) is a good place to stop.

FWIW, This advice works well not just for web services but also for rpc and distributed objects.

I have seen several systems that on the surface appear to be doing the RightThang (tm), in that they are developing with the latest and greatest technologies and platfoms, e.g. J2EE or .NET, they are following iterative and incremental development processes, and all of their application interfaces are exposed via web services.

The problem is that they didn't know when to say when, and that led to basically a massive collection of CRUD-oriented interfaces on top of entity/domain class concepts, i.e. there ends up being a service per entity, say, for example, Employee, and the operations are basically CreateEmployee, SaveEmployee, FindEmployee, DeleteEmployee. The service should be defined at a business semantic that expresses a sense of acheiving a user/business goal. For the Employee example, for example, business operations might be Hire, Terminate, Promote, etc. The Hire operation, in addition to saving the employee information in the HR database, could provision a new laptop, establish the user's domain account and assign them the appropriate role based on their position, and send out a welcome email.

It is very typical that most of the aforementioned problems are systemic in the development process, in that the use cases are too fine grained and do not address goals as much as functions. This is the appropriate point to once again reference Cockburn's Structuring Use Cases with Goals.

Alt-click Magic

For years I've used a sweet little application called GuruNet to do quick look-ups of anything and everything I read on my PC. With it, you just press the Alt key and double-click on any word in any application to get on online references including dictionary, thesorous, language translations, and web search via Google.

Recently I've been working on a computer that doesn't run GuruNet at start-up. And, I discovered when Alt-clicking out of habit on a word in MS Word, that Microsoft has co-opted that feature. In the absence of GuruNet, Word 2003 does a dictionary look-up itself, and provides a link for language translation. Give it a try!

Thought Leader: Jean-Jacque Dubray

Jean-Jacque Dubray has easily one of the best sites on the web related to Business Process Management. He also recently publish a speculative article on Microsoft's future frontier that is worth the read. If I didn't know any better, I would think he was fishing for a job. I have followed his site for a while, but for whatever reason, I hadn't noticed his rss feed until today.

One memorable excerpt from the text that serves as a cautionary tales for those automating business processes.

Another key characteristic of a modern Business Process Definition language should be its inability--yes, inability-- to describe a business process from A to Z with all explicit exception paths: I was talking to a large company’s chief architect at a conference last November. His conclusions were stunning: his company had just gone through a complete mapping of all their processes. I said, ”That must be a great asset. You are ready for BPM.” “Actually, no,” he answered. They had come to the conclusion that because of the way they had modeled the processes, the resulting definitions were far too complex and, consequently, brittle. Nobody could comprehend the impact of any changes, let alone move the enterprise architecture to a BPMS.

Obtaining Consensus

Jared M. Spool of User Interface Engineering Fame published a new article today on the K-J technique, a prioritization/consensus technique.

Some of the biggest hurdles to enterprise architecture in general and service-oriented architecture in particular are gaining buy-in and establishing priorities to scope incremental releases (Down with Big Bang). As you adopt a service-oriented architecture, if you start to recognize that you are struggling with priorities or consensus, try it out. Heck, try it out even if you don't recognize that you are struggling.

Architectural Qualities

Architecture is about addressing the desired/required qualities of a system. These qualities are sometimes called "non-functional," which offends my mathematical understanding of a function, or "behavioral," which seems easily confused with "functional," or "modal," which is interesting, closer to the heart of the problem, and completely non-obvious. I merely stick with "qualities." These qualities have the largest influence on shaping the architecture of the system. A system designed to generate an annual financial report will be designed drastically different if it is run once a calendar year versus if it is run every single day for the prior linear year. This is largely driven by the performance-oriented throughput requirements.

What follows is a sample list of architectural qualities. Please note that each system has a unique list of architecturally important qualities, and for that list, each has a unique set of values. The samples below are for illustrative purposes only. Further, take in to consideration that many of these qualities are "summary qualities," that is, they actually have many different facets. For example, performance can be broken into several different categories, such as response time, throughput, memory or storage efficiency, and so on.

• Performance
• Security
• Availability
• Extensibility
• Flexibility
• Reliability
• Scalability
• Usability
• Simplicity
• Robustness
• Transparency

Enterprise Integration Patterns

Addison Wesley got Martin Fowler to put his John Hancock on the Martin Fowler Signature Book Series, an excellent collection of books on software.

The Recommended.Reading.1 entry is reserved for Enterprise Integration Patterns, the most excellent book by Gregor Hohpe and Bobby Wolfe. The book is all about using messaging to address enterprise integration issues. Although the book talks primarily about messaging, it is, in my opinion, the best book on service-oriented architecture on the market today. The material covers the depth and breadth of techniques related to messaging solutions, and references while stopping short of diving into the details of business process management systems [and rightfully justifying that as beyond the scope of the book].

The authors have maintained a website throughout the writing of the book, publishing early versions of the patterns for feedback, so you can get a little bit of the "try" before you bite off on the "buy."

As with most pattern books, the two most tangible benefits are codifying implemented, working and workable solutions to common problems and providing a common vocabulary by which to discuss the application of the solutions to new problems. For instance, if you are familiar with the patterns in the book, then I can make a statement like "The transaction aggregation system consolidates transactions from all of the lines of business (LOB). To increase the reliability of the solution, we will use Guaranteed Delivery with an Idempotent Receiver. To minimize the server utilization, the receiver will be an Event-Driven Consumer Service Activator. The messages will contain a Correlation Identifier to allow the LOB applications to correlate the acknowledgements, and will utilize a Messaging Bridge to connect the WebSphere MQ on the AS/400 and the Windows 2003 MSMQ.