Introduction to ESBs: What are ESBs and Why Would You Need One?
Introduction to the Series
I am starting a new series that takes a deeper look at what Enterprise Service Buses (ESBs) are, and the benefits they bring to both integration as well as driving SOA adoption. I am going to focus on the Neuron ESB in the series for several reasons. First, it is really the only packaged out-of-the-box ESB built with 100% Microsoft technologies, that meets the full definition of an ESB that I will discuss shortly. I am very aware of the fine Open Source solutions, nServiceBus, and Mass Transit. In my opinion, they are excellent frameworks for doing Pub-Sub Asynchronous Messaging over MSMQ. However, both of those solutions are frameworks, like WCF, to program to, and not packaged products. In addition, they both use MSMQ as the underlying transport. This is a fine choice for reliable, asynchronous messaging using queues. However, Neuron, by building on top of WCF, offers the full breadth and diversity of using many different transports and qualities of service in addition to queued asynchronous messaging, and the ability to do this without changing any code. The second reason, I am focusing on Neuron, is that I simply find, for my purposes, it is the easiest ESB I have ever worked with. One can be up and going in minutes and the User Experience through the UI, makes complex things very simple. The third reason is that Neuron offers extensive integration capabilities, through both its Adapters as well as WCF LOB Adapters that allow integration opportunities to many types of systems.
Introduction
As IT departments and enterprises move towards Service Oriented Architectures (SOA), they find that they are dealing with a mixture of the old and the new. There is a strong need for integration, as well as for providing an infrastructure for services. The Enterprise Service Bus (ESB) is all about integrating the old and the new, and providing a central place for services, applications, and general IT resources to connect. In other words, what we want is an infrastructure and eventing system that will allow you to connect any IT resource regardless of technology or where it is applied. This infrastructure should allow you to handle changing requirements without disruption. This infrastructure should be robust and reliable. In comes the ESB. An ESB not only allows you to combine and re-assemble services, but also allows you to connect new applications, web services, and hundreds of other applications, including LOB applications, batch files, and legacy middleware applications via Adapters, while maintaining the same Publish-Subscribe messaging abstraction.
As enterprise adoption of services continues, the need for a service-oriented architecture will start to be felt. There's a big difference between the casual use of services and running your enterprise primarily over services. As enterprises travel down the road that will take them from light use of services to deep use of services, many issues will arise, such as how to manage large numbers of services well; how to overcome differences between services; how to enforce SLAs; and how to enforce enterprise policies across distributed collections of services.
There is another important factor in the Microsoft world, where we have excellent distributed communication frameworks like the Windows Communication Foundation (WCF). However, they are just that, frameworks. One thing I have seen us a lack of WCF adoption due to the steep learning curve. People like myself, find it incredibly simpler then the mess of distributed stacks we had to deal with before, but I am in the minority. Moreover, most IT shops don't have developers experienced in this kind of programming. What they have is C# and VB ASP.NET developers, who, as they should be, are focused on delivering business value.
I have seen similar pain points in SOA adoption. There are real benefits in SOA but people are either off on vendor exercises or writing a ton of infrastructure code. In either case, the real benefit of Business and IT dynamic alignment and focusing on Business Services has been neglected, leading some people to declare the end of SOA.
I have found, by personal experience that once you get above a few WCF services you start to get issues you didn’t have before. Services are JBOWS all over the place, instead of being part of an SOA that was developed incrementally with the business drivers and needs. There is no governance. Configurations are haphazard. There is no versioning. And so on. So, once you start getting serious about WCF and services, and getting more than five stood up, you start to need infrastructure more and more, putting the "A" in "SOA."
This article is designed to set a foundation for the series.
The Enterprise Service Bus
Pinning down the definition for ESBs can be just as elusive as pinning down the definition for SOA. There are certainly multiple definitions which some critics point as evidence that there is no such thing as an ESB. It’s also true that some vendors use ESB Definitions that are self-serving. My desire is to stay out of that, look at definitions and see what is common.
"A Web-services-capable infrastructure that supports intelligently directed communication and mediated relationships among loosely coupled and decoupled biz components." - Gartner Group
"The ESB label simply implies that a product is some type of integration middleware product that supports both MOM and Web services protocols." –Burton Group
"A standards-based integration backbone, combining messaging, Web services, transformation, and intelligent routing." –Sonic Software
"An enterprise platform that implements standardized interfaces for communication, connectivity, transformation, and security." - Fiorano Software
"To put it bluntly: If you have WebSphere MQ and other WebSphere brokers and integration servers, you have an ESB." –Bob Sutor, IBM
"The Enterprise Service Bus is a uniform service integration architecture of infrastructure services that provides consistent support to business services across a defined ecosystem. The ESB is implemented as a service oriented architecture using Web Service interfaces." –CBDI
However, I do think one can be practical about it, stay out of the wars and see what's common in the definitions. One practical view of the "common" ESB Functions can be found in the Wikipedia definition [1]
|
Category
|
Function
|
|
Invocation
|
Support for synchronous and asynchronous transport protocols, service mapping (locating and binding)
|
|
Intelligent Routing
|
Addressability, static/deterministic routing, content-based routing, rules-based routing, policy-based routing
|
|
Mediation
|
Adapters, protocol transformation, service mapping
|
|
Process Choreography
|
Implementation of complex business processes
|
|
Service Orchestration
|
Coordination of multiple implementation services exposed as a single, aggregate service
|
|
Complex Event Processing/EDA (Real Time Events)
|
Event interpretation, correlation, pattern matching
|
|
Other Quality of Service
|
Security (encryption and signing), reliable delivery, transaction management
|
|
Management
|
Monitoring, audit, logging, metering, admin console, BAM
|
From this, we extract a composite definition of which we would expect a “true” ESB to encompass a healthy subset:
· An ESB is a backbone for connecting and integrating an enterprise’s applications and services.
· An ESB provides the necessary infrastructure to create a service oriented architecture.
· An ESB is a convergence if EAI, MOM, and SOA concepts.
· An ESB is based on open standards such as XML, SOAP, and WS-*
· An ESB provides intelligent routing, such as publish-subscribe, message brokering, and failover routing
· An ESB provides mediation, overcoming data, communication, and security differences between endpoints
· An ESB integrates with legacy systems using standards-based adapters
· An ESB provides logical centralized management but is physically decentralized
· An ESB is able to apply EAI concepts such as rules and orchestrations
· An ESB is able to monitor and throttle activity as per a Service Level Agreement (SLA)
In addition to the Wiki definitions, I would add several other core capabilities, of which Neuron is very strong in. The first of these, Pipeline Processing, is in addition to the Event-Driven Architecture. An ESB should allow matters related to communication and integration to be completely orthogonal to those concerned with business logic, which should stay completely within the enterprise application. By offering a full set of “pluggable” message processing functions, an ESB like Neuron, can assemble these processing functions into “pipelines,” which can run when messages are sent and received. These functions can range from message transformations to kicking off workflows to executing a set of business rules, to name a few.
In addition, two other factors are very important in an ESB: Deployment and the Developer Experience. I will talk about this in detail in a future blog post, but Neuron ESB deployments can range from a single server at a single location to federation of high-availability server farms in multiple locations. Lastly, I really love the Neuron developer experience. While most ESBs require you to learn a whole new set of skills, Neuron builds on my .NET and WCF skills. I use the same C# and VB.NET skills that I would use in building any other .NET application. Furthermore, Neuron is now integrated into Visual Studio allowing you to view your Topics, Publishers, and Subscribers and generate the code for them with a mouse click.
In this article, we are interested in looking how an ESB can help us over the longer-term as an "investment" made now to help us accelerate our SOA efforts. In other words, buying an ESB will not implement a service-oriented architecture, but provide the features in which one may be built
But what is an Enterprise Service Bus and what is the value of it to my infrastructure and to my SOA? For our purposes, I am going to focus on three main areas:
An ESB is an operating environment for Services
An ESB provides the architecture needed for deep adoption of services. As I stated earlier, once your organization begins to move from casual use of services to running your enterprise primarily on services, the need for something like an ESB will be more acutely felt. As stated above, many issues will arise, such as how to manage large numbers of services well; how to overcome differences between services; how to enforce SLAs; and how to enforce enterprise policies across distributed collections of services. An ESB is a backbone for connecting and integrating an enterprise's applications and services. An ESB accelerates SOA by providing infrastructure services to complement business services.
What are these "infrastructure" services? They are a whole set of valuable services that you need to use over but should not have to write. These include routing, storing, and forwarding of messages, business activity monitoring, transformations and EAI functions. This is not rocket science as most enterprises contain common infrastructure services like domain controllers and directory services like LDAP or AD.
An ESB is a Common Messaging Fabric
In the ESB model, most or all applications and services in the enterprise connect to the ESB and communicate with each other over the ESB. Applications and services usually connect using SOA standards, whereas legacy systems require integration via EAI technologies such as adapters. The communication between endpoints is handled by message-oriented middleware arranged in a bus topology. The primary advantage of such an approach is that it reduces the number of point-to-point connections required to allow applications to communicate. This, in turn, makes impact analysis for major software changes simpler and more straightforward. By reducing the number of points-of-contact to a particular application, the process of adapting a system to changes in one of its components becomes easier. [2]
The ESB serves as a common messaging fabric for the enterprise. Programs connect to the ESB and send or receive messages. The ESB handles routing details, mediation of differences between endpoints, and the physical details of "It's far more sensible to put such matters in the hands of business analysts and I.T. personnel who can make enterprise-level decisions than having them controlled by developers at the application level." [3]
Much of the value an ESB provides is in the area of centralized management. An ESB provides a single place to handle management functions such as configuration, deployment, monitoring, and control. Having a central facility like this makes change management straightforward and rapid response possible. Not having centralized management creates a significant problem for I.T. departments and imposes unnecessary cost and delays when change is required. [4]
An ESB is a Long-Term Architectural Pattern
A subject of much debate is whether an ESB is a pattern or product. The most obvious answer would seem to be "both": an ESB can certainly be described as an architectural pattern and there is more than one path to creating one. You have to build your ESB out of something however, so its natural that one or more products would be used to accelerate the process.
It is my feeling that the ESB is first and foremost a pattern. Like most patterns in this space, it is cataloged in Hohpe and Woolf's Enterprise Integration Patterns as Message Bus. As such, an ESB for the long-term should be possible to survive incremental and generational changes in technology without having to throw away the pattern.
Where Are We?
In this first article of the series, I first gave my rationale for looking deeper at the Neuron ESB. Then I presented the value scenarios for an ESB. Next, I used Wiki’s ESB topic as a list of general common traits of an ESB. Following that, I focused on the ESB as an operating environment for services, the ESB as a common messaging fabric, and the ESB as an Architectural Pattern.
In the next installment of the series, I will look deeper at Neuron as an ESB, focusing on showing how easy it is to connect. Indeed, this is one of the major reasons I use Neuron as it literally takes minutes to get basic messaging up and going with the GUI and zero code. I will then look at the pieces of Neuron: the ESB Explorer, the ESB Service, the ESB Test Client, and the ESB Configuration. From there, I will introduce Neuron Topics as powerful abstractions of business conversations. See you next time!
References
Contents
ReBlogger Contents
Email TopXML