• Head of IT Operations -...
• IT Service Desk Analyst -...
• Systems Engineer (Senior)
• Wintel Systems Specialist
• Project Manager

SOA meets the real world

Eric Knorr, Information Age

09/06/2005 15:26:09

Service-oriented architecture is an idea, not a technology. Boundless in scope, it promises both unlimited software reuse and the interconnection of everything, as long as IT is willing to wrap legacy applications in standard interfaces and construct new apps as services, the capabilities of which other software can tap into.

The idea is simple, but the execution isn't, because SOA turns the conventional model of enterprise software development on its head. Normally, programmers write software based on a set of well-defined requirements. SOA demands that organisations create an ecosystem of services that may ultimately have an army of stakeholders inside and outside the firewall. The initial challenge of SOA is knowing where and how to start -- where to draw a box around a fixed set of requirements and how to build services that will yield tangible ROI while keeping an SOA fully extensible.

For this article, we evaluated dozens of SOA implementations to find five that had a major impact on an enterprise and/or its partners: British Telecom, Countrywide Financial, Guardian Life Insurance, the New England Healthcare EDI Network and Transamerica Life Insurance. These projects are largely works in progress; some are only in their initial phases of implementation. But all can help light the way for enterprises in search of their own strategies to make a simple, powerful idea come to life.

Massachusetts takes a spoonful of SOA By Galen Gruman

Many organisations are looking to SOA to tie together systems within the enterprise or among partners. But few face the diversity and complexity that the state of Massachusetts did when it tried to connect independent insurer, hospital, and physician systems with one another -- and with the state's own systems for care, reimbursement, and billing.

"How do you craft enterprise-like functionality across hundreds of moving parts that don't interoperate with each other?" was the question the state faced in 1997, recalls Harvard Medical School CIO D John Halamka, who spearheaded the effort. Because the Health Insurance Portability and Accountability Act of 1998 required that every doctor, hospital and insurer be able to exchange data for transactions, doing nothing was not an option.

The state's major hospitals and insurers examined three options. The first was to deploy a common platform and to require insurers, hospitals and physicians who had business with the state to implement and use it. This option, however, was too complex to pursue seriously, Halamka says.

The second option was to create a unified database for patient medical, billing, and insurance data that participants could access using their own systems. That solution would have cost $US50 million, Halamka recalls. But the third option -- to implement an SOA that would provide the data and application translation necessary for various services to interoperate without changing their code or data structures -- was viable and ended up costing just $1 million.

What Halamka calls a "Napster for health care" has also reduced the cost per transaction from $5 to 25 cents. The system now handles approximately 9 million transactions per month. To manage this network, a group of medical associations and insurers set up a nonprofit organisation called the New England Healthcare EDI Network (NEHEN). Funded by the hospitals and insurers, it has one common program management officer and an annual budget of $3 million.

The result is a "closed-loop system" that ensures accurate data and validates procedures, coverage, and billing up front, thereby reducing management costs for all participants across the state. For example, "insurance companies save money by not having [to hire as much] staff to deny claims", Halamka says.

Architecturally, the NEHEN system leaves data structures and applications alone, even if they are fragmented in different locations or in different systems. "The big win is in not having to rewrite old code," Halamka says, noting that some systems date from the 1970s. The system does, however, provide a central exchange that translates data structures from one system's format and standards to another's, and it maps specific transaction services from one system to another, aggregating multiple service requests and using multiple databases when necessary.

"The data and the services are very disassociated," Halamka notes. "But it doesn't matter whether they all sit in one place, as long as the doctor gets it all in a timely fashion."

From a tactical perspective, as long as the system providing the data or service can communicate through TCP/IP, "that's all I need", Halamka says, noting that most of the Web services in the network were developed using Microsoft .Net, with the gateways written in Visual C++ and deployed on IIS.

Strategically, the keys have been to define the business processes along with the architecture, to understand what the data means in its various repositories, and to know what applications provide what services so that middleware can be configured to make the appropriate calls and translations. "I have the ability to control the business logic without having to modify the underlying application," Halamka says. "The middleware approach is very non-intrusive to the [individual organisation's] IT agenda."

For example, patient IDs vary widely from institution to institution. Rather than require a common identifier for each patient, the NEHEN system uses a probabilistic service to check a variety of attributes -- name, nicknames (such as Johnny and Jack for John), postcode, gender, Social Security number, insurer, and physician -- and then maps patients' identities across systems. In the event that a new identifier class, such as employer ID, is used in identifying patients, the service can easily be modified to account for that, Halamka says. None of the other systems is affected or even aware of the change to the middleware, although system owners can decide whether they want their systems to use this new identifier class.

Because SOAP had not yet been developed and XML was not widely deployed, the NEHEN system initially used HTML as the common vehicle for data exchange. "In the pre-XML era, we had to use the Web for content rather than the semantic Web [XML] for data. So we used simple server-side COM components to fetch HTML pages from various hospitals and display them in a unified clinical viewer that we built," recalls Halamka, who wrote the code for this system. "It was not elegant since we had little control over the look and feel for the HTML content returned from each hospital system, but it worked. Today, with XML and XSLT [XSL Transformation] we can treat the content as data and format it as we like."

To ease adoption, NEHEN provides a Windows XP and Windows Server 2003 Web services suite that organisations can deploy to gain the required connectivity. For individual doctors, NEHEN provides a Web application that they can access either directly or through standard medical management applications, which vendors have modified to support the NEHEN system. In both cases, the underlying SOAP layer handles the communication to billing systems, medical records systems, and so forth. NEHEN may not be the answer to health care's ills, but it's eliminating lots of wasted motion in the Massachusetts system.

Countrywide Financial simplifies lending

A fast-growing lending company with widely distributed IT shops abstracts applications as services, links them via messaging bus, and gains new business agility. By Galen Gruman

For half a decade, Countrywide Financial has seen its loan, insurance, and banking services businesses grow dramatically -- and its IT systems increase in complexity -- as customers, products, and markets have multiplied. To meet this increase in demand, Countrywide decided to embrace a flexible SOA approach, the long-range goals for which are a familiar refrain in enterprise IT: decrease complexity, improve scalability, and reduce overhead.

Countrywide is divided into separate business units, each of which employs an IT staff that operates fairly autonomously. One unit, Countrywide Servicing Systems Development (CSSD), which primarily supports the company's loan division, began its SOA effort in 2002.

According to Peter Presland-Byrne, senior vice president of application development at CSSD, the unit chose the SOA approach because "applications support a business problem and so follow certain patterns" that lend themselves to two key attributes of an SOA: functional abstraction based on services and an emphasis on reusable components to provide those basic services. "We're trying to look at the construct of the business model from a services perspective," he says.

As it began implementing an SOA, CSSD quickly discovered that many applications had embedded within them services that duplicated functions in other applications.

"We needed to abstract the services, which is an ongoing process," and to decide which ones to choose when there were duplicates, Presland-Byrne says. He anticipates the need to abstract services further in order to support Web services because such support "doesn't come naturally" in an IBM iSeries midrange server environment, which is what CSSD uses.

Deriving core services and having applications access common ones rather than implement their own is a key part of the SOA approach and requires a development culture that focuses on reuse, Presland-Byrne notes. To encourage adherence to the SOA, Countrywide reviews new software development to ensure it fits the SOA, provides consistent interoperability and reuses existing services where possible. Countrywide originally looked at SOA as its central goal but later realised the central goal was reuse, which an SOA promotes. "If you truly support reuse, then you make SOA possible," Presland-Byrne says.

Countrywide also decided to use a messaging system as the connectivity mechanism between applications and data sources. Because Countrywide's enterprise uses several technologies, including Java and Microsoft .Net, "messaging had to be agnostic" to ensure no proprietary dependencies were introduced into the system, Presland-Byrne says. Countrywide also relies heavily on IBM's MQSeries and WebSphere MQ Integrator middleware for messaging and service handling, as well as Flashline's development environment for managing services and software components.

Although the messaging system is standardised across CSSD's applications, Countrywide does not require consistent data models. Instead, it uses middleware to ensure a consistent information flow, mapping and translating data formats as needed. For CSSD, imposing a consistent data model was believed to be unrealistic given that "the minute you bring in a third-party tool you lose the consistent data model anyway", Presland-Byrne says. "The middleware we brought in could translate these different standards. That's what integration tools are for."

More importantly, your "buffer" middleware -- which contains the translation of business logic and data formats between services -- must be kept separate from the service logic, Presland-Byrne says. Doing so allows separate applications to access the same service concurrently, without requiring you to touch the service code as the applications or data change. Plus, it allows you to run old and new versions of services simultaneously, either during a transition period or for different application needs. In both cases, IT can leave the services untouched.

Given that most of Countrywide's services are internal, the company does not rely heavily on Web services or associated technologies such as SOAP, although it does use Web services for a few applications accessed by customers and field agents. Countrywide has, however, tended to use XML as the semantic data standard for services and middleware because of its easy fit and wide popularity, Presland-Byrne notes. As its lines of business deploy SOAs, Countrywide is now examining how it can extend the approach to communication among units. That will require re-examining services and eliminating duplication, Presland-Byrne acknowledges. The company has already begun consolidating identity into one service that can be accessed via SSO (single sign-on) across the enterprise.

Because each business unit was faced with different growth patterns and technology lifecycles, implementing a company-wide SOA in one big bang was not possible in 2002. Now that each line of business has adopted the concept and has achieved similar levels of technology maturity, extending the architecture more broadly is something "we can now tackle", Presland-Byrne says.

SOA ensures Guardian gets it right By Galen Gruman

Five years ago, Guardian Life Insurance decided to rethink the basic structure of its application silos, which had been developed with little attention to business goals, says Jaime Sguerra, chief architect at Guardian. "There was no standard way to build or connect applications, or any habit of reusing code," he recalls.

A new IT management team decided to change that, mainly to make application development faster, more nimble, and better aligned with business priorities.

"We wanted to stay away from the one-off application and instead provide a single, common service wherever possible to reduce overall complexity. A service architecture is the way to make disparate technologies work together," Sguerra says, adding that, with an SOA in place, IT can focus on developing new applications, not reworking old ones.

"Our philosophy is reuse. There's a ton of money invested in the legacy technology, and we wouldn't be able to justify a business case just for modernization."

Sguerra estimates that the SOA approach has saved approximately 30 per cent of the application-development budget. After 28 months, about 60 services used by three key systems -- benefits plan administration, claims processing, and policyholder administration -- are now in place, as is the basic communications infrastructure. Of those services, about 50 are used by all three systems. And the work continues: Guardian plans to create 22 more services for those systems and then bring its other systems into the SOA model, Sguerra says.

At the heart of Guardian's SOA is its enterprise service manager, a collection of J2EE workflow and connector middleware tools and an IBM CICS/MQSeries message bus for managing requests. Requests come from one of three client systems -- a Web portal used by customers and independent agents, a CRM system, and an interactive phone system used by customers -- or from applications themselves. The enterprise service manager decides what services to invoke, in what order, and what data resources are needed. It then queues up the services and manages their interaction. At the end of the transaction, the client receives the requested result or an error message.

Before the SOA was implemented, "users needed a checklist of all the system to run" for each task; "now, that workflow is built into the enterprise service manager", Sguerra says.

"We chose a central enterprise service manager because it was the best way to gain reuse," Sguerra says. Although it may make sense to have a decentralized architecture where service logic resides in multiple locations so that services communicate directly, that approach increases the risk of ending up with multiple versions of the same service as development gets out of sync across the systems, he adds.

Because independent agents use the claims, policyholder, and benefits systems as well, Guardian chose to implement its SOA through Web services. "It's harder to deploy applications to someone else's computer," Sguerra says. Developing a Web-based interface proved crucial in serving internal and external users with a single communication system.

The SOA uses Web-based intermediaries in an IBM WebSphere server environment as the framework for translating and massaging data and procedure calls as needed between applications and data sources, as well as SOAP for messaging. Although services reside in a variety of physical locations -- within mainframe applications, as discrete component services on other application servers, as part of a modern application, or as an external service -- the Guardian SOA groups them logically by system or as a shared service.

Because Guardian uses a large number of mainframe applications, its IT team faced a challenge in exposing all those applications so they could be used as services, Sguerra notes. Guardian uses WSTL (Web Service Transaction Language) in most cases to translate service requests among services, but in some cases, mainframe applications don't support that. Rather than rewrite the mainframe apps to accommodate WSTL, Guardian uses EJBs to perform the translation outside the application. In the application itself, "we just open a door to see the EJB", he says.

Sguerra emphasizes that developing applications as part of an SOA requires a new way of thinking about application development. "You need to know up front that it is a cultural change. You can't go into it pretending it's not going to be a challenge. It takes a lot of coordination," he says, adding that business units and application developers oftentimes resist sacrificing custom interfaces that prevent services from supporting multiple applications.

Such objections usually disappear when the efficiency benefits have been proved, Sguerra adds. When a custom interface or function is truly needed, developers can usually supply a separate service that relies on a common service for the remaining functionality. There's always a trade-off, but Guardian is discovering its happy medium.

British Telecom dials into SOA By Leon Erlanger

Telecom providers are competing tooth and nail to provide consumer and business customers with the latest and greatest value-added services. This smorgasbord of offerings includes everything from ring-tone downloads to hosted messaging, accounting and other business services. An SOA makes perfect sense in this have-it-your-way environment because it enables providers to cobble together new offerings with those of third parties and integrate them quickly with their internal, mainframe-based billing, provisioning, and other support systems.

That's exactly the approach British Telecom (BT) wanted to take in serving its SMB broadband customers.

"SMB customers come to us for broadband access first," says Norman Street, head of Internet applications and technology at BT Retail. "But the scenario is that as they become more savvy and the Internet becomes more integral to their business they'll eventually start moving business processes online in an ASP model."

The hosted scenario is especially appropriate for businesses with fewer than 100 employees because oftentimes these organisations lack sufficient support services or suffer from understaffed IT departments. "We were looking to develop some of these service offerings in-house but also bundle them with third-party applications and mobile products," Street says.

To compete effectively, BT needed to be able to quickly test new services in the market. If a service proved profitable, it would have to scale quickly. If it didn't, BT had to be able to decommission the service and replace it with another without a lot of integration effort. BT was also looking to allow customers to manage their own subscribed services online through a Web-based interface.

It was clear that BT's current integration model couldn't support such a fast-paced scenario. "We were using a typical spoke model for integrating services with our back-end systems. We really needed to reduce the time and cost of integration and become a lot more agile," Street says.

After looking into several alternatives, BT quickly concluded that it needed an SOA. BEA Systems had supplied BT with integration technologies in the past, but for this project, BT decided to go with Microsoft's CSF (Connected Services Framework), an SOA-based service-delivery platform that functions as an extension of Microsoft BizTalk Server, SQL Server and Windows Server 2003.

CSF provides tools and components geared specifically to the needs of service providers looking to bundle services for a variety of devices, such as PCs, PDAs, and mobile phones, and to quickly plug them in to their back-end business and operational support systems. These include a number of adapters that hook into existing BSS (business support system) and OSS (operation support system) applications and expose them as Web services.

It also includes a UDDI/WDSL service directory and tools and standards for defining quality of service, managing identities using Active Directory and Microsoft Identity Integration Server 2003, and managing other service deployment and delivery functions.

"BizTalk Server provides a workflow engine and templates to configure the business logic," Street says. It also handles message flows and other integration functions. SQL Server holds the user and product data. And of course, it all runs on Windows 2003 server.

Although BEA provided a versatile set of tools, Street and others at BT liked the idea that Microsoft had a platform specifically targeted to service providers. "Microsoft was very conscious of the advantages of a complete platform approach," Street says. "With CSF we got much more out of the box, which would take away some integration steps."

BT was also aware that Microsoft had the kind of applications SMB users would be interested in. "If we were going to be selling those applications, it made sense to get the integration technology from the same source," Street says. "At the same time, CSF had well-defined Web service interfaces and the open standards to integrate with any application." And Street was aware that Microsoft was providing tools for and encouraging .Net developers to develop to CSF. "We felt that there would be applications from third-party .Net developers that would undoubtedly be useful for our small and medium-sized business customers," he says.

CSF's standard Web service interfaces would make it easy to plug in third-party applications, build composite applications, and tie it all into their back-end systems. Introducing and retiring services would mean simple changes to the CSF interface as opposed to building or unravelling multiple layers of custom integration.

Next summer BT plans to introduce its first set of hosted messaging services based on Microsoft's Solution for Hosted Messaging and Collaboration, which provides an adapter for CSF. Future plans include bundling third-party applications, possibly migrating some of BT's existing applications off its legacy platforms to integrate with CSF, and making BT Retail's CSF-based services available to other parts of BT's business. "CSF would make it relatively simple to, for example, provision a mailbox and then bundle it with a mobile service," Street says. As usual, when an SOA is operational, there's no shortage of ideas to expand it.

Transamerica turns silos into services By Leon Erlanger

One of the real promises of SOA is enabling companies to leverage existing legacy systems as a set of core, reusable Web service building blocks that can be assembled to create new processes and applications quickly and inexpensively. That's just what Transamerica Life Insurance was looking for when it sought to provide its business partners with self-service access in real time.

"We exchange a lot of data with our different distributors outside the firewall," says Jeff Gleason, director of IT strategies at Transamerica's annuity products and services division. "A lot of that was being done via flat-file batch data exchanges."

Gleason realised that to stay competitive, Transamerica would have to provide its business partners with real-time access to its numerous legacy back-end systems. That's a complex undertaking, however, for several reasons.

"We live in a very challenging legislative environment, with Sarbanes-Oxley, the Patriot Act, anti-laundering laws, tax laws and other types of controls," Gleason says. "As legislation and the competitive environment change, we need to be able to make changes to our internal systems quickly, including changing rules, the ways taxes are calculated, or the way a product functions given specific criteria.

At the same time, we often have to customise products and services for each of our different distribution channels. And sometimes we get requests from specific banks or broker dealers to create products for their particular niche markets or new areas they want to compete in. These things often impact our internal business processes."

To provide real-time access, Transamerica also needed ways to validate agents as licensed and appointed to sell specific products in specific states. "Validating an agent is not as simple as looking something up in a system," Gleason says. Depending on the commission structure there might be many different rules about how the commission hierarchy, which has up to 10 levels, is set up internally."

With all this complexity, Web services and SOA were natural choices for Transamerica. "We needed a solution that was both tightly integrated and loosely coupled," Gleason said. A lot of business logic exists within Transamerica's current back-office legacy systems. "Instead of continually recreating that logic, it made sense to create a set of core services to expose that logic so that it could be accessed by different applications, processes, and channels, whether they were batch processes, real-time processes over the Web, internal fat-client applications, or even IVR [Interactive Voice Response] systems."

To support all these methods of access, each Web service would have to be capable of accommodating not only straight SOAP Web services calls but also MQSeries and JMS (Java Messaging Service). These core services could then be mixed and matched as part of a larger group of composite services that could accommodate the different needs of various channels and individual business partners.

SeeBeyond's ICAN (Integration Composite Application Network) provides the tools for exposing as Web services the back-end mainframe transactions that provided much of Transamerica's existing functionality. One such tool, eGate Integrator, is used to provide the integration broker and message transformation from one data format to another. Other SeeBeyond tools leverage BPM capabilities to create the "agent hub" that handles the complex message routing required.

So, for example, in response to a request from a user application, one of Transamerica's three or four legacy policy administration systems might return a cryptic product descriptor. That product descriptor could then be passed to a separate distributor support system that would return a more user-friendly product name. That or another distributor support system might also handle commissions and manage the information on which agents are appointed in which states to which products.

The end-user applications use an insurance industry XML schema developed by the nonprofit Association for Cooperative Operations Research and Development (ACORD) standards organisation. The XML data is then transformed into the proprietary format required by each legacy back-end application. Basically, this ensures the loose coupling essential to an SOA. "If we acquire another company and we need to validate agents against their distributor support system, it's simply a matter of creating an adapter from ACORD to the proprietary format required by that system. Everything on top speaks ACORD and doesn't care what the implementation of the service is behind the scenes," Gleason says.

SeeBeyond also provides the tool for creating portals and graphical portlets. The ultimate dream is to provide each business partner with a single custom application and interface to all the back-end systems necessary to fulfil each partner's specific needs.

Gleason advises those getting involved with SOA to do as much planning and preparation as possible. "If we had it to do over again, we'd spend a lot more time up front prototyping, testing, and setting up the architecture and standards. After all, you're creating one object and one service that will be used by lots of different processes. You have to make sure you don't make changes to the service that help one project but break others," he says.