• NATIONAL SALES MANAGER. WIRELESS...
• Systems and Network...
• Project Manager - Trading...
• NETWORK PROJECT MANAGER
• Technical Lead

Australian Customs - more flak than facts?

Peter Davidson, Information Age

14/02/2006 11:26:31

For four years, Australian Customs' massive re-engineering exercise loomed on the ICT horizon as one of the brightest lights in our technological firmament.

Australia's most ambitious e-government project promised to set a world benchmark in cargo management, but instead the general perception of its launch on October 12 last year was of thousands of tons of cargo stranded at wharves and airports.

Manufacturers and merchants could see revenues dissolving and Christmas trade diving into losses as a media-fuelled firestorm of flak was directed at Australian Customs and its CIO Murray Harrison.

Politicians of various hue from local to federal joined the outcry, hurling invective at Customs' tumbrel as it rumbled onward. A host of freight forwarders, agents, importers and exporters - anyone with the slightest interest in trade processes - joined in.

But how much of it was deserved?

In October 2004, Information Age spoke to Harrison about Customs' Cargo Management Re-engineering (CMR) program, and its cornerstone, the Integrated Cargo System (ICS). He spoke to us again as 2005 drew to a close.

According to Harrison, the picture painted in the media that the whole thing was a disaster was based on the difficulties undoubtedly faced by a large number of parties involved in the import process.

However, the system did not, as suggested, collapse on its first day.

"Customs would not deny that there were delays in cargo movement emerging early after cut-over, nor that diversion of users to Customs Interactive did not place a much higher load than would be expected in an ongoing business situation.

"We had to move quickly to address both of those issues. However, if you believed everything you heard in the media, then the ICS had failed completely and nothing was getting off the docks or out of air cargo depots.

"The reality is that, in spite of difficulties experienced by users, the system successfully processed more than 50,000 messages on its first day," he says.

While some ICT issues emerged in the first two weeks as the system came under operating load, its overall integrity was not in doubt.

Harrison candidly accepts that some users were not ready for the new system, in spite of the efforts of all concerned to ensure that they would be.

"I am aware of some cases where companies were only receiving their own software to link to the ICS in the week leading up to the changeover. We do recognise that a lot of people had a lot of difficulties - there's no question about that."

The introduction of the new integrated system (ICS) meant turning Customs procedures on their head (see box). Harrison describes it as the "biggest e-government project ever undertaken here and the greatest change to Customs since Federation".

The system is complex, was four years in the making and thousands of transport companies, freight forwarders, customs agents, importers, exporters and others are required by law to use it.

Design detail in the 19,000 pages of analysis for ICS includes 800 screens, 16,000 business rules, 70 complex business messages, 850 database tables, 3700 executable load modules, 1800 CICS transaction types, 55 batch jobs, 90 reports and 35 system interfaces.

It replaces a decades-old legacy system called COMPILE, primarily designed to handle seaborne imports via electronic data interchange (EDI) channels. The system did not allow the comprehensive risk assessment and border security that the Government demanded, and was nearing the end of its useful life.

The export component of ICS went live in October 2004, a year ahead of the far more complex import component.

As the new system evolved, Australian Customs' Web site posted hundreds of pages of instructions on individual systems, customer registration and e-learning.

The external software element was also crucial. Customs regularly met with and supported parties developing software, both in-house and as third-party vendors for the cargo industry as there was a clear recognition that the industry was dependent upon delivery of software and applications, other than the ICS, for the October 12 launch.

Throughout all of this engagement, Customs stressed that the system required total accuracy in data input and reference files were (and still are) being provided to these vendors on a regular basis to facilitate this requirement.

No more the fuzzy information sufficient to drive COMPILE, ICS distributed data to a host of agencies outside of Customs and a misplaced digit in an electronic clearance request could send it to the end of the queue.

Bills of lading, air waybills, voyage numbers and other information had to be delivered strictly according to format, and data hygiene maintained to satisfy risk assessment and border security concerns. Make a mistake and start again.

"The system is pretty unforgiving," Harrison says, "but data has to go to many players in the chain, and security is important. We made that clear from the outset."

Testing of the system including the integration with externally provided software continued through 2005.

In the background, however, sectional interests were lobbying to have the new system either delayed, or better still, junked before it began.

The Customs Brokers and Forwarders Council of Australia (CBFCA) expressed grave concerns, with many unable to effectively test their processes because of software, hardware or other resourcing constraints.

Some of its members could see their businesses jeopardised because under ICS, importers could register directly with Customs, cutting out middlemen completely.

At a meeting in July last year, interested parties including Customs Minister Senator Chris Ellison gathered in Sydney to thrash out issues, and agreed to keep to the October launch with a number of caveats, mostly surrounding the end users' software.

At a September 26 meeting of 15 major software developers, both user and third-party, all generally agreed that the October date was achievable and that everything would be in place.

They included players like Qantas, DHL and Coles-Myer who, like Woolworths, had undertaken massive supply chain upgrades based on Customs' new system.

Three days after the meeting on September 26, an e-mail to all stakeholders confirmed October 12 as the date and reminded all and sundry that the required Public Key Infrastructure (PKI) digital certificates needed to be bought from VeriSign immediately to be ready to start.

It's show time At launch, things started to go awry as users, ready or not, struggled to get messages into the system.

Some had difficulty with ICS's accuracy demands, others were not familiar with the system (given the complexity associated with new screen layouts and the additional information required) - while others had not yet loaded the required external software.

Some users who were not getting the responses that they expected kept resending messages, creating bottlenecks. Others reverted to e-mails, faxes and phone calls to an expanded help desk where response times blew out from minutes to hours, compounding the problem.

Users who previously sent clearance requests by batched EDI (that is, most of them) but were encountering delays through software issues, turned to Customs Interactive, seeking to use the Web-based system instead. It too buckled under the load.

In the midst of the chaos, however, more than 50,000 messages did get through to be processed on the first day, but elsewhere imports started to pile up at air and sea ports.

Most of the significant difficulties with air freight were sorted out "within the first two weeks" largely because it is a more regulated system with widely accepted and used international data standards for cargo messaging.

At major seaports there were bigger problems. Customs responded by putting in place a series of contingency workarounds to ease congestion, dispatching teams to ports to handle clearances on site, and creating a new Web site listing the numbers of all cleared containers ready for pick-up.

By the second week of its operation, the ICS had handled more than two million messages and 30,000 containers were cleared.

Show stopper One of the significant problems that emerged came at the end of the first day: a user reported that a competitor's data was visible during an interactive session.

"This was a serious problem. During full-load tests, nothing like it had shown up previously," Harrison says.

It transpired that a user's query to the connection manager was designed to stay open for 30 seconds, and this allowed a leak between that and another concurrent session. Reducing the query parameters to zero seconds initially appeared to stop the problem.

"We were on the phone very quickly to the software vendors in the UK and Bahrain. They supported our technical team in tackling the problem urgently.

"We thought we had it nailed. Then a week later it happened twice more, which had us stumped. The vendor's engineers arrived really quickly from the US and we worked through the night to create a trap for the flaw so we could log the sequence.

"In the meantime, the vendors polled their international user base but no one had struck the problem under the same configuration as ours. They created a universal fix for it; the problem has not reappeared."

Record clearances Despite initial problems, log-jams at ports began to ease by the end of the second week and containers were being cleared faster than they were being collected.

NSW Ports Minister Eric Roozendaal reportedly predicted on October 25 that Sydney's Port Botany was at "100 per cent capacity" and would grind to a halt within 24 hours, with ships queuing off the coast the day after.

Stevedores P&O said that no ship would be turned away.

"In fact, in October last year, the Sydney Ports Corporation shifted a record number of containers through Port Botany, up 9 per cent on the same month in 2004," Harrison says.

Elsewhere, media reports continued to surface that unanswered messages sent to ICS were lost, that containers of perishable goods were left in the sun, that Customs mainframe had insufficient capacity, and that the Government was having to bail out Customs with an extra $100 million. All were refuted.

Harrison says, "It's about perception. From our end, consider that of the more than 16,000 business rules that ICS manages, there were probably problems with fewer than 200 them so the fail rate from this perspective was small.

"But from the perspective of a user who is having problems with any one of these rules, the system did not function and it made life very difficult. I can understand their frustrations.

"We had more than 10,000 people through information sessions for the new system, and it was up and running for user testing more than 12 months before it went live.

"With 23,000 function points, this system and the associated business processes are extremely complex and the range and number of stakeholders is large.

"Customs recognises that for border security, the data quality demands in ICS are onerous. Perhaps we could have made things easier while still meeting our security and risk management obligations.

"I think it is important to recognise that despite the initial difficulties, we were able to work through these problems Some aspects of the new system are still requiring adjustment .

"Customs has been working with industry in relation to the balance between the stringency of reporting requirements and industry business imperative to keep cargo moving quickly. A lot of credit therefore needs to go to the hundreds of staff from Customs, our contracting partners, the industry itself and staff from other Government agencies, who came together during a pretty tough time."

[sidebar] The path to integration

The decision to give the Australian Customs Service's systems a major makeover followed a 1996 review of its infrastructure, and an assessment of the capability to handle growing processing volumes. At the same time EDS was awarded a wide-ranging contract to support ICT operations.

Essentially, the re-engineering project was needed to create a secure Web-based "single face of government" for players in the import/export supply chain to cope with annual trade processing volumes of 3 million import entries, 1.2 million export clearances, 4 million containers and 100,000 flight movements, and the collection of over $5 billion in duties.

(It is a world first in this field; its nearest equivalent, the American ACE system, is so far estimated to have cost more than $US3bn and is still far from complete after five years' development. Its creators will look soon at the Australian system.)

Customs went to tender in 2001 with Computer Associates' consortium of Kaz, IOCORE and others, successfully bidding for the core cargo reporting and management system.

But first, some abbreviations: the overall project is called Customs Management Re-engineering (CMR), with the Integrated Cargo System (ICS) at its core. Customs Connect Facility (CCF) is the gateway to Customs' business apps, accessible by Customs Interactive (CI) using Web services or by batch mode EDI. An IBM-led group developed CCF to allow Web-based access via Customs to industry and other government agencies - quarantine, statistics, foreign affairs and trade for example, about a dozen in all.

CIO Murray Harrison, who joined Customs from Veterans' Affairs in 2002, bridles at commentators who describe the CMR program as a decade-long project: "It started pretty well from scratch at the beginning of 2002. The CA-led project cost about $50m when fully scoped, CCF the same and the five-year cost of embedding the system another $100m - GST fully paid".

Integrated Cargo System (ICS) The corner-stone of CMR, ICS is an integrated system giving enhanced risk assessment at the border and allowing more efficient cargo tracking. Its software suite has 23,000 function points.

It operates on an IBM OS390 mainframe running z/OS with transactions in a CICS environment with DB2 database management. MQ-series provides the mainframe interfaces with the CCF gateway and other business applications.

Customs' Web-based user interface, Customs Interactive (CI), has a WebSphere Java application server front end. CI system software is hosted on infrastructure managed as part of the CCF gateway.

Transaction application code to support the cargo management business rules for both EDI and CI channels was developed in the AdvantageGen/CoolGen environment.

ICS's transaction and event processing architectures create and manage events to prioritise and balance message loads across the system to maintain throughput, with automatic exception and recovery management.

Customs Connect Facility (CCF)

CCF is the gateway to Customs' business applications. Importers, exports and brokers can transact via an interactive mode (Customs Interactive) using industry standard Web services or with batch mode EDI.

A data transformation facility translates Customs and industry-agreed standards for data exchanges (eg UN/EDIFACT) to Customs' application requirements, significantly reducing customers' previous need to use a plethora of data formats.

It also allows Customs staff to track messages through the CCF.

Communication channel management and CI runs on Sun Solaris Unix platforms and Cisco routers, with validation and transformation processed on IBM P- and SP-series Unix platforms and Wintel servers running IBM AIX, Win2K, DB2 , WebSphere, Tivoli WebSeal and Baltimore's FormSecure.

Overall, the CMR architecture was designed to be multi-tiered, highly available, scalable and to have shared security components with common code bases (for services such as authentication and authorisation).

The CCF solution has its origins in the IBM e-business infrastructure reference architecture with J2EE, WS-Security (SAML), XML, UN/EDIFACT D99B and LDAP.