• IT and Projects Manager -...
• CHANNEL ACCOUNT MANAGER $ 210K...
• MANAGER STRATEGIC INDUSTRY...
• IT Engineer - Canberra
• Project Manager - Trading...

Adaptive business intelligence

Matthew Michalewicz, Information Age

15/02/2007 11:29:08

Since the computer age dawned on mankind, one of the most important areas in information technology has been that of "decision support". Today, this area is more important than ever. Working in dynamic and ever-changing environments, modern-day managers are responsible for an assortment of far-reaching decisions: Should the company increase or decrease its workforce? Enter new markets? Develop new products? Invest in research and development?

The list goes on. But despite the inherent complexity of these issues and the ever-increasing load of information that business managers must deal with, these decisions boil down to two fundamental questions:

• What is likely to happen in the future? • What is the best decision right now?

Whether we realise it or not, these two questions pervade our everyday lives - both on a personal and professional level. When driving to work, for instance, we have to make a traffic prediction before we can choose the quickest driving route. At work, we need to predict the demand for our product before we can decide how much to produce. And before investing in a foreign market, we need to predict future exchange rates and economic variables.

It seems that regardless of the decision being made or its complexity, we first need to make a prediction of what is likely to happen in the future, and then make the best decision based on that prediction. This fundamental process underpins the basic premise of Adaptive Business Analysis (ABI). Simply put, ABI is the discipline of combining prediction, optimisation, and adaptability into a system capable of answering these two fundamental questions: What is likely to happen in the future? and What is the best decision right now?

To build such a system, we first need to understand the methods and techniques that enable prediction, optimisation, and adaptability.

At first blush, this subject matter is nothing new, as hundreds of books have already been written on business intelligence, data mining and prediction methods, optimisation techniques, and so forth. However, none has explained how to combine these various technologies into a software system that is capable of predicting, optimising, and adapting; this text is the first on the subject.

When we set out to write ABI, we had three important objectives in mind: First of all, we wanted to explain why the future of the business intelligence industry lies in systems that can make decisions, rather than tools that produce detailed reports. As most business managers now realise, there is a world of difference between having good knowledge and detailed reports, and making smart decisions.

Michael Kahn, a technology reporter for Reuters in San Francisco, makes a valid point in the January 16, 2006 story entitled "Business intelligence software looks to future":

"But analysts say applications that actually answer questions rather than just present mounds of data is the key driver of a market set to grow 10 per cent in 2006 or about twice the rate of the business software industry in general.

'Increasingly you are seeing applications being developed that will result in some sort of action,' said Brendan Barnacle, an analyst at Pacific Crest Equities. 'It is a relatively small part now, but it is clearly where the future is. That is the next stage of business intelligence.'"

We could not agree more.

Second, we wanted to explain the principles behind many prediction methods and optimisation techniques in simple terms, so that any business manager could grasp them. Even though most business managers have a limited technology background, they should not be intimidated by terms such as "artificial neural networks", "fuzzy logic", "evolutionary algorithms", "ant systems", or "agent-based modelling".

They should understand the strengths and weaknesses of these methods and techniques, their operating principles, and applicability. Armed with such knowledge, business managers will be in a better position to control the application of these methods and techniques in their respective organisations.

And, third, we wanted to underscore the enormous applicability of ABI to many real-world business problems, ranging from demand forecasting and scheduling, to fraud detection and investment strategies. From a high-level perspective, most of these business problems have similar characteristics, and the application of ABI can provide significant benefits and savings.

To facilitate the discussion in this book, we have divided the chapters into three parts that correspond to the three objectives listed above. In Part I, we present the fundamental ideas behind ABI, and explain the different roles that prediction, optimisation, and adaptability play in producing near-optimal decisions. We also discuss the characteristics that many business problems have in common, and why these characteristics increase the complexity of the problem-solving exercise.

Furthermore, we introduce a particular distribution problem that is used throughout the text as a running example. Given that the prediction and optimisation issues in this example are common to most business problems, it should be relatively easy for the reader to extrapolate this example to many other business domains.

Because countless texts have already been written on the subject of database technologies, data warehousing, online analytical processing, reporting, and the like, we saw little point in rehashing the tools and techniques that are routinely used to access, view, and manipulate organisational data.

Instead, Part II of the book discusses the various prediction methods and optimisation techniques that can be used to develop an ABI system. The distribution example is continued throughout these chapters, effectively highlighting the strengths and weaknesses of each method and technique. Each chapter in Part II is concluded by a Recommended Reading section that provides suggestions for readers who want to learn more about particular methods or techniques.

Part III begins with a chapter on hybrid systems and adaptability, explaining how to "combine" the various methods and techniques discussed in Part II, and how the component of adaptability can be added to the final design. In the remaining chapters of the book, we discuss the definitive solution to the distribution problem that was used throughout the text, as well as the application of ABI to several other complex business problems.

"The answer to my problem is hidden in my data... but I cannot dig it up."

This statement has been around for years as business managers gathered and stored massive amounts of data in the belief that they contain some valuable insight. But business managers eventually discovered that raw data are rarely of any benefit, and that their real value depends on an organisation's ability to analyse them. Hence, the need emerged for software systems capable of retrieving, summarising, and interpreting data for end users.

This need fuelled the emergence of hundreds of business intelligence companies that specialised in providing software systems and services for extracting knowledge from raw data. These software systems would analyse a company's operational data and provide knowledge in the form of tables, graphs, pies, charts, and other statistics.

For example, a business intelligence report may state that 57 per cent of customers are between the ages of 40 and 50, or that product X sells much better in Florida than in Georgia.

Consequently, the general goal of most business intelligence systems was to: (1) access data from a variety of different sources; (2) transform these data into information, and then into knowledge; and (3) provide an easy-to-use graphical interface to display this knowledge.

In other words, a business intelligence system was responsible for collecting and digesting data, and presenting knowledge in a friendly way (thus enhancing the end user's ability to make good decisions). The following diagram illustrates the processes that underpin a traditional business intelligence system:

Although different texts have illustrated the relationship between data and knowledge in different ways, the distinction between data, information, and knowledge is quite clear:

• Data are collected on a daily basis in the form of bits, numbers, symbols, and "objects".

• Information is "organised data", which are preprocessed, cleaned, arranged into structures, and stripped of redundancy.

• Knowledge is "integrated information", which includes facts and relationships that have been perceived, discovered, or learned.

Because knowledge is such an essential component of any decision-making process (knowledge is power), many businesses have viewed knowledge as the final objective. But it seems that knowledge is no longer enough.

A business may "know" a lot about its customers - it may have hundreds of charts and graphs that organise its customers by age, preferences, geographical location, and sales history - but management may still be unsure of what decision to make.

And herein lies the difference between "decision support" and "decision making": All the knowledge in the world will not guarantee the right or best decision.

Moreover, recent research in psychology indicates that widely held beliefs can actually hamper the decision-making process. For example, common beliefs like "the more knowledge we have, the better our decisions will be", or "we can distinguish between useful and irrelevant knowledge", are not supported by empirical evidence. Having more knowledge merely increases our confidence, but it does not improve the accuracy of our decisions.

Similarly, people supplied with "good" and "bad" knowledge often have trouble distinguishing between the two, proving that irrelevant knowledge decreases our decision-making effectiveness. Today, most business managers realise that a gap exists between having the right knowledge and making the right decision. Because this gap affects management's ability to answer fundamental business questions (such as "What should be done to increase profits? Reduce costs? Or increase market share?"), the future of business intelligence lies in systems that can provide answers and recommendations, rather than mounds of knowledge in the form of reports.

The future of business intelligence lies in systems that can make decisions. As a result, there is a new trend emerging in the marketplace called ABI. In addition to performing the role of traditional business intelligence (transforming data into knowledge), ABI also includes the decision-making process, which is based on prediction and optimisation:

While business intelligence is often defined as "a broad category of application programs and technologies for gathering, storing, analysing, and providing access to data", the term ABI can be defined as "the discipline of using prediction and optimisation techniques to build self-learning 'decisioning' systems" (as the above diagram shows). ABI systems include elements of data mining, predictive modelling, forecasting, optimisation, and adaptability, and are used by business managers to make better decisions.

This relatively new approach to business intelligence is capable of recommending the best course of action (based on past data), but it does so in a very special way: An ABI system incorporates prediction and optimisation modules to recommend near-optimal decisions, and an "adaptability module" for improving future recommendations. Such systems can help business managers make decisions that increase efficiency, productivity, and competitiveness.

Furthermore, the importance of adaptability cannot be over-emphasised. After all, what is the point of using a software system that produces sub-par schedules, inaccurate demand forecasts, and inferior logistic plans, time after time? Would it not be wonderful to use a software system that could adapt to changes in the marketplace? A software system that could improve with time?

The concept of adaptability is certainly gaining popularity, and not just in the software sector. Adaptability has already been introduced in everything from automatic car transmissions (which adapt their gear-change patterns to a driver's driving style), to running shoes (which adapt their cushioning level to a runner's size and stride), to Internet search engines (which adapt their search results to a user's preferences and prior search history).

These products are very appealing for individual consumers, because, despite their mass production, they are capable of adapting to the preferences of each unique owner after some period of time.

The growing popularity of adaptability is also underscored by a recent publication of the US Department of Defense. This lists 19 important research topics for the next decade and many of them include the term "adaptive": Adaptive Coordinated Control in the Multi-agent 3D Dynamic Battlefield, Control for Adaptive and Cooperative Systems, Adaptive System Interoperability, Adaptive Materials for Energy-Absorbing Structures, and Complex Adaptive Networks for Cooperative Control.

For sure, adaptability is here to stay. It is a vital component of any intelligent system, as it is hard to argue that a system is "intelligent" if it does not have the capacity to adapt. Moreover, modern definitions of natural and artificial intelligence include the term "adaptive".

For humans, the importance of adaptability is obvious: our ability to adapt was a key element in the evolutionary process. In the case of artificial intelligence, consider a chess program capable of beating the world chess champion:

Should we call this program intelligent? Probably not. We can attribute the program's performance to its ability to evaluate the current board situation against a multitude of possible "future boards" before selecting the best move. However, because the program cannot learn or adapt to new rules, the program will lose its effectiveness if the rules of the game are changed or modified. Consequently, because the program is incapable of learning or adapting to new rules, the program is not intelligent.

The same holds true for any expert system. No one questions the usefulness of expert systems in some environments (which are usually well defined and static), but expert systems that are incapable of learning and adapting should not be called "intelligent" -- some expert knowledge was programmed in, that is all.

It is not surprising that the fundamental components of ABI are already emerging in other areas of business. For example, the Six Sigma methodology is a great example of a well-structured, data-driven methodology for eliminating defects, waste, and quality-control problems in many industries. This methodology recommends the following sequence of steps:

Note that the above sequence is very close "in spirit" to part of the previous diagram, as it describes (in more detail) the adaptability control loop. Clearly, we have to "measure," "analyse", and "improve", as we operate in a dynamic environment, so the process of improvement is continuous. The SAS Institute proposes another methodology, which is more oriented towards data mining activities. Their methodology recommends the following sequence of steps:

Again, note that the above sequence is very close to another part of our diagram, as it describes (in more detail) the transformation from data to knowledge. It is not surprising that businesses are placing considerable emphasis on these areas, because better decisions usually translate into better financial performance. And better financial performance is what ABI is all about.

Systems based on ABI aim at solving real-world business problems that have complex constraints, are set in time-changing environments, have several (possibly conflicting) objectives, and where the number of possible solutions is too large to enumerate. Solving these problems requires a system that incorporates modules for prediction, optimisation, and adaptability.

Footnotes

We would like to thank everyone who made this book possible, and who took the time to share their thoughts and comments on the subject of ABI. In particular, we would like to thank the most famous fictional detective of all time, Sherlock Holmes, for providing us with the entertaining quotes at the beginning of each chapter.

Holmes remains one of the most famous problem-solvers of all time, and his methodology is based on prediction ("It is a capital mistake to theorise before you have all the evidence"), optimisation (" I had best proceed on my own lines, and then clear the whole matter up once and for all"), and adaptability ("I have devised seven separate explanations  But which of these is correct can only be determined by the fresh information, which we shall no doubt find waiting for us").

His methodology bears a striking resemblance to ABI.

Note that business intelligence can be defined both as a "state" (a report that contains knowledge) and a "process" (software responsible for converting data into knowledge).

About the authors of "Adaptive Business Intelligence", Springer, $US49.95: Matthew Michalewicz is CEO of Adelaide-based SolveIT Software and serial entrepreneur. Prof Zbigniew Michalewicz is Chair Professor of Artificial Intelligence at the University of Adelaide and a world authority on genetic algorithms. Dr Martin Schmidt is an expert in the commercialisation of software solutions that are based on evolutionary algorithms, simulated annealing, large-scale simulations, artificial neural networks, fuzzy logic and hybrid systems. He is also an Adjunct at the University of Adelaide at the School of Computer Science. Constantin Chiriac is a co-founder of SolveIT Software, and holds masters' degrees in computer science and economics.

www.AdaptiveBusinessIntelligence.com.au