website paiken:publications

Data Reverse Engineering:
Slaying the Legacy Dragon
(ISBN 0-07-000748-9)

"Anyone charged with developing a migration strategy from one application environment to another will find this book useful." - Terry Moriarty

by Peter Aiken is available now direct from a number of online bookstores including Amazon.com and McGraw-Hill (1-800-822-8158) and at neighborhood book stores. (If you are having trouble finding a copy, click here to have ACSES perform an electronic search.) This is the first book describing the process of recovering data architectures from existing information systems and using it to develop a foundation for enterprise integration and other reengineering efforts.

The Necessity of Data Reverse Engineering

by Elliot J. Chikofsky e.chikofsky@computer.org

The competitive international environment of the mid-1990s demands that an organization make the best use it can of its resources. Resources that can contribute directly to the organization's bottom line are of significant value. A resource that can be utilized for strategic and tactical advantage is invaluable. In the information-intensive age we have just entered, data is that invaluable resource.

LEVERAGING DATA

Some organizations have been very successful at leveraging the use of data. American Airlines, which introduced the first frequent-flyer program in 1981, proved the value of capitalizing on data about repeat customers. Besides rewarding key clients for brand loyalty, well-managed frequent-flyer programs provide a wealth of information about preferences, buying habits, and sales of companion products and services. This yields both current and historic information, on the individuals and on the group, from which patterns can be discerned and trends can be detected. When tied with program partners, such as hotels and car-rental companies, the frequent-flyer data opens important opportunities for marketing of packaged services and related products. The same ability to leverage the use of data was demonstrated by the telephone company MCI when it introduced the first "Friends and Family" program in 1991. MCI capitalized on an understanding of its data and a computing situation that its principal competitor, AT&T, could not match at the time. MCI knew who its customers were because it handled the preparation of phone bills directly. MCI's customer records and telephone call logs were already being processed together. AT&T, on the other hand, billed its customers through the seven regional Bell operating companies and local phone companies. As a result of the Bell System divestiture, the local phone companies were intermediaries between AT&T; and its customers. Unlike MCI, AT&T; did not have the infrastructure in place to readily link customer identity with data on telephone calls.

With its existing billing operations and a firm knowledge of its available data, MCI had the mechanisms in place to detect and act on customer calling patterns. MCI turned this into a marketing coup by offering discounts when calls were made regularly between MCI customers. The result was a significant influx of new customers as existing customers became an adopted MCI sales force, encouraging their regular telephone partners to become MCI subscribers for the savings. This is a prime example of an organization capitalizing on an information technology-driven marketing opportunity.

These days, organizations adept at leveraging the use of data can be as near as the corner supermarket. In fact, they ARE the corner supermarket. With the introduction of bar codes and laser scanners, supermarkets sped up the grocery checkout process. In the same action, they revolutionized the grocery retail and wholesale business by providing a mother lode of data to be mined for tactical advantage. When your purchase is wrung up at the checkout stand, it is now recorded in the store's database. Besides allowing the store to monitor sales revenue and check inventories, the database itself is a product of the store. The data is of great value to the supermarket chain, wholesalers, distributors, and product manufacturers who readily purchase it. The supermarket data tells much more than how many of each product were sold. Simple analysis reveals facts such as: how often sets of products were bought at the same time; how the quantity bought relates to the total size of the shopper's purchase; or, whether the store's customers were affected by local advertising and discounts. Leverage of this data leads to revised marketing campaigns and targeted sales promotions, including coupons printed right at the cash register. A recent addition to the supermarket checkout as a tactical data source is the acceptance of credit cards, unheard-of for food purchases just a few years ago. Now, the customer identity is tied to the purchase. By combining data from appropriate sources, demographic information and purchasing profiles can become a potent source of information for direct target marketing. Already, it is not unheard-of for someone to purchase disposable diapers for a friend and then receive months of baby food offers and toy catalogs in the mail.

The ability to leverage data gives an organization advantages in many ways. Consider, for instance, customer service: having moved from Michigan to Massachusetts some years ago, my family missed a favorite Pillsbury product, orange danish rolls. We did not find them anywhere, though stores in the area carried other products of the same brand. Seeing a customer service address on one such package, I wrote to Pillsbury. Their response was a nice letter with a printout of every store within ten miles of my zip code that regularly received the item I was looking for. Here is an organization that knows its data and how to use it to make a customer very happy.

RECOGNIZING THE NEED

It is unfortunate that the success stories of leveraging data pale when compared to the untold failures and lack of effort across business and industry. Most organizations do not have a clue as to what their data is all about-never mind how to leverage it to best advantage. Without an understanding of the nature of their data, they have too much data and not enough information.

Data sources exist in an organization in many forms. Some are organized databases and files. Some kinds of data are locked in arcane structures in legacy systems whose designers are long retired. Other kinds of data are in quick little applications that someone threw together as a temporary fix five years ago and have been in use ever since. Then, there are the stealth data sources that the organization doesn't even know it has.

Today, organizations must change their information infrastructure quickly to satisfy rapidly changing needs. As the business climate changes, flexibility and response are key to taking advantage of new opportunities and meeting new goals. The rapid shift in hardware and software economics toward distributed, client/server, decentralized facilities has increased flexibility but also increased support challenges. It has also increased the need for a greater understanding of the data content and functional capability of the systems we are distributing access to. The data needs of the organization frequently require that information maintained in legacy systems be made available throughout the organization.

Yet, the data is only as good as the organization's knowledge of it. We have a vast store of potentially very useful information, already paid for by prior investment. If only we knew what was there and how to tap it. An organization needs to have the right data, know what it has, know where to find it, and know what it means. That is where Data Reverse Engineering comes it.

Reverse engineering is a process to achieve understanding of the structure and interrelationships of a subject system. It is a goal of reverse engineering to create representations that document the subject and facilitate our understanding it-what it is, how it works, and how it does not work. As a process, reverse engineering can be applied to each of the three principal aspects of a system: data, process, and control. Data Reverse Engineering concentrates on the data aspect of the system that is the organization. It is a collection of methods and tools to help an organization determine the structure, function, and meaning of its data. With that knowledge, the organization can develop meaningful plans to leverage its investment in data.

NOT AN OPTION

Data reverse engineering is not an option. For many organizations, it is a necessity. Understanding the data that drives the business is, more and more, becoming critical to success. An organization needs to be competitive. It must get new products and services out ahead of, and better than, its competition. Many of those products and services are information products, and the rest are information-dependent products. The race to capitalize on the Internet and the World Wide Web by beating competitors to customer-generating ser-vices is an information-intensive contest. Organizations that understand their data, and through it their marketplace, are destined to fare best in both short-term responsiveness and long-term innovation.

There is another very pressing reason why organizations need to embrace data reverse engineering now. It's called the millennium date problem. As an industry, we know for certain that our computer information systems are fundamentally flawed. Hundreds of thousands of computer programs worldwide, billions of lines of code, and an unimaginable number of entries in databases and data files were written to store date values with a two-digit year format. When we enter the year 2000, comparisons involving two digit year values will invert their sign or change direction (greater/less-than result). This will crash many computing systems, produce devastating errors in calculations, and could create chaos in both business and safety-critical systems from banking and insurance to communications and air traffic control.

To avoid the millennium date problem, an organization needs to locate and correct hidden date problems throughout its information systems, computer programs, and data files. Finding all of the myriad ways in which dates are stored, encoded, and embedded throughout the business requires an understanding of the nature of the organization's data and a hard look at the systems which process it. In this regard, millennium solutions and data reverse engineering are complementary and codependent. The millennium problem provides the impetus for data reverse engineering, while reverse engineering provides the methods and techniques to discover the extent of hidden date problems needing action before the deadline.

The future belongs to organizations that understand their data and leverage its use to full potential. Data reverse engineering is key to that objective.

Elliot Chikofsky
Principal Consultant
DMR Consulting Group
Burlington, Massachusetts
August 1995

Select below to:

Return to Peter Aiken's Home Page or to the Data Reverse Engineering Home Page.

This page and all web site contents were last updated and are copyright
8/9/07 and previous years by Peter Aiken - all rights reserved.