Monday, June 3, 2002

Information Technology

Scaling the database mountain

By Matthew French

The history of database management extends even further back than does the term "database." In the mid-1960s, several technologists and technology companies were working on file management and file sharing systems.

In 1965, Charles Bachman developed the first network model, called the Integrated Data Store. IBM followed a year later with its hierarchical management system, called the Information Management System.

But both technologies lacked a key function that would bring file and database management into its own. While both provided the facilities necessary to extract data and share files, neither implemented a way to connect one record to another. They needed to establish relationships between the files and write an application into the code that would allow like data to be searched, retrieved and handled by the user.

Hierarchical databases were good for quick retrieval of information, but they required data to be retrieved in a top-down fashion. If the relationship between the data changed, however, this method proved cumbersome and inefficient.

The network model emerged as the stronger of the two, allowing a user to search for like data by using pre-defined relationships. The problem was that if the user did not know the relationship between the data, even this method proved difficult to support.

In the early 1970s, E.F. Codd., a technologist with IBM, published his research on a new form of database management based on a mathematical foundation: the relational database.

"The relational database was really a milestone that coincided with the beginning of the minicomputer and PC boom," said Bernard Drost, the vice president of technology at Westborough's Akibia Consulting. "The storage capacity of computers began to get much larger, and people needed a way to sort through this information."

Oracle, Informix and SQL emerged as three vying standards for management systems, with none emerging as the de facto winner. Oracle shipped its first relational database system in 1978. The company still maintains a grip on the market lead, but it is not a monopoly.

Individual market segments drifted toward common platforms, and that led to an integration problem years down the road.

"Database management became specialized for medical, financial and insurance companies, each of which needed a different way to retrieve its own information," Drost said. "The integration done to customize them became proprietary and was good for its market niche but difficult to integrate with other databases."

Text-based database management was an important tool of the day that could not only search for like fields but could delve further into documents maintained on the database and find like information within them.

In the early 1990s, the idea for an object-based database management system emerged and Sybase began exploring its potential uses. But, as with Java in about the same period, the idea was ahead of its time. Object database management fizzled out for a few years, only to re-emerge with a vengeance in the '90s.

Object databases are capable of determining relationships between objects - pictures, films, audio files and text - rather than just a text or field-based system. Extensible markup language (XML) has become a quasi-standard and remains the preferred method of data retrieval from object databases.

But, Dorst said, XML is slower than many would like.

"Speed is always the problem with database management," he said. "Especially with XML. It's just not as thin as some other protocols."

Robert Nagle, the director of software development at InterSystems Corp. in Cambridge said the next big application in database management will come as a cross between the relational database and the object database.

"We're past the age of the pure relational database, and most end users don't want to deal with the complexity of an object database management system," Nagle said. "Object databases were not able to solve the reporting requirements that many required."

InterSystems has developed a "post-relational" database management system that allows programmers to naturally store items in an object database, but the end user can still retrieve through a relational model.

"It's neither relational nor an object database management system, but it plays both on TV," Nagle said.

Nagle and Dorst agreed that what database management users want now is a more intelligent system that can paint a complete picture of data, including all information that relates to it.

"You want a picture of your customer and where he lives and what he buys, yes," Dorst said. "But you also want to figure out what motivates that person to buy and when. Who are the people that customer influences? That is where we're heading."