Evolution of Data Integration
In the next years, the need for data integration and data exchange will continue to increase in complexity, volume and frequency. The growth of business analytics and e-business will force organizations to implement bigger data warehouses, near-to-real-time data exchange solutions and to consolidate their information systems in order to provide more information on-demand.
At the same time, both the number and the nature of computing components will continue to expand in every information system, making requirements for data integration even more strategic for IT professionals.
The new era of data integration started with Extraction, Transformation and Loading (ETL) tools. The rapid development of the business intelligence (BI) market originated the genesis of the ETL market. But rapidly, companies began to realize that the power of ETL tools could be used for other applications. The simple architecture, where Online Transaction Processing (OLTP) data were made available to end users through On-line Analytical Processing (OLAP) environments, started to evolve in more complex data warehouses and datamarts. All components of the information system, including ERP, CRM and other production systems, were incrementally incorporated in the decision-support platform to deliver a “single version of the truth.” The classic data warehousing architecture became more like a hub-and-spoke architecture, where all kinds of applications would communicate and synchronize. The Enterprise Application Integration (EAI) market was born.
Data Integration Matures
The next step in the evolution of the data integration market was introduced by the accelerated growth of the Internet and e-commerce, where organizations had to communicate with each other across the Web (HTTP), usually using XML or EDI messages. Data integration tools had to satisfy additional requirements in data movement and integration in order to fulfill organizations’ business-to-business integration (B2Bi) requirements.
Although all these data integration types—ETL, EAI and B2Bi—address different business requirements and provide specific functionalities, they often share many concepts:
- Source data should first be accessed or extracted, using specific connectivity or transport layer;
- Data should be processed to implement technical transformations or business rules; and
- Data should be delivered or loaded into target system.
The main differences between the three types of data integration solutions will reside in the nature of the data (e.g. raw data, business transactions, etc.), the format of the data (e.g. database record, XML message, etc.), the process type (e.g. event based, batch oriented, workflow, etc.) and the transport mechanisms that will be used (e.g. database connector, middleware, Web services, etc.).
Today, organizations that want to implement these three types of data integration solution typically have to use multiple products. This results in increased complexity of their information system, reduced data lineage capabilities and higher total cost of ownership of the data integration platform.
In the late ‘90s, Hummingbird was one of the first companies promoting the “Information Broker” paradigm, which is premised on the new era of Enterprise Service Bus (ESB)—a new architecture that exploits Web services, messaging middleware, intelligent routing and transformation.
Single and Accurate Version
To increase their competitiveness, organizations have to make more informed, strategic decisions based on a single and accurate version of the truth. This means that they require a solution that will deliver them accurate, consistent and timely information by connecting their entire data source to any target system throughout the enterprise. Also, organizations have to improve their business processes by implementing faster and more efficient communication channels between all components of their information system and other internal or external systems.
To implement this vision, organizations need a comprehensive solution that offers a strong ETL package, with extensions into the EAI world (XML connectivity, support for IBM WebSphere® MQ, etc.) and an enterprise-wide metadata management solution. This strategy gives customers an integrated and simple solution that can be used for all data-integration applications. It will provide a complete solution in which the EAI component will leverage the high transformation capacity of ETL, and the ETL component will benefit from real-time or near-real-time data exchange to enhance existing batch-oriented processes. By providing better synergy within the data integration world, customers can utilize the same data flow to implement asynchronous and synchronous data exchange between systems. This general-purpose solution will be used to implement any data integration processes that are required to implement decision support systems, application integration or B2B processes.
A data integration solution can be the backbone of an enterprise content management (ECM) system and can interconnect all components within the enterprise and with external systems. Combining information from unstructured content, structured repositories and line of business systems within business processes will give an organization an unprecedented advantage in gaining knowledge from their information systems.
Current versions of Hummingbird ETL collaborate with message-brokering technologies to combine the business benefits of real-time message transportation with near-real-time complex data transformation needed to provide an integrated ETL/EAI solution. The support of IBM WebSphere MQ and the experience gained from building SAP-certified cross-application data integration enables Hummingbird to further develop this strategy.
To improve information brokering capabilities, a solution should extend to include more features:
- The current message format support should be enhanced and extended to allow better processing of all messages. Current support for XML, IDoc and flat files should be enhanced to provide easier and faster implementation of integration processes. Support for new message formats should also be added to natively process messages such as EDI, HL7, SWIFT, ISO15022, etc.;
- In addition, new messaging connectivities should be incorporated to enable real-time integration with all existing systems. For example, specific connectors would interface with corporate messaging technologies such as MS MQ, Tibco, JMS, etc. or to communicate with specific packaged applications (e.g. SAP, PeopleSoft, Oracle, etc.);
- Also, compliance with Service-Oriented Architectures (SOA), new standards (e.g. Web Services (WDSL, SOAP), UDDI, JCA, etc.) should be supported;
- Communication protocols such as HTTP, HTTPs, FTP stream, SMTP, etc. should be supported to enable further integration with other systems; and
- New components should add to an existing data integration solution to process data in real-time mode and manage data flows. Workflow tools will become an important component of any data integration solution. It will permit the creation, management and monitoring of business processes allowing the flow of work or data between individuals and/or departments to be defined and tracked.
This comprehensive and fully integrated solution will make data exchange between systems, faster, better, cheaper and more reliable.
With data integration, structured content from within an organization (e.g. decision support platform, line-of-business systems, etc.), or external environments will be seamlessly integrated with unstructured documents. Enterprise content management systems unlock information repositories and external applications to business decision makers, and data integration is the key.
Feeding Chick-fil-A’s ETL Needs
With 50 years of business success behind it, Chick-fil-A® is the third-largest quick-service chicken restaurant company in the United States. While selling 91,800,000 chicken sandwiches a year, the chain’s 960 restaurants generate tremendous amounts of sales, inventory and payroll data. Corporate headquarters in Atlanta, GA, uses this information to develop financial reports, provide payroll services, create profit-and-loss statements and provide other financial services.
Chick-fil-A’s previous system for sharing data between multiple database systems was not as hands-off or as customizable as required. Information generated by back-office computers at restaurants throughout the US was stored in a local client-server database. This information was replicated to a consolidated database in Atlanta, then transferred from this database to the company’s enterprise resource planning (ERP) database.
Hummingbird ETL Simplifies Data Transformation
Under the previous system, Chick-fil-A used replication to mirror its consolidated Sybase SQL Anywhere database in another Oracle database. Custom PL/SQL code was used to transform the data and move it to the ERP database via a database link. If there was an error in the replication process or a transaction failed, the mirrored Oracle database would become out of synch with its source and was difficult to rematerialize. In order to rematerialize, each transaction had to be reapplied in the order it was received. Since only 10%-20% of the data was critical, this system could be extremely unforgiving. Chick-fil-A needed an Extract Transform Load (ETL) solution that would allow them to select only the necessary information and identify specific transactions that were necessary to meet their deliverables.
Chick-fil-A turned to Hummingbird to replace the existing data extraction system with a simpler, more efficient means of transferring information. Several vendors were considered, but after close evaluation, Hummingbird ETL, a universal data exchange solution, was chosen.
“Hummingbird ETL has been very successful in improving the quality of our data integration processes, while reducing the resources required to support it,” says Jason Headley, Senior Oracle DBA at Chick fil-A. “The product was relatively easy to install, and its intuitive interface allowed us to get users up-and-running with a minimal training investment.”
Read the full in-depth success story at Hummingbird
Hummingbird Ltd. is a global enterprise software company employing 1,300 people in nearly 40 offices around the world. Hummingbird Enterprise creates a 360° view of content with products that are both modular and interoperable, including Document and Records Management, Portal and Knowledge Management, Business Intelligence and Data Integration. Please visit:Hummingbird.