TDWI PreConference Education

The Data Warehouse Institute offered great prices on two different pre-conference education opportunities:

• Data Integration Techniques
• Data Warehousing Architectures

I attended the Data Integration Techniques workshop, presented by Dave Wells, director of education for TDWI. He did an absolutely outstanding job on the workshop and brought together a lot of different ideas about what it takes to do data integration projects of any kind: data warehousing, data consolidation, data synchronization, data integration from mergers, etc. Though the general ideas aren't really new to veteran data integration/warehousing people, the organizational tools that Dave presented were extremely valuable:

• Definition of a continuous life cycle for data integration activities, reflecting the fact that none of us ever can really build something and walk away -- the BI/DW space is too connected to business rules and business units that are ever changing. (In fact, one of the things that BI is supposed to do is drive change in the business!)
  
• A taxonomy of terms and ideas to use when looking at the activities within the data integration lifecycle.
  
• A methodology for (1) identifying and qualifying source systems for use in an integration project; (2) mapping sources to targets (or targets back to sources as he prefers) at the various levels of abstraction from business entity to physical table/file to field level; (3) classifying types of transformations that are potentially needed in the integration activity; (4) digging up and validating business rules using continuous data profiling; (5) evaluating and measuring data quality as a formal metric over the data integration activities; and (6) working with business to put it all together into a successful and iterative lifecycle. (Wow! What a mouthful.)

I highly recommend taking a look at the TDWI courseware and taking the time to attend some of these workshops. Even if you already know everything, there's always a different way of looking at things that can help you tweak your own system for approaching data integration challenges -- and most of us don't know anything close to everything!

Thanks Dave!