Data Warehousing Comes of Age

Organizations manage costs with global development teams

Time was when development of a data warehouse had to occur in-house or at least in the backyard of the organization using it. The technology was so complicated and the tools so primitive that there were too many unknowns to let a part of the development go, especially overseas. The minute-to-minute need to coordinate the on-going inputs while building the ETL (Extract/Transform/Load) layer made it necessary for all the players to be in constant communication and close proximity.

As a result, only the biggest of the big companies were able to create these robust depots, rich with business-building information. But data warehousing has matured over the past five to ten years, and with this has come deeper knowledge of the development process and more standardized best practices. Nowadays companies can pull out discrete pieces of the development work that can be safely performed by external vendors. Furthermore, the cost of the platforms for building data warehousing solutions is spiraling downward as open source solutions apply pressure. This makes labor costs a higher percentage of the total cost of these initiatives, which increases the motivation to leverage cost efficiencies through the use of global teams.

The ETL layer, in particular, has been siphoned off and handed to Coherent Solutions and other global development firms. This layer is time-intensive and requires on-going oversight as both inputs and outputs evolve. It is an ideal place to realize cost savings. With these savings, our mid-size clients can create complex data warehousing and reporting services that help them compete in a global marketplace. In addition, time differences work to the advantage of global teams who can perform data validation and other maintenance during evening hours.

While details may vary, a typical iterative data warehouse development lifecycle will consist of the following phases: (1) Initial Definition/Inception, (2) Design, (3) Build/Validate (4) Deploy, (5) Operate, (6) Enhance.

With the current state of technology and toolsets it is fairly straightforward to hand some or most of the work in phases 2, 3, 4 and 5 to an offshore vendor. To create a truly integrated team, however, it is best for the global provider to have a local presence. When the supplier's project managers, architects, and business analysts can interact directly with users and other stakeholders, it ensures a smooth hand-off between phases, reduces the amount of rework, and increases overall agility of the team. The result is better quality work in a shorter timeframe.

With process standards stabilizing and costs coming down, many more mid-size organizations can consider investing in a data warehouse that will help them understand and develop stronger customer relationships to grow their business.

Learn more about our Java software development, as well as .Net Web development services.