Riles of successful data migration

IT managers can no longer avoid the inevitable: they are going to have to bite the bullet and migrate business-critical data.

Tony Sceales (above), CTO of third-generation migration technology vendor Celona Technologies and John Morris, director of Iergo and author of Practical Data Migration, have written this article to help IT departments with successful data migrations.

What makes successful data migration?

Everyone knows while failure is not an option, industry failure rates for complex migrations are high.

Successful data migrations share a number of common characteristics.

Not all migrations are equal.
Many are fairly straightforward and motivated purely by technical or IT drivers.

For example, at the storage level companies might need to migrate their data to cheaper forms of storage, and at the database level they may need to mirror data to another disk.

At the application level, however, data migration can be very complex in large enterprises as it needs to span both technical and business issues.

According to Bloor’s Phil Howard, Forbes 2000 companies already spend at least US$5 billion per year on migrations, yet 80 percent still go over time or budget.

To understand why this is the case it is firstly important to recognise how complex IT infrastructures have arisen.

Often they mirror an enterprise’s history and have come about as the result of mergers and acquisitions, organic growth, product launches, new initiatives and so on compounded over many years. While individual parts of the infrastructure might be well architected, robust and functional, the overall structure might be poorly understood and far from optimal.

However while IT optimisation might be desirable, unpicking this complexity is far from easy and risks catastrophic failure – such as lack of availability of business-critical data or failure of service or product delivery systems.

Analysing successful IT migration projects reveals, however, common success factors.

Rule one: data migration is a business not a technical issue

Historically migration has been viewed as a purely technical issue and the most pressing concern has been how to migrate data.

But since IT doesn’t necessarily ‘own’ either the source and target applications or the data that the business uses to function, it doesn’t have the power or the necessary knowledge to deliver what is required by the business.

Further, an increasing feature of the modern IT environment is that IT has come out of the data centre and is no longer controlled by the IT department in the way that it might have been previously.

Involving business managers in data migration projects is therefore essential. In fact one of the most common characteristics of a successful migration is that they are business- rather than technology-led.

Putting the business in the driving seat means that before we ask “how do we migrate data” we first answer a series of important related questions that help us frame and scope the project:

· Why are we migrating data?
· What data should be migrated?
· When should it be migrated?

These questions can’t be answered by technicians, only by business managers – one reason it is essential for the business to drive migration.

Ensuring the business makes the decisions and drives the project also frees IT up to do what it is best at – the technical aspects of moving data.

This suggests any technical solution to the migration must provide the business with clear visibility and control over the way the program is progressing.

Rule two: the business knows best

The second rule of successful data migration is closely linked to the first. This is that the business drivers, not technical ones, should take precedence.

It is critically important that business goals should define the solution and approach selected, and not the other way around.

The best technical solution is not always the best business solution. Therefore to be successful, chief business stakeholders must define their requirements and take responsibility for driving the project.

The business is also better placed to prioritise the migration, deciding which data to migrate first.

For example, it might decide that migration be triggered by a customer selecting a new service which can only be supported on the new application stack. Or it might decide the most important customers be migrated first, or second, or last (according to business need).

By allowing the business to drive the migration an enterprise has a better chance of delivering the right solution for its needs and maximising the ROI on its investment.

However, IT must be aware the business won’t be able to predict at the outset what issues will surface during the migration, and moreover in a long-running program they will absolutely have to handle changes in priority and direction.

The chosen migration technology must be able to support such changes without restarting every time.

Rule three: no one needs, wants or will pay for perfect data

Applications are only as good as the data they have available to them.

We also know that many a data migration has been scuppered by overestimating the quality of, or not understanding, source data.

Oh the joy of legacy data with its gaps, inconsistencies and redundancies.

However, while enhancing data quality is a worthy goal, it is really important not to go off on a tangent mid-project in the quest for perfect data quality.

Like over-specification of an application, the quest for data perfection can result in negative consequences for the project.

It is where many, many projects run aground – inflating both the cost and time to deliver the project. To avoid this trap, data owners and users need to determine the level of quality they need at the start of the project so the technologists have an appropriate goal to aim at.

It’s also why project managers need to be aware of the true quality of their legacy data and allow adequate time and budget to achieve the requisite data quality.

A successful migration strategy also needs to incorporate a range of cleanse strategies at different points in the life of the program – sometimes pre-migration, sometimes in-flight, and sometimes post-migration – but always with a conscious decision from a business manager.

Too many historical migration approaches mean bringing the whole process to a halt while a data quality issue is explored and resolved.

A modern platform will provide the flexibility to continue migrating while this is done.

Rule four: if you can’t count it, it doesn’t count

Another challenge is how to measure data quality in order to assess the state of your legacy data and determine the level of quality your business users require.

To make matters worse, data quality is not static but erodes and improves over time.

It’s really important the measures you use make sense to business users, not just to technologists.

This enables you to measure deliverables, perform gap analyses, and monitor and improve ongoing data quality. It also ensures you concentrate your efforts on where business users see value and can quantify the benefits.

Reconciliation of data migrated from source(s) to target(s) is always a critically important activity – how do you know when you’re done? When dealing with dynamic environments where you can’t freeze the data this becomes even more challenging: you need to be able to handle a shifting scope.

Having a flexible data model and closely coupled reporting capability is key to understanding and driving this process.

Achieving a business-driven migration

Having put a business-driven migration project in place the trick then is to select methodologies and technologies that can deliver against these requirements.

Business-driven migration involves decoupling the technical problem of moving data from the business processes that use it.

This requires a migration solution that enables you to easily encapsulate the business problems you face, while being flexible enough to cope when those requirements change. This ensures that ROI from new application investment is maximised and operations enhanced rather than adversely affected.

The importance of getting migration right from a business perspective was addressed at a recent British Computer Society meeting by BT’s Phil Dance: “Increasingly our business case will depend on how good we are at getting our data across.

A bad data migration ultimately means a bad customer migration, in a competitive market that’s very bad news.”