Contact us

0333 772 1963

Get in touch

    4 steps to successful data migration

    Anyone who has ever been involved in a data migration will tell you how important it is to have data that is fit for purpose ahead of starting the process. They will also tell you how important it is to be able to generate the metrics by which you will measure and gauge success of the migration when the data eventually lands in the new system. They will also tell you that they don’t ever want to do it again!

    Quick fixes, workarounds, decisions around how to process and store data are taken at various stages of an application system’s lifecycle. Typically driven by expediency, to get things done to support operational day to day issues that arise in the real world of data processing. Some of these become embedded in the way the business processes its data and as a result by-pass the way the system was intended to be used.

    Migrating data from one system to another (typically a newer system) is an event that typically brings home to roost all the data and processing nasties that have built up over years of system use and abuse. If your data is not fit for purpose before starting the process, it is inevitably going to generate significant problems (both costs and other consequences) when you come to try and run a new system on the same data. For example, if your new system dispatches personnel to addresses using SATNAV systems and you don’t hold a valid postcode for the target address it will cause operational problems and likely lead to customer complaints and other bad business outcomes. If you come to collect payments from a client and outstanding charges and account balances have not been migrated to the correct account then mayhem could ensue.

    What does a successful data migration look like?

    For a data migration to be successful it should be evidenced to have migrated all of the data necessary for the new system to operate efficiently and effectively but more importantly enable the business to realise the benefits and value the system was intended to deliver. If post go live, this does not happen then your new system benefit realisation model will quickly unravel and as a project you will have some serious explaining to stakeholders to do.

    4 steps to success

    Step 1 – Assess the quality of your data against the benefits and rules of the new system

    Obviously if you have been sustainably managing the quality of your data using infoboss before you start, then the quality of your data will be fit for current purpose. Sadly, not everyone has put in place a process to manage and maintain the quality of their data and as such we recommend an assessment of the quality of the data is undertaken before you start. A key factor of this assessment should be to measure the quality of the data against the processing rules that will be needed in the new system. So in our earlier example, postcode was not mandatory in the current system, but it is fundamental to delivering an efficiency benefit in the new system so the quality of the data should be measured in terms of what is needed for the new system NOT what is needed for the current system. This exercise will inevitably throw up data issues that you can correct in advance on the current system, but also opportunities to augment and enhance your data with additional data that will maximise business benefit in the new system. For example, we don’t currently have a place to hold a client email address in our current system but in the new system it will be needed for sending bills, we therefore need to augment our data with the valid email addresses that we have captured for existing customers as part of the migration.

    Step 2 – Understand how you will measure the success of the migration

    It is not as simple as saying that we want all of our data in the new system. You may want to take the opportunity to only migrate data that you need in the new system, certainly customer data for which you don’t have a legal basis for processing should be minimised anyway right? Most will want to migrate only data relating to current clients or former clients for the past 2 years say. To measure success you need to be able to determine the measures and quality for all of the entities required in the new system from the data you have in the current system. When you come to reconcile the new system to the current you will need to be able to compare the two, apples with apples!

    Step 3 – Assess, compare and reconcile across each stage of the migration process

    There are typically three main storage locations for your data on its journey to the new system.

    • Current system – raw data (A)
    • Transformed and augmented data as built by the migration processing
    • New system – raw data (B)

    As the data progresses along the pipeline it should in theory be improving in quality and getting closer to being fit for purpose in the new system. However, things can go wrong and not all data makes it from A to B in one piece. Being able to quickly assess reconciliation issues during development of your migration process will greatly minimise the risk of failure but also improve the efficiency of the team undertaking the development as they will be able to analyse against the target metrics at each stage and understand the impact and consequences.

    Step 4 – Sustainable data quality management

    Don’t make the same mistakes again! Research from the University of Denmark suggests that the quality of data degrades at a rate of 2% per month if left un-managed. At infoboss we recommend that you embed business as usual processes for monitoring and managing the quality of your data in a sustainable way. By this we mean put in place the tools and processes to ensure that your business data rules are not compromised, but maintained and nurtured to ensure that your data remains at the level of quality arising from the migration forever.


    1. It is imperative that before starting a data migration you fully understand the strengths and weaknesses of your data and its quality before starting the process. This needs to be undertaken on your current data with the opportunity presented by your new system in the way that it will process the data in mind.
    2. Build the metrics and means of reconciling your data on its migration journey
    3. Consider the longer term and how you will maintain the quality of your data in a sustainable way

    If you’ve been affected by any of the issues raised in this article, planning a data migration or in the maelstrom of one right now, then please don’t hesitate to get in touch to discover how infoboss can support your data migration project, reduce risk and help you to sustainably manage the quality of your data forever…