Blog

Migrating your data without losing it: steps, trial runs on a copy, cutover and rollback

For organizations · By KTMA SOLUTIONS · Published · 9 min read

Changing management software, moving to a new ERP, leaving an aging server: sooner or later, you have to migrate your data. It is also the moment when an organization discovers what its files really contain: duplicates, unusable free-text fields, customers with no address, invoices tied to customers who no longer exist. If you are looking for a way to move your information from one system to another without nasty surprises, this article presents an eight-step method, the checks that really matter and how to decide on a cutover and a rollback plan.

Our sources are guides from public administrations (The National Archives of the United Kingdom, the GOV.UK Service Manual) and Quebec's access to information commission (Commission d'accès à l'information du Québec). Where a recommendation comes from our own practice rather than from these sources, we say so.

Why a migration rarely fails "all at once"

A migration can fail for several reasons: poorly defined success criteria, anomalies discovered late, transfer errors or the absence of a rollback plan.

The checklist from The National Archives (United Kingdom) for migrating records management systems starts there: objectives, success criteria, governance, resources, risks. It asks whether the criteria are "sufficiently clear and measurable," and poses a question that is often forgotten: is migration the right approach?

Step 1: Take inventory and decide what migrates

Before any technical work, you need to know what you have. The National Archives checklist asks you to identify which information is affected, how the business needs to use it and which information must be migrated. Not everything has to come along.

For a useful inventory, record at a minimum:

  • the source systems (database, spreadsheets, emails, files on a server, applications that sprang up "in a corner") and their business owner;
  • approximate volumes and formats;
  • dependencies (reports, integrations, automatic exports that read the data);
  • the personal information present, because it triggers obligations (see below);
  • what can be archived or destroyed rather than migrated.

On that last point, Quebec's access to information commission (Commission d'accès à l'information, CAI) reminds us that businesses must have rules for retaining and destroying personal information. Migrating data that you no longer have a reason to keep means carrying an unnecessary risk.

Step 2: Assess data quality and assign responsibilities

Poorly maintained data migrates just fine, and stays poorly maintained. Use the inventory to measure quality: empty required fields, inconsistent formats (dates, phone numbers), duplicates, codes that no longer match anything. The National Archives checklist insists that the information owner understand the business needs in enough detail to judge whether the migrated information meets them, and that there be clear owners before and after the migration.

We recommend (practice, not a cited source) designating, for each domain, a person on the business side who settles ambiguous cases.

Step 3: Establish the source-to-destination mapping

The mapping is the correspondence table: this field in the old system becomes that field in the new one, with this transformation. The National Archives checklist details its questions:

  • does the export contain all the required information?
  • can the destination system accommodate it?
  • what transformation is needed to make the export importable?
  • can information missing from the export be captured another way?

Document what will be missing from the migration (The National Archives checklist also asks you to judge whether the old system should be kept longer because of this): a documented and approved loss can be a migration choice, provided that legal retention obligations and operational needs are met; a loss discovered six months later can lead to an operational or compliance problem. Formalize the complete process (extraction, transformation, import, settings, manual steps) so that the process is documented, reproducible and verifiable on cutover day, taking into account changes made since the last trial run.

Step 4: Rehearse on a copy, with representative data

Never test for the first time on the live environment. The National Archives is explicit: the solution must be developed outside the production system, then the migration must be tested outside production, with representative data and volumes, refining and iterating as needed.

Our recommendation, as good practice:

  1. Take a recent copy of production (protecting personal information: restricted access, secure environment).
  2. Run the complete migration process from start to finish.
  3. Measure the duration: it will determine the cutover window.
  4. Fix, then start over, until you get a stable result.

Each rehearsal should produce a validation report (step 5) so that you can see the improvement.

Step 5: Check the relationships, not just the totals

This is the classic trap. You compare the number of customers (1,000 before, 1,000 after) and conclude that all is well. But an identical total can hide serious errors: invoices attached to the wrong customer, orders with no lines, orphaned contacts.

For this reason, we recommend three levels of checking (they come from our practice, not from the cited sources):

  • The totals: numbers of records by type, sums of amounts, minimum and maximum dates. This is a safety net, not proof.
  • The integrity of relationships: every invoice points to an existing customer, every line to its order, every attachment to its record. Look for orphans and duplicates.
  • Targeted samples and edge cases: compare, record by record, complex files chosen with the business side (the customer with ten years of history, the partial credit note, the closed project with adjustments).

The National Archives reminds us that you need to evaluate the test results and verify that the migrated information meets the requirements defined at the outset.

Step 6: Choose between a full and a phased cutover

There are two main ways to move to the final state.

Full cutover ("big bang"): you freeze the old system, migrate everything, open the new one. Advantage: a single transition, no double entry. Drawback: everything rests on a limited window, and a problem affects everyone.

Phased cutover: you migrate in batches (by site, by data type, by team). The GOV.UK Service Manual describes the "strangler" approach on the technical side: you first wrap the old system in a new interface, then replace the pieces one by one, which lets you "control the pace" of the transition. Advantage: the risk is divided up. Drawback: for a time you have to manage two systems and the consistency between them.

The choice depends on the tolerance for downtime and on how intertwined the data is. In both cases, the GOV.UK Service Manual gives three scheduling tips:

  • agree on a plan with stakeholders;
  • set a date, allowing for a delay buffer (the guide mentions a month or two);
  • do not choose a date too close to the end of a contract or licence for the old system, otherwise a delay can force a renewal.

Step 7: Plan the rollback before you need it

The National Archives checklist asks the question twice: "can a failed or partial transfer be undone (rolled back)?" It places it among the risks to examine during preparation and at the time of the transfer. That is the right way to ask it: before the crisis, not during.

A useful rollback plan includes, based on our practice:

  • a full backup of the old system, whose restoration has been tested, along with a documented procedure for recovering or reconciling the operations carried out since the cutover;
  • stop criteria written in advance (for example: "if the relationship check reveals unresolved anomalies, we go back");
  • an identified point of no return: from when a rollback becomes impossible or requires reconciliation, because operations entered in the new system can no longer be replayed in the old one, with the rollback conditions and the data to preserve;
  • the decision to keep the old system read-only for an agreed period.

The National Archives checklist also provides for a process for deleting the source, to be defined and tested, but as the very last step: you destroy the old system only once continuity has been verified.

Step 8: Obtain validation from business stakeholders

Tools say "done"; only the business side can say "correct." The National Archives provides for a continuity test step after the transfer: test the migrated information, evaluate the results and, if it does not meet the requirements, correct, redo the trial and repeat the transfer.

Have real scenarios run (produce an invoice, find an old file, generate the monthly report) and obtain written validation from a manager before declaring the migration complete.

If your data contains personal information, the migration can trigger obligations. According to the CAI guide (April 2024), a privacy impact assessment (PIA) is mandatory in the private sector notably when you plan to:

  • disclose personal information outside Quebec (section 17 of the Act respecting the protection of personal information in the private sector (Loi sur la protection des renseignements personnels dans le secteur privé));
  • acquire, develop or overhaul an information system or electronic service delivery system involving personal information (section 3.3).

The guide specifies that the scope of the assessment must be proportionate to the sensitivity of the information, its purpose, its quantity, its distribution and its medium. In addition, the CAI indicates that in the event of a confidentiality incident presenting a risk of serious injury, the business must notify the Commission and the individuals concerned, and keep a register of incidents. A poorly secured migration (export files left on a workstation, overly broad access) can lead to that. This article is not legal advice: validate your situation with your person in charge of the protection of personal information.

Checklist before the cutover

  • The success criteria are written down and approved.
  • The inventory is up to date; what does not migrate has been decided (archived or destroyed according to your rules).
  • The mapping is documented, including what will be lost or transformed.
  • At least one complete rehearsal has been done on a copy, with representative volumes.
  • The checks include totals, relationships and edge cases.
  • The cutover duration is known and the date has a buffer.
  • A verified backup exists; the stop criteria, the point of no return and the procedure for recovering or reconciling operations after the cutover are documented and tested.
  • The business side has validated real scenarios.
  • The PIA has been done if the law requires it.

Frequently asked questions

How many rehearsals are needed? The sources do not set a number: The National Archives asks you to redo the trial for as long as the imported information does not meet the requirements.

Do we have to migrate everything? No. What no longer has any use can be archived or destroyed according to your retention rules.

When should the old system be deleted? Once continuity has been verified and the rollback period is over. Deletion must be a defined and tested process.

KTMA SOLUTIONS expertise and solutions

Sources

Back to the blog · Blog RSS feed