Skip to content

Migration

The Migrate feature in Matillion ETL enables you to copy resources from one ETL instance to another, typically to a newer version.

Migration is a distinct and separate process to In-place updating. Read Updating and migrating overview to learn the difference between these two processes and determine whether migrating is the best option for you.

You need two Matillion ETL instances for a migration: (1) your original instance which contains the resources you want to copy (referred to as the Source instance), and (2) a new instance to copy those resources into (referred to as the Target instance). The new Matillion ETL instance isn't created as part of this process; you must have the new instance ready and operational before you begin. Read Create a target instance for details.

No resources in the source instance are deleted or otherwise altered as a result of the migration.

You should not migrate from a dbt-enabled instance to a non-dbt-enabled instance.

Migration best practices

Before you begin the process, consider the following factors.

  • A migration is not a 100% duplication of one instance into another. Ensure you have a working and tested migration strategy for features that don't get automatically migrated. The following resources will not be migrated:
    • Custom licences.
    • SSL certificates.
    • Task history.
    • Python libraries.
    • Non-standard JDBC drivers (click Admin in the top-right of the user interface, and then click Manage Database Drivers to view the list of database drivers that come with Matillion ETL "out of the box").
    • Customized "standard" API profiles.
    • LDAP setup.
    • Backend configuration changes (for example: emerald.properties file changes).
    • Git resources in versions of Matillion ETL prior to 1.61. See below for more details.
    • Local user configuration.
  • Ensure that your new server is given the same cloud privileges as the original, including feature access rights and firewall rules.
  • If you rely on any operating system (OS) customization (such as additional libraries or drivers), ensure that you can replicate those customizations on the new server. Consider creating your own AMI based on Matillion's, with your customizations "baked in". Then, upon each release, remake your AMI from our latest one and test. This could also be used to ensure that drivers, user configuration, and so on, are "baked in".
  • We recommend that you configure any non-migratable resources on your target instance before the migration.

For more information on best practices, see this Matillion blog post on the subject.

Perform a backup

While the risk of issues during a migration is much smaller than performing an in-place upgrade, it is still prudent to take a backup of your Matillion ETL instance before migrating. Refer to the following topics for more information:

Create a target instance

If you don't have a target Matillion ETL instance (where you are migrating resources to) created already, you will need to launch it before migrating.

Make sure that the target Matillion ETL instance:

  • Is running a version of Matillion ETL that is identical to or newer than the source instance. For example, if the source instance is running version 1.53, then the target instance must be running version 1.53 or newer.
  • Is on the same cloud data warehouse service as the source instance. For example, Snowflake to Snowflake.
  • Is hosted on the same marketplace platform as the source instance. For example, AWS to AWS.
  • Is given the same cloud privileges as the original, including feature access rights and firewall rules.
  • Allows incoming TCP/IP connections from the source Matillion ETL server.
  • Does not connect to the same Matillion ETL database as the source Matillion ETL server.

Ensure that the following URL is allowed, to allow the target instance to update:

artifacts.matillion.com

There are CentOS and PostgreSQL repositories that are enabled on the instance by default. The CentOS repositories are mirrors that can change, so can be difficult to allow. You can disable or point to an internal repository instead if required.

Security

Before migrating, ensure that:

  • Any connections are secured so that data is protected while in transit.
  • The user credentials (see User Configuration) that you plan to use to authenticate to each instance (source and target) have the necessary server-level user roles assigned, as follows:
    • API (this applies to all migrations).
    • Global Project Admin (applies when migrating project content such as orchestration jobs, transformation jobs, environments, passwords, queues, schedules, variables, versions, CDC configuration, and CDC tasks).
  • All users are logged out when performing a migration. This is recommended to avoid any issues with changes to resources that are actively being worked on, or any credentials that are in use.

Permissions apply as usual to both the source user and target user. For example, if the target instance user doesn't have permission to write shared jobs, then that user can't import shared jobs via the Migrate wizard. Read Groups and permissions for more information on how these are set.

Performing the migration

In Matillion ETL, click Admin → Migrate. This will open a wizard with several pages of details to complete, as described below.

Note the following before proceeding:

  • All user, group, and permissions data will be overwritten in the target instance if it already exists.
  • Users who wish to migrate user resources must have the role Server Admin for their source user and their target user.
  • A user's username and password are securely migrated from source to target.

Additionally, your authentication method in the source and target instances must be the same. You will not be able to migrate users and user configuration if the authentication methods differ.

Read LDAP integration to learn more.

Configure target instance

  1. Configure the details of the target instance.

    • Target instance URL: The URL of the Matillion ETL instance you wish to migrate resources to. You must include the http:// or https:// portion of the URL.
    • Username: Your Matillion ETL username on the target instance.
    • Password: Your Matillion ETL password on the target instance.

  2. Click Test Connection to confirm that your details are correct. You won't be able to continue if the test fails.

  3. Once the test is completed successfully, click Next to continue.

Users

  1. Tick the Migrate all users, groups and permissions checkbox if you wish to migrate user configuration settings, including users, user groups, usernames, passwords, roles, and permissions.
  2. Click Next to move to the next page.

Project content

  1. Use the file explorer to select the project content you wish to include in the migration.

    Note

    • If migrating only part of a project group or project, make sure that the relevant hierarchy exists on the target instance. For example, if migrating a version, the project and project group must already exist.
    • If you're migrating project content, the source and target users must have the Global Project Admin role in Matillion ETL. See User Configuration for details of how to do this.
    • Projects are filtered based on an access control list, and you won't be able to migrate a project unless you can access it on both the source and target instances.

  2. If your Matillion ETL instance has projects that use Git, this page of the wizard will ask How would you like to migrate projects which use Git? Select either Migrate with Git or Migrate without Git. The projects that are using Git are identified in the wizard with a Git icon next to the name. Read Migrating Git resources for more information.

  3. Click Next to move to the next page.

Instance configuration

  1. Use the file explorer to select the instance content you wish to include in the migration.
  2. Click Next to move to the next page.

Migration options

  1. Choose additional migration options.

    • Disable schedules: Disables the scheduler on the target instance. This is selected by default. This won't affect individual schedules, which will retain their Enabled/Disabled status, but schedules won't run until the scheduler is re-enabled. Users can re-enable the scheduler at their own discretion.
    • Disable queues: Disables queues on the target instance. This is selected by default. Users can re-enable their queues at their own discretion.
    • Disable CDC: Disables Change Data Capture (where available) on the target instance. This is selected by default. Users can re-enable CDC at their own discretion.
    • Disable CDC task source creation: Disables CDC task source creation (where available) on the target instance. This is selected by default.
    • Delete project if already exists: Select this if you are migrating a project that already exists on the target instance in some form and you wish to delete the existing version of the project on the target instance. This isn't selected by default.

  2. When you are happy with the configuration, click Migrate to begin the migration process.

  3. On completion, a Migrate dialog shows the status of the migration, listing each migrated element with a "Success" message or error message if it has failed.

Migrating Git resources

As of version 1.61, Matillion ETL supports the migration of Git resources. You can choose to migrate project content with or without Git, as described below. When you perform a migration with Git enabled, commits for orchestration and transformation jobs in every project in the target instance will be flagged as Modified. You will need to perform a one-time commit of all jobs after the migrate, which will remove the Modified flag.

This is expected behavior, due to some of the formats of the jobs changing during the upgrade, and isn't a cause for concern. These are valid changes to the structure of the jobs and you can commit them safely.

Migrating projects with Git will migrate the local Git repository, including remote repository configuration if it exists.

Warning

Any uncommitted changes are not migrated.

To learn more about Git integration within Matillion ETL, read Git Integration with Matillion ETL.