Updating to version 1.60 and above

This page covers the technical details of how to upgrade and revert between the Matillion ETL non-universal and universal packages along with an overview of the reasoning behind the project.

Background: consolidation of .rpm packages

Prior to v1.60, Matillion ETL was published as three yum packages:

matillion-emerald
matillion-emerald-byol
matillion-emerald-cdata

Each installation of Matillion ETL would then have one combination of:

matillion-emerald AND matillion-emerald-cdata

Or:

matillion-emerald-byol AND matillion-emerald-cdata

Both of the matillion-emerald and matillion-emerald-byol packages then had both cloud platform and cloud data warehouse specific builds. For example:

emerald-1.46.10-centos7-aws-snowflake-platform.rpm   (matillion-emerald package)
emerald-1.59.9-centos7-azure-snowflake-platform.rpm  (matillion-emerald package)
emerald-1.51.9-centos7-aws-redshift-byol.rpm (matillion-emerald-byol package)

However, matillion-emerald-cdata was always published as a single universal build. For example:

emerald-cdata-1.50.12.rpm (matillion-emerald-cdata package)
emerald-cdata-1.56.10.rpm (matillion-emerald-cdata package)

As will be obvious, this large amount of builds/variation resulted in quite a complex installation and complicated the tasks of building, validating, distributing, and providing support.

What's new: matillion-universal package

Therefore, as part of the v1.60 build, there have been two important changes:

The matillion-emerald and matillion-emerald-byol packages have been consolidated to form the matillion-universal package.
There are no longer cloud platform and cloud data warehouse specific builds for the matillion-universal package. Instead, there is one universally compatible build per Matillion ETL version.

For example:

emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)
emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)

From v1.60 onwards, each Matillion ETL installation will require:

matillion-universal *AND* matillion-emerald-cdata

Finding the matillion-universal packages

Two new yum repositories have been created, which can be found at the following URLs:

matillion-universal ——— https://artifacts.matillion.com/rpm/matillion-metl/stable/
matillion-emerald-cdata ——— https://artifacts.matillion.com/rpm/matillion-cdata/stable/

An example yum repo config for each is shown below:

Usually found at /etc/yum.repos.d/matillion-cdata.repo:

[matillion-cdata]
name=matillion-cdata
baseurl=https://artifacts.matillion.com/rpm/matillion-cdata/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-cdata/stable/matillion.key
gpgcheck=1
enabled=1

Usually found at /etc/yum.repos.d/matillion-universal.repo:

[matillion-universal]
name=matillion-universal
baseurl=https://artifacts.matillion.com/rpm/matillion-metl/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-metl/stable/matillion.key
gpgcheck=1
enabled=1

Note

If the machine is running in an egress-restricted environment, ensure artifacts.matillion.com is allowed within the domain/network.

Upgrading to the universal packages

Users have two available methods to upgrade to the universal packages (Matillion ETL v1.60 and beyond).

Note

Before performing any of these upgrades, ensure relevant backups of root disks and managed databases are taken in case of the need to roll back. Read our documentation for more information on how to do this: In-place updates.

Method 1: Updating/Upgrading via migration

To upgrade via migration, launch a machine image of version 1.60 or higher and follow the steps here: Migration.

Method 2: Updating/Upgrading in-place (non-clustered environments)

Updating via the GUI is possible for non-clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package.

To do this, follow the steps here: In-place updates.

Note

It is only possible to upgrade between the final version of the old packages (1.59.12) to the universal packages (>1.60.2).

In practice, if the installation is on a version below 1.59.12, the machine must be updated to 1.59.12 first, then the universal packages will be available (i.e. two upgrade hops).

Both these upgrades will be possible to do in the UI in series.

In the UI, the upgrade between 1.59.12 and >1.60 will only note the matillion-emerald-cdata package being available on the destination version. This is expected, and is due to the nature of the yum transaction.

Updating/Upgrading in-place (clustered environments)

Updating via the console is possible for clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package in place.

Ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:
```
service tomcat stop
```
To identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:
```
yum list installed | grep matillion-emerald
```

This will return either:

matillion-emerald-byol-<some version>

Or:

matillion-emerald-<some version>

To "swap" to the universal packages, the following command can be run:
```
yum swap <package from above> matillion-universal
```

For example, if swapping matillion-emerald-byol:

yum swap matillion-emerald-byol matillion-universal

When you receive the prompt
```
Total download size: 808 M 
Is this ok [y/d/N]: y
```
Input y and await completion.
Finally, restart the service once both nodes have been upgraded:
```
service tomcat start
```

Once the upgrade is confirmed to be successful, to prevent the old packages being installed alongside matillion-universal, run the command:

mv /etc/yum.repos.d/matillion.repo /etc/yum.repos.d/matillion.repo.bkp

This will de-index the repository file mirror for the old packages.

Ongoing updates on matillion-universal

Updates on 1.60 and beyond are then consumed by running the following command:

yum update matillion-universal

Not by running yum update matillion-*

Revert to a virtual machine backup

Rolling back via the console is possible for all environments. This will transfer the existing installation from matillion-universal back onto either the matillion-emerald or matillion-emerald-byol.

First, ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:
```
service tomcat stop
```
If previously the matillion.repo repository file was de-indexed, re-index by running:
```
mv /etc/yum.repos.d/matillion.repo.bkp /etc/yum.repos.d/matillion.repo
```
This will re-index the repository file mirror for the old packages.
Now, to identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:
```
yum list available | grep matillion-emerald
```
This will return either:
```
matillion-emerald-byol-<some version>
```
Or:
```
matillion-emerald-<some version>
```
To "swap" back off the universal packages, the following command can be run:
```
yum swap matillion-universal <package from above>
```

For example, if swapping for matillion-emerald-byol:

yum swap matillion-universal matillion-emerald-byol

When you receive the prompt

Total download size: 351 M Is this ok [y/d/N]: y

Input `y` and await completion.

In the above transaction, the matillion-emerald-cdata package was not pushed down, so that needs to be downgraded to match the version of either the matillion-emerald or matillion-emerald-byol package. Therefore, run the command:
```
yum downgrade matillion-emerald-cdata-1.59.12
```

When you receive the prompt

    Total download size: 458 M
    Is this ok [y/d/N]: y

Input y and await completion.

Finally, restart the service once both nodes have been downgraded:
```
service tomcat start
```

Upgrading to a specific release

To list available versions:

yum list matillion-universal --showduplicates

To upgrade to a specific version (don't include < and >):

yum update matillion-universal-<your-chosen-version>