Skip to content

Manage Stages

This article applies only to Matillion ETL for Snowflake.

Overview

Stages are storage areas used for intermediate data processing steps between your source system and the target data warehouse. In Matillion ETL, some data warehouses can use their own storage systems such as Amazon S3, Azure Blob Storage, or Google Cloud Storage. Snowflake allows users to define their own staging area, both external and internal, and these areas can be managed through Matillion ETL.

Stages belong to Snowflake schemas. Manage Stages will be populated with existing stages for the default database in your environment configuration. These stages can then be used in many connector components in Matillion ETL when loading data.

To access Manage Stages, right-click on an environment that has a valid Snowflake configuration and select Manage Stages.

Internal vs External stages

Snowflake offers both internal and external staging types. Both of these are supported in Matillion ETL for Snowflake.

  • Internal: The staging area is provisioned by, and staged data is stored within, Snowflake. This requires minimal setup as there is no further configuration for a storage endpoint or credentials required, as long as your Snowflake connection is valid.
  • External: The staging area is outside of Snowflake and must be provisioned by the user. Snowflake supports Amazon S3, Google Cloud Storage, and Azure Blob Storage containers as staging areas. The user is required to provide the location and credentials to use their selected service.

Creating stages

The Manage Stages menu allows users to create, configure, and delete Snowflake stages from within Matillion ETL.

Using Manage Stages

  • Method: Use the drop-down menu to choose one of the following methods:
    • Create: Create this resource. Will generate an error if a resource of the same name already exists in your Snowflake schema.
    • Create if not exists: Will create only if a resource of the same name does not exist in your Snowflake schema.
    • Create or replace: Creates a new resource or replaces an existing one of the same name.
  • Schema: Select a schema for your new stage. This must be a pre-existing schema in the Snowflake database that your environment is configured for.
  • Name: Give your stage a descriptive name.

Click Next to continue to configuring your new stage.

  • Type: Set the type of stage as Internal or External.
    • Internal: Stage is managed entirely by Snowflake. No further configuration required.
    • External: Point to an existing Amazon S3 bucket, Azure Blob Storage container, or Google Cloud Storage bucket to use as a staging area.
  • Platform: The cloud platform (AWS, Azure, or GCP) that hosts the data.

AWS Platform configuration

  • Authentication: Select the method you want to use to access your chosen external table.
  • Credentials: Select your credentials. Click Manage to open the Manage Credentials dialog, where you can set roles and permissions on your hosting platform of your Matillion ETL instance.
  • S3 URL: Select the S3 Container URL you wish to point the created stage at.
  • Encryption: Decide how the files are encrypted inside the storage containers.

Azure Platform configuration

  • Authentication: Select the method you want to use to access your chosen external table.
  • Credentials: Select your credentials. Click Manage to open the Manage Credentials dialog, where you can set roles and permissions on your hosting platform of your Matillion ETL instance.
  • Blob Storage URL: Select the Azure Blob Storage Container URL you wish to point the created stage at.
  • Encryption: Decide how the files are encrypted inside the storage containers.
    • Client Side Encryption: Encrypt the data within the Matillion ETL client. Users will also need to provide a Master Key in the following field.
    • None: No encryption.

GCP Platform configuration

  • Integration: Select a platform-specific Cloud Storage integration. For more information, read Cloud Storage Integration Setup Guide.
  • URL: Select the Google Cloud Storage (GCS) Container URL you wish to point the created stage at.

Using the CLI

Alternatively, the CREATE STAGE Snowflake command can be used to create both internal and external stages.

Deleting a Stage

In the Manage Stages dialog you can remove a stage you have previously created. To do this:

  1. Click the trashcan icon for the corresponding stage.
  2. Click Yes in the Confirm Deletion dialog.
© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement