Skip to content

Installing Matillion agent for Snowflake

An agent can be installed directly into your Snowflake account's Snowpark Container Services, using a Snowflake Native App.

The compute pool for the agent will consume Snowflake credits. For more details of credit consumption, read Compute pool cost in the Snowflake documentation.

Note

The agent isn't currently available on the Google Cloud Platform, as GCP currently doesn't support Snowflake Native Apps with Snowpark Container Services.

Snowflake is used to store secrets used by the Data Productivity Cloud. For details of how to configure Matillion agent for Snowflake to use these secrets, read Secrets in Matillion agent for Snowflake.

Prerequisites

To deploy the agent to your Snowflake account, the Snowflake environment must be prepared. This requires the following actions in your Snowflake account. A user with the ACCOUNTADMIN system role will be required to perform these steps.

Create a new role

This is the role that the Container will run as, and is the role needed to perform all subsequent deployment and configuration for the agent.

  1. Create the role, using the following commands.

    USE ROLE ACCOUNTADMIN;
CREATE ROLE <native_app_role>;

    Where <native_app_role> is the name you want the role to have.

  2. Grant permissions to the role.

    GRANT ROLE <native_app_role> TO USER <user_name>;
GRANT CREATE INTEGRATION ON ACCOUNT TO ROLE <native_app_role>;
GRANT CREATE DATABASE ON ACCOUNT TO ROLE <native_app_role>;
GRANT CREATE APPLICATION ON ACCOUNT TO ROLE <native_app_role>;
GRANT CREATE COMPUTE POOL ON ACCOUNT TO ROLE <native_app_role> WITH GRANT OPTION;

    Where <native_app_role> is the name you gave to the role in step 1, and <user_name> is the user you want to grant the role to.

Set up a warehouse

A warehouse is required when activating the application and creating the agent, and will be used by the agent when completing tasks such as running pipelines. You may use an existing warehouse if a suitable one exists, or create a new warehouse for the agent to use.

Use an existing warehouse

To use an existing warehouse, you must give the new role USAGE permissions for the warehouse, as follows:

GRANT USAGE ON WAREHOUSE <warehouse_name> TO ROLE <native_app_role>;

Where <warehouse_name> is the warehouse you wish to use, and <native_app_role> is the role you created previously.

Note

As this warehouse will be used by the agent, it should have the appropriate suspend configuration and sizing.

Create and use a new warehouse

If you choose to create a new warehouse, it must be created by the role created earlier. The role will require the following additional permission to allow this:

GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE <native_app_role>;

Where <native_app_role> is the role you created previously.

You can then create the new warehouse as follows:

CREATE OR REPLACE WAREHOUSE <native_app_wh> WITH
  WAREHOUSE_SIZE = 'X-SMALL'
  AUTO_SUSPEND = 180
  AUTO_RESUME = true
  INITIALLY_SUSPENDED = false;

Where <native_app_wh> is the name of the warehouse.

Note

Warehouses consume Snowflake credits, so you should size the warehouse appropriately. For guidance, read CREATE WAREHOUSE in the Snowflake documentation.

Install the application

Once you have configured the appropriate prerequisites in your Snowflake environment, you can proceed to install and configure the Matillion agent for Snowflake.

Create the Data Productivity Cloud agent

  1. Create an agent in the Data Productivity Cloud as described in Create an agent in your infrastructure.
  2. In the Cloud Provider drop-down, select Snowflake.
  3. The Deployment drop-down will display Native App. This can't be changed.

  4. On clicking Create agent, you will see the Agent details screen that shows you the parameters you will need for installing into Snowflake:

    • ACCOUNT_ID
    • AGENT_ID
    • MATILLION_REGION
    • Client ID
    • Client Secret

Install into Snowflake

  1. Ensure you are using the role created previously.
  2. Open the Snowflake Marketplace.
  3. Click Private Sharing in the left-hand menu.

  4. Locate Matillion Data Productivity Cloud in the list of apps, and click to select it.

    Note

    If the app is not listed, contact your Matillion representative for advice.

  5. Click Get to access the listing.

  6. If required, expand the Options panel in order to change:

    • The application name. This is optional, and you can keep the default name if you wish.
    • The warehouse to use for the installation. The warehouse should be the one created or selected previously.

  7. Click Get.

  8. Click Open to view the app, or Done to finish.

Warning

If the role you're using to install and set up the application does not have the appropriate permissions (OWNERSHIP or USAGE) to the selected warehouse, the agent will not start.

Configure the application

  1. Open the application.

  2. You will be prompted to grant access to allow the application to create a compute pool and a network rule allowing all access to ports 80 and 443. You must grant this access. Once the permissions have been granted, you will be able to activate the application.

    Note

    These are the minimum ports required by the agents. If you need to open additional ports for any reason, follow the instructions in How to open more ports.

  3. Under Privileges to objects, add the Snowflake warehouse that the agent will use to run queries within the account.

  4. Click Launch App. On the initial configuration screen, enter the details you previously obtained from the Agent details screen in the Data Productivity cloud:

    • Matillion Region
    • Account ID
    • Agent ID
    • Client ID
    • Client Secret

  5. Leave the Environment property unchanged.

  6. You can leave the Default Secrets Schema unchanged (recommended), or change it to another schema of your choice. Read Secrets in Matillion agent for Snowflake for further details.

    Note

    If you uninstall the application, any secrets created within the listed schema will be deleted.

  7. Click Submit. This will start the agent.

The agent status will initially show PENDING. This should eventually move to a READY state, meaning the agent is connected and available to run pipelines. If the agent doesn't change to a READY state, follow the guidance in the Troubleshooting document.

When the agent is in a READY state, you can begin creating Data Productivity Cloud projects or environments that use the new agent.

Connecting an environment

When creating a Data Productivity Cloud environment using Matillion agent for Snowflake, you must use the Snowflake "Format 1" account identifier as the environment's Account property.

To get the correct Format 1 identifier, run the following query within the account:

SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME();

For further details of Format 1, read Format 1 (preferred): Account name in your organization in the Snowflake documentation.

Installing multiple agents

You can run multiple agents, which may be a requirement for the level of performance and concurrency you need. You can do this by installing multiple Native App instances, with each instance running a single agent instance.

For a given Snowflake account, up to 10 Native App instances can be installed.

To increase the number of instances, open the installed app and click Add instance. Configure the new instance as described in the sections above.

You can use the same role and warehouse for each app instance, or you can create new roles and warehouses for each.

Uninstalling the application

You can uninstall the application from the Apps area within Snowflake. From your Snowflake Home screen, click Data Products → Apps. You must be using the role that originally installed the application.

Following an uninstall, some artefacts are left behind and must be manually removed. These are:

  • The <APPLICATION_NAME>_APP_DATA database.
  • The <APPLICATION_NAME>_ALL_EXTERNAL_ACCESS_EXTERNAL_ACCESS external integration.

To remove these, run the following commands:

DROP DATABASE <APPLICATION_NAME>_APP_DATA CASCADE;
DROP INTEGRATION <APPLICATION_NAME>_ALL_EXTERNAL_ACCESS_EXTERNAL_ACCESS;

Where <APPLICATION_NAME> is the name of the application set during installation. By default, this is MATILLION_DATA_PRODUCTIVITY_CLOUD.

Updating agents

The agent update process is controlled automatically by Matillion. By default, you will only be able to install the version shipped with the Native App.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement