Hybrid SaaS setup guide

The Data Productivity Cloud needs access to both a cloud data warehouse and a cloud platform. This topic outlines the accounts and resources you need for your cloud data warehouse and cloud platform before you can set up the Data Productivity Cloud as a Hybrid SaaS solution for creating your data pipelines.

The following table shows the data warehouse and platform combinations currently supported by the Data Productivity Cloud.

	Snowflake	Databricks	Amazon Redshift
AWS	✅	✅	✅
Azure	✅	✅	❌

---

Cloud data warehouse

Snowflake

You need a Snowflake account, with the following information:

• Your Snowflake account name and region (found in the URL you use to log in to Snowflake).
• Your login credentials (either username and password or Snowflake private key).
• A Snowflake role, warehouse, database, and schema.

Databricks

You need a Databricks account, with the following information:

• Your Databricks instance name.
• Your Databricks personal access token.
• A Databricks cluster endpoint, Unity Catalog, and schema.

Amazon Redshift

You need an Amazon Redshift account, with the following information:

• Your Redshift cluster endpoint and port.
• Your Redshift credentials (username and password).
• A Redshift database and schema.

---

Cloud platform

AWS

You need an AWS account with privileges/permissions to use the CloudFormation template and the ability to create the following AWS resources:

• ECS Clusters.
• Task Definitions.
• IAM roles for task execution. For more information, read IAM roles for Amazon ECS.
• S3 Buckets.
• CloudWatch Log Groups.
• AWS Secrets Manager.
• The AWSServiceRoleForECS role.

For a visualization of the deployed resources, read Created resources in agent installation using a CloudFormation quick-create link.

Your AWS account also needs access to the following AWS resources:

• A virtual private cloud (VPC).
• A subnet (private).
• A security group, minimally allowing access. For more information, read Allowing IP addresses.
• A secrets manager to store your passwords, API keys, bearer tokens, and so on.

Azure

You need a Microsoft Azure account with the following privileges/permissions:

• Create:
  • Subnet
  • Managed identity
  • Key Vault
  • Log Analytics Workspace
  • Container App & Container App Environment
• Modify:
  • Subnet delegation
• Permissions:
  • Role assignments in the resource group
  • Key Vault

Your Azure account also needs access to the following Azure resources:

• A resource group.
• A virtual network.
• A key vault (optional; a new key vault can be created as part of the Data Productivity Cloud agent installation).

---

Launching process

Before you begin using the Data Productivity Cloud, you need to:

• Register for a Hub account
• Set up and deploy the agent
• Store required credentials

Getting started in the Matillion Hub

• Register for a Matillion Hub account. For more information, read Registration. By default, newly registered Hub accounts start the 14-day free trial. For more information, read Free trial.
• Create accounts in the Hub for all users and admins who will be active during the trial.
• It's recommended that you set up a Pay-Go subscription. This subscription isn't required for the period of your trial, but having a subscription in place will prevent system lockout when the trial period expires.

Agent setup

• Create a new Data Productivity Cloud agent.
• Deploy the agent in your infrastructure:
  • Deploy a Fargate agent in AWS using the recommended CloudFormation method.
  • Deploy a Container App agent in Azure using the recommended ARM template.

Check out the video walkthrough to learn how to create a Data Productivity Cloud agent and see how you can run AWS CloudFormation.

Required credentials

• Matillion lets you create referential secret definitions to store passwords, credentials, and secrets. You'll use secret definitions for connecting to any cloud data platforms, APIs, and data sources (Facebook, Salesforce, and so on) using passwords, API keys, and bearer tokens.
• If you're using cloud storage, such as Amazon S3, Azure Blob storage, or Google Cloud Storage, you are required to set up cloud provider credentials to connect your cloud provider to your Data Productivity Cloud project.

---

Create a project

Once you have satisfied the prerequisites for getting started:

1. Log in to Matillion Hub or register for an account.
2. Select your account.
3. From the dashboard, click ☰ → Designer.
4. Click Add new project and create your new project.