Setup guide - Hybrid SaaS Amazon Redshift on AWS

This document describes the necessary steps to follow to set up your first working project in the Data Productivity Cloud for the following configuration options:

Deployment type:

Hybris SaaS

Cloud platform:

AWS

Cloud data warehouse:

Amazon Redshift

---

Prerequisites

AWS requirements

• An AWS account with privileges/permissions to use the CloudFormation template.
• An AWS user account or role with permissions to create:
  • ECS clusters.
  • Task definitions.
  • IAM roles for task execution, including as a minimum AWSServiceRoleForECS. If your account doesn't have this role, create it following the instructions in Creating a service-linked role for Amazon ECS.
  • S3 buckets.
  • CloudWatch log groups.
  • AWS Secrets Manager.
• Access to the following AWS resources:
  • A virtual private cloud (VPC).
  • A private subnet.
  • A security group, minimally allowing access.
• Allowed access to the IP addresses listed in Allowing IP addresses.

Amazon Redshift requirements

• An Amazon Redshift account with the following information:
  • The physical address of the Amazon Redshift leader node. This will be either a name or an IP address.
  • The port Amazon Redshift communicates on, configured when setting up your Amazon Redshift cluster.
  • Your Amazon Redshift username.
  • The Amazon Readshift secret key created in AWS Secrets Manager.

Connectivity requirements

• Access enabled for the IP addresses listed under the Hybrid SaaS section of Allowing IP addresses.

Git requirements

If you choose to use your own Git provider instead of the Matillion-hosted Git option, you need the following:

• The Matillion Git app installed in your organization's account with one of the supported Git providers:
  • GitHub.
  • Azure DevOps.
  • GitLab.
  • Bitbucket.

---

Setup steps

1. Register for a Data Productivity Cloud account.
2. Create accounts for users and admins who will be active in the Data Productivity Cloud.
3. Create an agent in the Data Productivity Cloud.
4. Deploy a Fargate agent in AWS using CloudFormation.
   • If you have multiple VPCs, or link your VPCs to on-premises environments for accessing privately hosted databases, APIs, and other data sources, the agent's VPC's CIDR and subnet's IP range should be compatible and properly linked with other networks, where required.
5. Create a project, making the following choices:
   • Select Advanced settings.
   • Select the agent you created and deployed previously.
   • Select the Git provider you wish to use.
6. Create an environment using your Redshift credentials.
7. Set up secret definitions for passwords, API keys, and tokens.
8. Create a Git branch in which to begin pipeline work.
9. Create your first pipeline.