Skip to content

Connecting agents via AWS PrivateLink

Editions

This feature is for customers on our Enterprise edition only. Visit Matillion pricing to learn more about each edition.

AWS PrivateLink is an AWS service that allows you to connect services such as the Data Productivity Cloud to your own AWS virtual private cloud (VPC) via a secure, private connection. Using AWS PrivateLink, no traffic is exposed to the public Internet when it travels between two different VPCs. For further details of the service, read What is AWS PrivateLink?.

Prerequisites

AWS PrivateLink connectivity to the Data Productivity Cloud requires a Matillion Hybrid SaaS agent running on AWS.

Note

Use of AWS PrivateLink will incur a cost with AWS. For details, read AWS PrivateLink pricing.

Cross-region support

AWS PrivateLink can enable connectivity to the Matillion Data Productivity Cloud region from a different AWS region. To do this, you need to:

  1. Configure a VPC in the region in which the endpoint service resides.
  2. Create an inter-region VPC peering connection from the PrivateLink connected VPC to the remote VPC.

For further details, read What is VPC peering?

The Data Productivity Cloud will reside in one of the following regions:

  • eu-west-1 (eu1)
  • us-east-1 (us1)

Apply a security group

You will need to apply a security group to control who can access the Elastic Network Interface (ENI) and the target application.

Create the VPC endpoint

Note

Before creating the AWS PrivateLink endpoint, you must have created the VPC and subnets you wish to use.

  1. Log in to the AWS Console.
  2. Type VPC in the search bar, and click VPC (it should be the top search result).
  3. Under PrivateLink and Lattice in the left-hand menu, click Endpoints.
  4. Click Create endpoint.
  5. On the Create endpoint screen, select Endpoint services that use NLBs and GWLBs.

  6. For Service name, enter the appropriate name for your Matillion Data Productivity Cloud region, as follows:

    Region Service name
    eu-west-1 com.amazonaws.vpce.eu-west-1.vpce-svc-05d76c667b72daf2d
    eu-east-1 com.amazonaws.vpce.us-east-1.vpce-svc-0e24b7e2cd2b24e3f

  7. Click Verify service and ensure you see a "Service name verified" response.

  8. From the VPC drop-down, select the VPC in which your Data Productivity Cloud agent is located.
  9. In the list of Subnets, select the VPC subnets that your Data Productivity Cloud agent uses.
  10. Click Create endpoint.
  11. Copy the DNS names listed under the details of the new endpoint. These will be needed to configure Route 53, as described below.

Configure DNS requirements

Create a hosted zone in Amazon Route 53 and create alias records that point at your VPC endpoints. Use the DNS names that you noted when creating the endpoint, above.

Read Routing traffic to an Amazon Virtual Private Cloud interface endpoint by using your domain name for more details.

The DNS entries used by the Data Productivity Cloud are:

  • opentelemetry.eu1.privatelink.matillion.com
  • api.agent-gateway.eu1.privatelink.matillion.com

Authentication

Authentication will be handled by Keycloak at https://keycloak.core.matillion.com, where a token will be generated. This will be the only connection over the public Internet prior to connecting to services over AWS PrivateLink.

Configure the agent

To enable the Data Productivity Cloud agent to use AWS PrivateLink, you need to add the environment variable MATILLION_PRIVATELINK_ENABLED = TRUE. This requires you to create a new task revision and restart the agent service. Ensure that there are no pipelines actively using the agent before you begin this process.

  1. Log in to your AWS console.
  2. In the AWS console, type Elastic Container Service in the search bar, and select that service.
  3. In the left-hand menu, click Task definitions.
  4. Select the task defintion for your agent and click Create new revision.

  5. On the Create new task definition revision screen, under Environment variables, add the following:

    Key Value type Value
    MATILLION_PRIVATELINK_ENABLED Value TRUE

  6. Click Create.

  7. Return to Update service.
  8. Select the latest task definition and click Update.
© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement