Connecting to an RDS in a private VPC
Overview
There are following scenarios for accessing an RDS DB instance by the Matillion Instance.
- A DB Instance in a VPC Accessed by Matillion Instance in the Same VPC.
- Create a VPC security group for RDS DB instances to be in. This security group can be used to restrict access to the RDS DB instances.
- Create a VPC security group for Matillion instances to be in. This security group can, if needed, allow access to the Matillion instance from the internet by using the VPC's routing table.
- Create custom rules in the security group for RDS DB instances that allow connections from the security group you created for the Matillion instances. This would allow any member of the security group to access the DB instances.
- A DB Instance in a VPC Accessed by Matillion Instance in a Different VPC
Important Information
The simplest way to manage access between Matillion instances and RDS DB instances in the same VPC is to do the following:
When your DB instance is in a different VPC from the Matillion instance , you can use VPC peering to access the DB instance.
A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IP addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, with a VPC in another AWS account, or with a VPC in a different AWS Region.
This guide looks at how to set up VPC Peering to allow Matillion instance available in one VPC to use VPC Peer connection to access the data from the another VPC (Amazon RDS Databse). The concept is the same for any data held within the VPC.
For security reasons, RDS database might be held in a Private Subnet of the VPC that doesn't allow public access. Whereas, Matillion instance, which might be in a Public Subnet of the another VPC, can still connect to this RDS database using VPC Peering.
The following diagram depicts what final architecture will look like if Matillion Instance and RDS Database placed in different VPC.
Create a VPC
Navigate to the Amazon VPC Dashboard in the AWS Management Console. Click Create VPC and fill in all the details required to create a VPC.
Please Note
A VPC in AWS is private if it doesn't have an Internet Gateway or an NAT Gateway that can be used to connect into it. This can be set up from the AWS Console.
Create and Attach Internet Gateway
On the Internet Gateways tab of the VPC Dashboard, create a new Internet gateway. Attach the gateway to the newly created VPC.
Create a Custom Route Tables
On the Route Tables tab of the VPC Dashboard, click Create route table.
Add a route to the route table for the Internet gateway by clicking Edit routes tab.
Please Note
The route table for the VPC will only have an entry for local routes.
Create Public and Private Subnet
Create a subnet in the required availability zone by clicking Create subnet
Change the route table for the previously created subnet from the main route table to the custom route table, if needed. And , click Edit route table association
Create Security Group
Create a Security Group for the Matillion instance to be provisioned in the public subnet.
Authorize inbound traffic from your local IP address. The default outbound rules should be fine.
Similary, create a security group for the RDS instance to be provisioned in the private subnet.
Peering Connection
A Peering Connection is required to route traffic between two VPCs. These VPCs can be in the same AWS account, or in different accounts if required.
Create VPC Peering Connection
To create Peering connection, navigate to Peering Connection to the left of the menu and click Create peering connection.
Accept VPC Peering Connection
This peering request will appear in Peering Connections and now needs to be accepted by the account the VPC to peer with is in:
VPC Peering Connection Status
Check that the status is now active on both VPCs.
Some additional steps to complete the connection
Once the Peering Connection completed but still we need to follow some additional steps to complete the connectivity between two VPCs.
Create Routes on both the ends.
On VPC 1 where Matillion resides, we need to provide CIDR for destination VPC (VPC 2) - where RDS resides, and target should be "Peering Connection". Click on Save routes.
For the VPC that Matillion is in, all traffic to the IP associated with the Private VPC needs to be directed to the Peering Connection:
For the RDS VPC, all traffic to the Matillion instance needs to be directed to use the Peering Connection. This was done by directing all other traffic to the Peering Connection but this IP range could be limited:
Matillion now has a private route to the RDS instance without having to use the internet. The RDS Query and RDS Bulk Load components will now work as expected from the Matillion instance.
Please Note
Allow respective CIDR on the "Security Group" of particular service so that they can establish the connection to that service.