Instance sizes
If you're a registered customer of the Hub, follow the instance sizes recommendations below for launching a Matillion ETL instance through the Hub.
This document explains the factors that affect the performance of your Matillion ETL instance. There are three important factors that determine Matillion ETL instance performance:
- CPU: The number of virtual CPU cores.
- Memory: The available memory to process instance data.
- Network: The quality, speed, and capacity of your network connection.
Both your edition and the physical requirements of your Matillion ETL instance, as well as its users, are key factors to consider when choosing your instance size. We will discuss these factors in detail to help you make the best decision for your needs.
Note
Matillion ETL is tested to run on instances with processors using x86_64 instruction sets and should be used as such. Other instance types—such as those using ARM processors—should be avoided and in many cases can't be selected for use with Matillion ETL.
Instance specification
CPU
The number of vCPU cores is directly related to the maximum performance of your Matillion ETL instance. Broadly speaking, more cores result in better performance and insufficient cores could result in computational delays and even fatal processing errors, particularly under heavy scheduling or CDC. Different editions provide more features which will in turn require greater CPU usage.
The number of vCPUs dictates the number of tasks an instance can execute at one time. Matillion ETL can run 16 jobs at a time, and each job can run 2 parallel tasks per vCPU. The total number of concurrent tasks per vCPU is 32 (16 Jobs × 2 Tasks × vCPU). For example, a 2 vCPU machine can handle a total of 64 parallel tasks. For more information, read Job Concurrency.
Instances with more vCPUs consume more credits per hour. For more information, read Credit Consumption Terms. As such, it's important that you choose the appropriate CPU for your needs.
Memory
Insufficient memory will cause your Matillion ETL instance to fail. The required memory directly scales with the number of tasks being processed and the number of concurrent users. It's recommended that you increase your memory if you want to have more concurrent users.
Note
Please contact sales@matillion.com if you require further help and advice when choosing the correct Matillion ETL instance size for your needs.
Credit consumption
The size of your Matillion ETL instance has a direct effect on your credit consumption in the Hub. Matillion ETL instances with more vCPUs consume more credits per hour, and credit consumption is per vCPU per hour. Choose carefully when considering your edition and making your instance selections. Read Credit consumption dashboard for more information.
Note
- For information about Credits in the Hub, and how they're consumed, read 2. Matillion ETL Instances Usage in Credit Consumption Terms.
- The cost of your AWS or Azure payment subscription is not included in your Hub billing. Read Credit consumption dashboard for more information.
Sizing recommendations
Matillion ETL will run on many different instance sizes providing they use an x86_64 instruction set. The following tables give just a few example instance sizes for AWS and Azure. Please read, Amazon EC2 and Azure Virtual Machines for more specific information.
When making your instance size selection, we recommend starting with the minimum CPU and memory that satisfies your requirements, and increase as needed.
The minimum Matillion ETL instance size for AWS and Azure is: 2vCPU and 4GiB of memory.
More vCPUs means higher credit consumption. Additionally, the cloud cost of your VM is not included in credit pricing and generally cost more to run with higher specifications.
AWS
| Name | vCPU | Concurrent Tasks | Memory (GiB) | Notes | Suitable For* |
|---|---|---|---|---|---|
| m5.large | 2 | 64 | 8 | A balanced starter instance with relatively low AWS costs and credit consumption. | Teams of 2—5 persons. |
| m5.xlarge | 4 | 128 | 16 | A larger instance leaning toward heavy production use. At 4 vCPU, it can handle double the tasks of an m4.large with 2vCPU and will consume credits twice as quickly. | Teams of 6—12 persons. |
| m5.2xlarge | 8 | 256 | 32 | An extreme production instance for heavy workloads and high concurrent user count. Consumes 8 credits per active hour, 4 × more than the m5.large. | Teams of 6—25 or a highly-available production environment. |
Azure
| Name | vCPU | Concurrent Tasks | Memory (GiB) | Notes | Suitable For* |
|---|---|---|---|---|---|
| D2s_v4 | 2 | 64 | 8 | A balanced starter instance with relatively low Azure costs and credit consumption. | Teams of 2—5 persons. |
| D4s_v4 | 4 | 128 | 16 | A larger instance leaning toward heavy production use. At 4 vCPU, it can handle double the tasks of D2s_v4 with 2vCPU and will consume credits twice as quickly. | Teams of 6—12 persons. |
| D8s_v4 | 8 | 256 | 32 | An extreme production instance for heavy workloads and high concurrent user count. Consumes 8 credits per active hour, 4 × more than the D2s_v4. | Teams of 6—25 or a highly-available production environment. |
GCP
| Instance Size | Max. Concurrent User | Max. Environments | Threads | GCP VM Size | vCPUs | Memory (GiB) | Enterprise Mode Available? |
|---|---|---|---|---|---|---|---|
| Medium-M | 2 | 6 | 4 | n2-standard-2 | 2 | 8 | No |
| Large-L | 5 | 15 | 8 | n2-standard-4 | 4 | 16 | Yes |
| XLarge-XL | 12 | 36 | 16 | n2-standard-8 | 8 | 32 | Yes |
* The given team size estimates depend greatly on how you are using your instance and are there to give a more practical idea of an instance's capabilities. Consult your sales representative or Matillion Support for more precise guidance.
Choosing your instance size
Use the following steps to choose a suitable instance size for your Matillion ETL instance:
- Log in to the Hub. If you are a new customer, you must register for a Hub account before you can launch a Matillion ETL instance.
- Follow the steps for creating a Matillion ETL instance.
- Once you have been re-directed to your cloud provider's console page, you will be required to configure your new Matillion ETL instance. Follow the steps listed in your chosen console. For more information, read Launching Matillion ETL overview.
- Refer to AWS instance recommendations, Azure instance recommendations, or GCP instance recommendations for recommendations on the instance size that best suits your needs. For more information, read Amazon documentation on instance sizes, Azure documentation on instance sizes, or GCP documentation on instance sizes, respectively.
- Review your configuration settings in the cloud provider console, and create your Matillion ETL instance.