Create a Maia runner in your infrastructure🔗
This topic explains how to create a Maia runner in your own infrastructure. For a discussion of why you might want to deploy in your own infrastructure rather than use a Matillion Full SaaS solution, read Maia runner overview. We currently support Maia runners running in Snowflake, AWS, or Azure environments.
There are two different types of Maia runner:
- Maia runner: Required to run Designer pipelines.
- Streaming runner: Required to run Streaming pipelines.
This article covers Maia runners only. For Streaming runners, read Create a Streaming runner.
The installation process is similar for both types of Maia runner, but you must select the correct Maia runner type for the type of pipeline you want to run. You can install both Maia runners in the same infrastructure if you intend to run both types of pipeline.
Read Scaling best practices to understand how to size your Maia runner for the level of performance you need.
Prerequisites🔗
- A Maia Foundation account. To register, read Registration. Once you have signed up, log in to the Maia Foundation.
- An account in either Azure or AWS to host the Maia runner.
Create a Maia runner🔗
- In the left navigation, click the Runners & Instances icon
. Then, select Runners from the menu. You will see a list of all Maia runners currently created. - Click Add runner.
- Click Maia Foundation.
-
Complete the following properties:
- Runner name: A unique name for your new Maia runner. Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
- Description: Optionally enter a brief description of the Maia runner.
- Version track: Select the version track that this Maia runner will use: Current or Stable. Read Version tracks for details of this option.
- Cloud provider: The cloud platform that the Maia runner will be deployed to. Currently, Snowflake, AWS, and Azure are supported.
- Deployment: The supported deployment method for the given cloud provider. Currently, Native App for Snowflake, Fargate for AWS, and Container App/ACI for Azure are supported.
- Auto update: Select whether you want your Maia runner to automatically update or not. Read Auto update for details of this option.
-
Click Create runner.
Set up the Maia runner in your cloud infrastructure🔗
When created with the above process, the Maia runner needs to be set up in your cloud infrastructure. There are several different ways of doing this, and you can use whichever method suits you:
-
For Snowflake Maia runners:
-
For AWS Maia runners:
-
For Azure Maia runners:
To complete these processes, you will require certain details of the created Maia runner. To obtain these details, locate the Maia runner in the list of Maia runners, and click the three dots ... next to it, then click Runner details. The parameters and values in the sections Agent image URI, Agent environment variables, and Credentials are required when configuring your Maia runner in your cloud infrastructure.
Check Maia runner status🔗
After deploying the Maia runner in your cloud infrastructure, you should return to Maia Foundation to verify that it's correctly connected and running.
- Click the Maia Foundation icon in the top left of Maia Foundation, then click Manage runners.
-
Locate the Maia runner in the list and check the status:
- Running: The Maia runner is connected and ready for development and pipeline tasks.
- Unknown: The Maia runner is in an unknown state.
- Pending: The Maia runner has been created but has not yet connected to Maia Foundation.
- Stopped: The Maia runner has been stopped.
-
When the Maia runner status shows Running, it's ready to use.
Restarting Maia runners🔗
There are a number of reasons why you may need to restart your Maia runner, such as receiving Maia runner updates.
To restart a Maia runner, follow the steps in Restart a Maia runner. In some cases, you may need or choose to pause the Maia runner before restarting.
Deleting Maia runners🔗
To delete a Maia runner from Maia Foundation, locate the Maia runner in the Runners list, then click the three dots ... → Remove runner.
Deleting the Maia runner from the Runners list doesn't remove the underlying Snowflake, AWS, or Azure resources. You should go into the Snowpark Container Services, AWS Console, or Azure Portal, and clean up any resources that you no longer require.
- Snowflake: Use Snowpark Container Services to manage and clean up any Snowflake resources associated with the Maia runner.
- AWS: Use the AWS Console to identify and delete any resources associated with the Maia runner, such as EC2 instances and IAM roles.
- Azure: Use the Azure Portal to locate and remove any related resources, such as virtual machines and resource groups.
Warning
Deleting a Maia runner that is currently running may interrupt scheduled pipelines or pipelines that are currently running. Therefore, you should always stop the Maia runner service before deleting it.
Using the Maia runner to develop and run pipelines🔗
To use a Maia runner for developing and then running pipelines, you need to select the Maia runner when you create an environment. See below for details of how to make Maia runners available to specific environments.
You can change which Maia runner an environment uses at any time by editing the environment and selecting a different Maia runner. This is safe to do, as the change will only affect pipelines that are run after the change. So:
- Your open Designer session won't change.
- Any pipelines that are currently running will continue to run using the old Maia runner.
- Any existing schedules will continue to run using the old Maia runner. To use the new Maia runner, you would have to change the schedule.
Restricting Maia runners🔗
You can restrict which Maia runners are available to a specific project or environment. This is useful if you have multiple Maia runners and want to ensure that a project only uses specific ones.
An unrestricted Maia runner (the default state when a Maia runner is created) can always be used by any project or environment. A restricted Maia runner can only be used by the projects and environments you specify. If you restrict a Maia runner but don't specify any projects, the Maia runner won't be accessible to any project.
For details of how to restrict a Maia runner, read Restricting Maia runners.