Skip to content

Kubernetes deployment guide for Matillion agents🔗

This document provides an overview of Kubernetes deployment options for Matillion Data Productivity Cloud agents. For cloud-specific architecture decisions and implementation details, see:

Kubernetes deployment is one of several options for running Data Productivity Cloud agents in your own infrastructure (Hybrid SaaS model). This approach offers:

  • Standardized deployment via Helm charts.
  • Native cloud authentication (AWS IRSA, Azure Managed Identity).
  • Horizontal pod autoscaling (available on EKS and AKS, not available in ECS/ACI alternatives).
  • Platform portability across AWS, Azure, and self-managed Kubernetes clusters.

Note

This document is not a step-by-step guide or tutorial, but rather an overview of what a Kubernetes administrator needs to know to deploy and manage Data Productivity Cloud agents. We assume you have a working knowledge of Kubernetes, and have existing Kubernetes infrastructure, before attempting this deployment.

When to choose Kubernetes🔗

Choose Kubernetes for Data Productivity Cloud agent deployment when:

  • You have existing Kubernetes expertise and infrastructure.
  • You need autoscaling capabilities (EKS/AKS provide HPA support).
  • You want standardized deployment across multiple cloud providers.
  • You require integration with existing Kubernetes monitoring and security tooling.

Alternatives to consider:

Prerequisites🔗

Required tools🔗

Matillion account🔗

Before deploying, create an agent in the Data Productivity Cloud console to obtain:

  • Account ID
  • Agent ID
  • OAuth Client ID and Secret
  • Region (us1 or eu1)

For details, read create an agent.

Cloud provider deployment guides🔗

Amazon EKS🔗

For AWS deployments, the EKS guide covers:

  • IAM Roles for Service Accounts (IRSA) for credential-free AWS access.
  • VPC architecture decisions (new vs. existing, public vs. private cluster).
  • Worker node sizing and autoscaling strategies.
  • CloudWatch monitoring integration.
  • EKS-specific cost optimization.

Read the EKS deployment guide for details.

Azure AKS🔗

For Azure deployments, the AKS guide covers:

  • Workload Identity/Managed Identity for credential-free Azure access.
  • VNet architecture decisions (new vs. existing, public vs. private cluster).
  • Node pool sizing and autoscaling strategies.
  • Azure Monitor integration.
  • AKS-specific cost optimization.

Read the AKS deployment guide for details.

Self-managed Kubernetes🔗

For self-managed Kubernetes clusters (customer-managed control plane in cloud or other environments) you need to consider:

Container registry: Use AWS ECR Public (no authentication required).

  • public.ecr.aws/matillion/etl-agent:stable
  • public.ecr.aws/matillion/etl-agent:current

Authentication: OAuth credentials stored in Kubernetes Secrets.

Deployment: Use the standard Helm chart from the Deployment Library.

Other considerations:

  • Native cloud authentication (IRSA, Workload Identity) not available for self-managed clusters.
  • Use OAuth credentials for Matillion control plane authentication.
  • Ensure outbound HTTPS connectivity to Matillion regional endpoints.
  • Horizontal Pod Autoscaler supported (requires metrics server).
  • Cluster Autoscaler availability depends on your infrastructure provider.

Note

For managed Kubernetes services on other cloud providers, contact Matillion for platform availability.

Container images🔗

Agent images are available in cloud-specific container registries:

Cloud provider Image repository Tags
AWS (EKS) public.ecr.aws/matillion/etl-agent :stable, :current
Azure (AKS) matillion.azurecr.io/cloud-agent :stable, :current
Self-managed Kubernetes public.ecr.aws/matillion/etl-agent :stable, :current

Image Tag Strategy:

  • :stable - Slower release cycle, maximum stability.
  • :current - Faster release cycle, earlier access to new features.

Both tags are production-ready. Choose based on your preference for stability vs. early feature access.

The tag you select here must match the version track configured for this agent in Agent Manager. Mismatching the two can result in unexpected version behavior.

Common Kubernetes deployment components🔗

All Kubernetes deployments use these same core components.

Helm charts🔗

Standard Helm chart deploys:

  • Agent pods (deployment with configurable replicas).
  • Kubernetes service (Prometheus metrics endpoint on port 8080).
  • ConfigMaps (agent configuration).
  • Secrets (OAuth credentials for Matillion control plane).
  • ServiceAccount (for cloud-native authentication where available).

Read the Helm charts documentation.

Prometheus metrics🔗

Agents expose Prometheus-compatible metrics at /actuator/prometheus on port 8080:

  • app_version_info: Agent build version.
  • app_agent_status: Running status (1=running, 0=stopped).
  • app_active_task_count: Currently executing tasks.
  • app_active_request_count: Active HTTP requests.
  • app_open_sessions_count: Open data warehouse connections.

The Helm chart includes service discovery annotations for automatic Prometheus scraping.

Security standards🔗

All Kubernetes deployments implement pod security best practices:

  • Run as non-root user (UID 65534).
  • Read-only root filesystem.
  • No privilege escalation.
  • All Linux capabilities dropped.
  • Seccomp profile applied (RuntimeDefault).

Pre-deployment validation🔗

The deployment library includes automated validation scripts in agent/helm/checks/:

  • run-check.sh - orchestrator that:

    • Auto-discovers agent pods using Helm chart labels.
    • Performs cluster-level security scans.
    • Executes in-pod validation.

  • pre-deployment-check.sh - validator that checks:

    • Python 3 and Java runtime availability.
    • Filesystem permissions.
    • Environment variables.
    • Network connectivity to Matillion control plane.
    • Security agents that might interfere (Crowdstrike, Prisma Cloud).

Usage:

./agent/helm/checks/run-check.sh --namespace matillion --release matillion-agent

The output uses color-coded results (🟢 PASS, 🟡 WARN, 🔴 FAIL), with exit code 0 for success and 1 for failure.

Next steps🔗

© Copyright 2026 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement