Getting Started in the Community Forums/Discussions Unanswered Questions Roadmap and Ideas User Groups & Events

Academy Get Certified Instructor-Led Training

Shared Jobs Shared Connectors Request a Component

Data Productivity Cloud Docs Matillion ETL Docs

Blogs Request a Demo Services Status

Knowledge Base My Support Cases About Support

Initializing search

Home
Matillion ETL
Data Productivity Cloud
Style guide
Data Loader CDC

Matillion Docs

Home
Matillion ETL
Matillion ETL
- Using the site
  Using the site
  - Matillion ETL documentation
  - Getting support
- Matillion Hub
  Matillion Hub
- Matillion ETL basics
  Matillion ETL basics
  - Accessing your instance
    Accessing your instance
    
    Accessing the Matillion ETL Client (Amazon EC2)
    
    Accessing the Matillion ETL Client (Google Cloud Platform)
    
    Accessing the Matillion ETL Client (Microsoft Azure)
  - Components
    Components
    
    Assert components in Matillion ETL
    
    Expression editors
    
    Incremental load tools
    
    URL safe characters
  - Jobs
    Jobs
    
    Generate Job Documentation
    
    Jobs
    
    Job concurrency
    
    Notes
    
    Shared jobs
    
    Task management
  - Environments
    Environments
    
    Environments
    
    Manage Stages
    
    Multiple environment connections
    
    Table properties
    
    Creating a Snowflake Zero-Copy Clone
    
    Create External Schema
  - Variables
    Variables
    
    Variables
    
    Component Exports
    
    Date and time methods
    
    Environment Variables
    
    Grid variables
    
    Job Variables
  - Matillion ETL product overview
  - Enterprise mode
  - Scope of Matillion ETL features
  - UI and basic functions
  - Admin menu
  - Schema
  - Notices
  - Search tab
  - Performance monitor
  - Views
  - Project collaboration
  - Software versions
  - Glossary
- Administration
  Administration
  - Manage Interpreters
  - Administrative functions
    Administrative functions
    
    Audit log
    
    How to place restrictions on Bash and Python components
    
    How to configure Catalina log rotation
    
    Control session timeout expiration
    
    Product improvement metrics
    
    Restart server
    
    Publicly available warning
  - Backups
    Backups
    
    Backups (AWS)
    
    Backups (GCP)
    
    Backups (Azure)
    
    Manage backups
  - Cloud data platform configuration
    Cloud data platform configuration
    
    Snowflake configuration for Matillion ETL
    
    Redshift configuration for Matillion ETL
    
    Delta Lake on Databricks configuration for Matillion ETL
    
    Configuring an AWS VPC
    
    How to generate a new Databricks token
  - Connectivity
    Connectivity
    
    Configuring Matillion ETL to use a Proxy
    
    Setting up an external connection to a Matillion database
    
    Manage connections
    
    Matillion ETL access ports
    
    Configuring a connection from Matillion ETL to Matillion Hub
    
    Setting up Matillion ETL in a private VPC
  - Drivers
    Drivers
    
    Adding a third-party JDBC driver
    
    Getting started with the API driver in Matillion ETL
    
    SAP Hana JDBC driver installation for Matillion ETL
    
    Manage Database Drivers
  - Permissions
    Permissions
    
    Groups and Permissions
    
    Permissions list
  - Preview Labs
    Preview Labs
    
    Preview Labs
  - SSL
    SSL
    
    SSL commands
    
    How to add a certificate chain file for SSL configuration
    
    Recreating self-signed SSL certificates on a Matillion ETL instance
    
    SSL Configuration FAQ
  - Updating and migrating
    Updating and migrating
    
    Updating and migrating overview
    
    Migration
    
    In-place update
    
    Updating to a specific release
    
    Updating a high availabilty cluster
  - User configuration
    User configuration
    
    User configuration
    
    Stateless authentication
    Stateless authentication
    
    Configuring stateless authentication
    
    Internal security (stateless authentication)
    
    OpenID integration (stateless authentication)
    
    LDAP integration (stateless authentication)
    
    Stateless authentication rollback steps
    
    OpenID
    OpenID
    
    OpenID overview
    
    Azure Active Directory OpenID setup
    
    Google OpenID setup
    
    Microsoft Active Directory OpenID setup
    
    Okta OpenID setup
    
    OneLogin OpenID setup
    
    OpenID setup
    
    Troubleshooting OpenID setup
    
    LDAP
    LDAP
    
    LDAP integration
    
    Okta LDAP Configuration
    
    Read-only users
    
    Reverting from external to internal security
    
    SAML in Matillion ETL
  - Matillion ETL security best practices
  - Using CSRF tokens to safeguard Matillion ETL instances
  - Installing DBT on Matillion ETL
  - Connecting to an external PostgreSQL database
  - Automatic security updates
  - Manage optional features
- Launching Matillion ETL
  Launching Matillion ETL
  - Launching Matillion ETL overview
  - Launching Cloud Platform Permissions
  - Launching - AWS
    Launching - AWS
    
    Launching Matillion ETL using CloudFormation Templates
    
    List of CloudFormation Templates
    
    Launching Matillion ETL using Amazon Machine Image
    
    Launching Matillion ETL for Delta Lake on AWS
    
    Launching troubleshooting (AWS)
  - Launching - Azure
    Launching - Azure
    
    Overview
    
    Launching Matillion ETL from Azure Marketplace
    
    Launching Matillion ETL using an Azure ARM Template
    
    Launching Matillion ETL for Delta Lake on Azure
    
    Launching troubleshooting (Azure)
  - Launching - GCP
    Launching - GCP
    
    Launching Matillion ETL for Snowflake - GCP
    
    Launching Matillion ETL for BigQuery - GCP
  - High Availability (HA)
    High Availability (HA)
    
    Launching a Matillion ETL HA Cluster via AWS
  - RPM installations
    RPM installations
    
    Installing Matillion ETL using the Universal Installer (RPM install)
  - Non-Hub existing customers
    Non-Hub existing customers
    
    Overview
    
    Launching Matillion ETL using CloudFormation Templates
    
    List of Snowflake Launch Templates
    
    Migrate from Snowflake Partner Connect to Matillion ETL for Snowflake
    
    Launching Matillion ETL from AWS Marketplace
    
    Launching Matillion ETL for Redshift from AWS Marketplace
    
    List of Redshift Launch Templates
    
    Converting to be an Annual Customer
    
    Launching Matillion ETL via Azure CLI
    
    Finding and Launching Matillion BYOL Images
    
    Instance Sizes Guide
  - Applying a licence
- Project menu
  Project menu
  - Create Project
    Create Project
    
    Create Project (Snowflake)
    
    Create Project (Delta Lake on Databricks)
    
    Create Project (Amazon Redshift)
    
    Create Project (Google BigQuery)
    
    Create Project (Azure Synapse Analytics)
  - Manage functions
    Manage functions
    
    Manage Project
    
    Manage Credentials
    
    Manage Extract Profiles
    
    Manage Passwords
    
    Manage Query Profiles
    
    Manage OAuth
    
    Manage Schedules
    
    Manage Sequences
    
    Manage Shared Jobs
    
    Manage Versions
    
    Manage Webhook Paylods
    
    Integrating Matillion ETL with secret managers
    
    Manage API Profiles wizards
    
    Manage External File Sources
  - Queue Messaging
    Queue Messaging
    
    Azure Queue Message configuration
    
    Manage Pub-Sub configuration
    
    Manage SQS Configuration
  - CDC
    CDC
    
    Manage CDC
    
    Enable Manage CDC
    
    Configuring a source database for CDC
    
    DMS migration instances
  - Git integration
    Git integration
    
    Git Integration with Matillion ETL
    
    Git Integration Frequently Asked Questions
    
    When to choose Git
    
    MergeManager
  - API Profiles
    API Profiles
    
    API Profiles Overview
    
    API Query Profiles
    
    API Extract Profiles
    
    API Connector Wizard
    
    API Profiles - Pagination
    
    API Profiles - Parameters
    
    API Profiles - Authentication
    
    API Profiles - RSDs
    
    API Profiles Example - MongoDB and DynamoDB
    
    API Profiles Example - Jira Cloud
    
    API Profiles Example - Salesforce Lightning
  - Switch Project
  - Using data structure variables
  - Change My Password
  - Extract to new job
  - Task History
  - Import - Export
  - Input data report
  - Manage Error reporting
  - Project user access
  - Recycle Bin
  - Job references
  - Job reference renaming
  - Databricks job compute configuration
- Orchestration
  Orchestration
  - Data staging components
  - Data stagers - support
  - Data models
  - Connecting to external services securely
  - Assert components
    Assert components
    
    Assert External Table
    
    Assert Scalar Variables
    
    Assert Table
    
    Assert View
  - Connectors
    Connectors
    
    Connectors overview
    
    Acquiring Azure Credentials
    
    Amplitude
    Amplitude
    
    Amplitude Extract
    
    Amplitude Extract authentication guide
    
    Anaplan
    Anaplan
    
    Anaplan Bulk
    
    Anaplan Bulk authentication guide
    
    API Queries
    API Queries
    
    API Query
    
    API Extract
    
    API Query functions
    
    Create your own Matillion ETL Extract connector
    
    Populating parameters with API Query
    
    Apache
    Apache
    
    Apache Hive Query
    
    Apache Spark SQL Query
    
    Azure
    Azure
    
    Azure SQL Query
    
    Azure Cosmos DB Query
    
    Bing
    Bing
    
    Bing Ads Query
    
    Bing Ads Query authentication guide
    
    Bing Search Query
    
    Bing Search Query authentication guide
    
    Box
    Box
    
    Box Extract
    
    Box Extract authentication guide
    
    Cassandra
    Cassandra
    
    Cassandra Query
    
    CloudWatch Publish
    CloudWatch Publish
    
    CloudWatch Publish
    
    Couchbase
    Couchbase
    
    Couchbase Query
    
    Data Transfer
    Data Transfer
    
    Data Transfer Object
    
    Dropbox
    Dropbox
    
    Dropbox Extract
    
    Dropbox Extract authentication guide
    
    Dynamics 365
    Dynamics 365
    
    Dynamics 365 Query
    
    Dynamics 365 Query Authentication Guide
    
    Dynamics 365 Business Central Query
    
    Dynamics 365 Business Central Query authentication guide
    
    Dynamics 365 Sales Query
    
    Dynamics 365 Sales Query Authentication Guide
    
    Dynamics CRM
    Dynamics CRM
    
    Dynamics CRM Query
    
    Dynamics CRM Query authentication guide
    
    Dynamics 365 NAV
    Dynamics 365 NAV
    
    Dynamics NAV Query
    
    DynamoDB
    DynamoDB
    
    DynamoDB Load
    
    DynamoDB Query
    
    EMR
    EMR
    
    EMR Load
    
    Elasticsearch
    Elasticsearch
    
    Elasticsearch Query
    
    Email
    Email
    
    Email Query
    
    Excel
    Excel
    
    Excel Query
    
    Facebook
    Facebook
    
    Facebook Query
    
    Facebook Ads Query
    
    Facebook Authentication Guide
    
    Facebook Connector FAQ
    
    Google
    Google
    
    Gmail Query
    
    Gmail Query authentication guide
    
    Google Ad Manager Query
    
    Google Ads Query
    
    Google Ad Manager Query authentication guide
    
    Google Analytics Query
    
    Google Analytics Query - add new data sources
    
    Google BigQuery
    
    Google Custom Search Query
    
    Google Custom Search Query authentication guide
    
    Google Drive Table
    
    Google Sheets Query
    
    Google Query authentication guide
    
    Google third-party OAuth using Service Accounts
    
    How to set your own Google Ads developer token
    
    HubSpot
    HubSpot
    
    HubSpot Query
    
    HubSpot Query authentication guide
    
    Instagram
    Instagram
    
    Instagram Extract
    
    Instagram Extract authentication guide
    
    Instagram Business Account ID Guide
    
    Intercom
    Intercom
    
    Intercom Extract
    
    Intercom Extract authentication guide
    
    Jira
    Jira
    
    Jira Query
    
    Jira Query authentication guide
    
    LDAP
    LDAP
    
    LDAP Query
    
    LinkedIn
    LinkedIn
    
    LinkedIn Query
    
    LinkedIn Query authentication guide
    
    LinkedIn Ads Query
    
    LinkedIn Ads Query authentication guide
    
    Magento
    Magento
    
    Magento Query
    
    Mailchimp
    Mailchimp
    
    Mailchimp Query
    
    Mailchimp Query authentication guide
    
    Mandrill
    Mandrill
    
    Mandrill Extract
    
    Mandrill Extract authentication guide
    
    Marketo
    Marketo
    
    Marketo Query
    
    Marketo Query Authentication Guide
    
    Microsoft Exchange
    Microsoft Exchange
    
    Microsoft Exchange Query
    
    Microsoft Exchange Query Authentication Guide
    
    MindSphere
    MindSphere
    
    MindSphere Extract
    
    MindSphere Extract authentication guide
    
    Mixpanel
    Mixpanel
    
    Mixpanel Query
    
    Mixpanel Query authentication guide
    
    MongoDB
    MongoDB
    
    MongoDB Query
    
    NetSuite
    NetSuite
    
    NetSuite Query
    
    NetSuite Query Authentication Guide
    
    NetSuite SuiteAnalytics Connect
    
    NetSuite SuiteAnalytics Connect Authentication Guide
    
    OData
    OData
    
    OData Query
    
    Open Exchange Rates
    Open Exchange Rates
    
    Open Exchange Rates Query
    
    Open Exchange Rates Query authentication guide
    
    Oracle Eloqua
    Oracle Eloqua
    
    Oracle Eloqua Query
    
    Oracle Eloqua Query authentication guide
    
    Pardot
    Pardot
    
    Pardot Extract
    
    Pardot Extract authentication guide
    
    PayPal
    PayPal
    
    PayPal Query
    
    PayPal Query Authentication Guide
    
    QuickBooks
    QuickBooks
    
    QuickBooks Online Query
    
    QuickBooks Online Query Authentication Guide
    
    RDS
    RDS
    
    RDS Query
    
    Redis
    Redis
    
    Redis Query
    
    Recurly
    Recurly
    
    Recurly Extract
    
    Recurly Extract authentication guide
    
    Sage Intacct
    Sage Intacct
    
    Sage Intacct Query
    
    Salesforce
    Salesforce
    
    Salesforce Query
    
    Salesforce Incremental Load
    
    Salesforce Query authentication guide
    
    Salesforce Bulk Query
    
    Salesforce Bulk Query authentication guide
    
    Salesforce Marketing Cloud Query
    
    Salesforce Marketing Cloud Query authentication guide
    
    Salesforce Output authentication guide
    
    SAP NetWeaver
    SAP NetWeaver
    
    SAP NetWeaver Query
    
    SAP NetWeaver Query - add new data sources
    
    SAP ODP
    SAP ODP
    
    SAP ODP Extract
    
    SAP ODP connection options
    
    SAP ODP troubleshooting
    
    SAP ODP FAQ
    
    SendGrid
    SendGrid
    
    SendGrid Query
    
    SendGrid Query authentication guide
    
    ServiceNow
    ServiceNow
    
    ServiceNow Query
    
    ServiceNow Query authentication guide
    
    SharePoint
    SharePoint
    
    SharePoint Query
    
    SharePoint Query authentication guide
    
    Shopify
    Shopify
    
    Shopify Query
    
    Shopify Query Authentication Guide
    
    Snapchat
    Snapchat
    
    Snapchat Extract
    
    Snapchat Extract authentication guide
    
    Splunk
    Splunk
    
    Splunk Query
    
    SQL databases
    SQL databases
    
    Database Query
    
    JDBC Incremental Load
    
    Square
    Square
    
    Square Query
    
    Square Query authentication guide
    
    Stripe
    Stripe
    
    Stripe Query
    
    Stripe Query authentication guide
    
    Sugar CRM
    Sugar CRM
    
    Sugar CRM Query
    
    SurveyMonkey
    SurveyMonkey
    
    SurveyMonkey Query
    
    SurveyMonkey Query authentication guide
    
    Twilio
    Twilio
    
    Twilio Query
    
    Twilio Query authentication guide
    
    Twitter
    Twitter
    
    Twitter Query
    
    Twitter Query authentication guide
    
    Twitter Ads Query
    
    Workday
    Workday
    
    Workday Extract
    
    Workday Custom Reports
    
    Workday Extract authentication guide
    
    Workday Integration System User setup
    
    Finding a Workday object's integration ID
    
    Managing access to data sources
    
    Xero
    Xero
    
    Xero Query
    
    Xero Query authentication guide
    
    YouTube
    YouTube
    
    YouTube Query
    
    YouTube Analytics Query
    
    Zendesk
    Zendesk
    
    Zendesk Support Query
    
    Zendesk Talk Query
    
    Zendesk Support/Talk Query authentication guide
    
    Zendesk Chat Extract
    
    Zendesk Chat Extract Authentication Guide
    
    Zoho CRM
    Zoho CRM
    
    Zoho CRM Query
    
    Zoho CRM Query authentication guide
    
    Zuora
    Zuora
    
    Zuora Query
    
    Zuora Bulk Query
    
    Zulk Query and Bulk Query authentication guide
  - Output components
    Output components
    
    Output components overview
    
    Oracle Output
    
    Microsoft SQL Server Output
    
    Salesforce Output
    
    Pardot Output
    
    Intercom Output
    
    Azure SQL Bulk Output
    
    RDS Bulk Output
  - Cloud Storage
    Cloud Storage
    
    Amazon S3
    Amazon S3
    
    S3 Load
    
    S3 Manifest Builder
    
    S3 Unload
    
    Azure Blob Storage
    Azure Blob Storage
    
    Azure Load Snowflake
    
    Azure Load Delta Lake
    
    Azure Load Synapse
    
    Azure Unload
    
    Google Cloud Storage
    Google Cloud Storage
    
    Cloud Storage Load
    
    Cloud Storage Unload
  - DDL
    DDL
    
    External Schema and Tables
    
    Snowflake
    Snowflake
    
    Alter External Table
    
    Create External Table
    
    Refresh External Table
    
    Alter Table
    
    Alter Warehouse
    
    Assign Tag
    
    Alter Masking Policy
    
    Create File Format
    
    Create Stream
    
    Amazon Redshift
    Amazon Redshift
    
    Add Partition
    
    Alter Session WLM Slots
    
    Analyze Tables
    
    Create External Table
    
    Delete Partition
    
    Nested Data Load
    
    Refresh Materialized View
    
    Schema Copy
    
    Text Output
    
    Delta Lake on Databricks
    Delta Lake on Databricks
    
    Delta Live Streaming
    
    Optimize
    
    Refresh Table
    
    Run Delta Live Table
    
    Start Cluster
    
    Google BigQuery
    Google BigQuery
    
    Create External Table
    
    Refresh Materialized View
    
    Azure Synapse Analytics
    Azure Synapse Analytics
    
    Alter Database
    
    Create External Table
    
    Create Table
    
    Delete Table
    
    SQL Script
    
    Truncate Table
    
    Vacuum Table
  - Flow components
    Flow components
    
    Flow components overview
    
    Iterators
    Iterators
    
    Iterator components
    
    File Iterator
    
    Fixed Iterator
    
    Grid Iterator
    
    Loop Iterator
    
    Table Iterator
    
    Transactions
    Transactions
    
    Begin
    
    Commit
    
    Rollback
    
    And
    
    End Failure
    
    End Success
    
    If
    
    Or
    
    Retry
    
    Run Orchestration
    
    Run Transformation
    
    Start
  - Load generators
    Load generators
    
    Load generators overview
    
    Azure Blob Load Generator
    
    Cloud Storage Load Generator (Snowflake)
    
    Cloud Storage Load Generator (BigQuery)
    
    S3 Load Generator (Snowflake)
    
    S3 Load Generator (Redshift)
    
    S3 Load Generator (Delta Lake)
  - Messaging
    Messaging
    
    SQS Message
    
    SNS Message
    
    PubSub
    
    Azure Queue Storage Message
    
    Webhook Post
  - Scripting
    Scripting
    
    Bash Script
    
    Python Script
    
    Sync File Source
    
    Commands for dbt Core
    
    Python Script additional settings
    
    Run Notebook
  - Shared jobs
    Shared jobs
    
    CDC shared jobs overview
    
    Sync All Tables shared job
    
    Sync Single Table shared job
    
    Create or Refresh External Table shared job
    
    Drop CDC Tables shared job
    
    Data typing with CDC shared jobs
    
    Tables created
    
    Append metadata
  - Variables
    Variables
    
    Append To Grid
    
    Describe To Grid
    
    Show to Grid
    
    Query Result To Grid
    
    JDBC Table Metadata To Grid
    
    Query Result To Scalar
    
    Remove From Grid
    
    Table Metadata To Grid
  - Load generators overview
  - Send Email
- Transformation
  Transformation
  - Join
    Join
    
    Except
    
    Intersect
    
    Join
    
    Unite
  - Read
    Read
    
    Fixed Flow
    
    Generate Sequence
    
    Multi Table Input
    
    Stream Input
    
    Table Input
    
    Wildcard Table Input
  - Transform
    Transform
    
    Aggregate
    
    Calculator
    
    Construct Struct
    
    Construct Variant
    
    Convert Type
    
    Table Delete Rows
    
    Detect Changes
    
    Distinct
    
    Extract Nested Data
    
    Filter
    
    First-Last
    
    Flatten Variant
    
    Lead-Lag
    
    Map Values
    
    Pivot
    
    Rank
    
    Rename
    
    Replicate
    
    Split Field
    
    SQL
    
    Transpose Columns
    
    Transpose Rows
    
    List Aggregate
    
    Unpivot
    
    Window Calculation
  - Write
    Write
    
    Copy Table To External Schema
    
    Create View
    
    External Table Output
    
    Rewrite External Table
    
    Rewrite Table
    
    Table Output
    
    Table Update
- Matillion ETL API
  Matillion ETL API
- Cloud configuration
  Cloud configuration
  - Authorization and authentication
  - Setting up Let's Encrypt for SSL on a Matillion ETL Instance
  - Snowflake
    Snowflake
    
    Snowflake role privileges with Matillion ETL
  - Amazon Web Services
    Amazon Web Services
    
    Attaching AWS IAM roles To EC2 instances
    
    IAM roles & permissions (AWS)
    
    Changing EC2 instance size
  - Google Cloud Platform
    Google Cloud Platform
    
    IAM roles & permissions (GCP)
    
    GCP service accounts
    
    BigQuery data set setup
    
    GCP enabling APIs
  - Microsoft Azure
    Microsoft Azure
    
    Changing Azure instance size
    
    Roles & permissions (Azure)
    
    Configuring a high availability cluster (Azure)
- Guides
  Guides
  - AWS services
    AWS services
    
    Spectrum
    Spectrum
    
    Getting Started with Amazon Redshift Spectrum
    
    Using Spectrum in Matillion ETL
    
    Accessing files in S3 using Pre-signed URLs
    
    Automating Redshift maintenance
    
    AWS S3 lifecycle rule
    
    Cross-account S3 access
    
    Parallelism with Matillion ETL for Redshift
    
    Snowflake AWS Storage Integration setup guide
  - Azure services
    Azure services
    
    Snowflake Azure Storage Integration setup guide
    
    Troubleshooting connection to Azure Blob Storage
    
    Creating secrets in Azure Key Vault
  - GCP services
    GCP services
    
    Snowflake GCP Storage Integration setup guide
  - Common operations
    Common operations
    
    Populating tables
    Populating tables
    
    Adding filename as a column to tables (BigQuery)
    
    Adding Filename as a column to new table
    
    Populating an audit table
    
    Building a data vault
    
    Building a data vault (Snowflake)
    
    Data transfer between databases
    
    Ingesting AWS ElasticSearch data via the API Query component
    
    Year-on-year analysis
    
    Designing a job for a high availability cluster
    
    Incremental or high water mark data Loading
    
    Microbatch replication
    
    How to receive emails by subscribing to a cloud Pub/Sub topic
    
    Flattening nested arrays
  - Installation configuration
    Installation configuration
    
    Changing the host file
    
    How to configure SSL protocols
    
    Outbound IP requirements
  - Triggering Matillion ETL
    Triggering Matillion ETL
    
    Triggering ETL from an S3 event via AWS Lambda
    
    Triggering ETL from an email via SES and Lambda
    
    Triggering ETL from Amazon Alexa via AWS Lambda & Amazon SQS
    
    Triggering a Matillion ETL job from your Google Home device
    
    Triggering Matillion ETL from a storage queue via an Azure function
  - Collibra integration
  - Connecting to an RDS in a private VPC
  - Diagnostic data policy
  - Executing Python scripts outside of Matillion
  - Helping with the GDPR
  - Integrating Slack with Matillion ETL
  - Using grid variables to apply business rules in a transformation job
  - Making multiple API queries
  - Matillion data quality framework
  - NRT replication In Redshift
  - Pivoting and unpivoting tables
  - SCM integration
  - Tracking loaded files
  - Using incron to automatically copy data to S3
  - Using KMS encrypted passwords in Python
  - Using R with Matillion ETL for Redshift
  - Using Table Metadata to Grid
  - Managing Python on a Matillion ETL virtual machine (VM)
  - How to retrieve missing Task History entries after 1.47 upgrade
  - Matillion Exchange
- Migrate to the Data Productivity Cloud
  Migrate to the Data Productivity Cloud
- Release notes
  Release notes
- Tech notes
  Tech notes
Data Productivity Cloud
Data Productivity Cloud
- Get started
  Get started
  - Free trial
  - Data Productivity Cloud new user tutorial
  - Prerequisites
    Prerequisites
    
    Full SaaS prerequisites
    
    Hybrid SaaS prerequisites (Snowflake AWS)
  - Quickstart guides
    Quickstart guides
    
    Set up a Matillion managed project
- Core concepts
  Core concepts
  - Data Productivity Cloud overview
  - Architecture overview
  - Connectors overview
  - Getting support
  - Supported browsers
  - Data Productivity Cloud release FAQ
  - Cloud data platforms
    Cloud data platforms
    
    Snowflake
    Snowflake
    
    Snowflake role privileges
    
    Using Snowflake key-pair authentication
    
    Databricks
    Databricks
    
    Databricks
    
    Troubleshooting data warehouse connectivity
  - Managing secrets
    Managing secrets
    
    Secrets overview
    
    AWS Secrets Manager
    
    Azure Key Vault
    
    GCP Secret Manager
- Security
  Security
- Matillion Hub
  Matillion Hub
  - Basics
    Basics
    
    Getting started
    
    Registration
  - Administration
    Administration
    
    Change email address
    
    Edit user permissions
    
    Create an account
    
    Manage account details
    
    Manage account users
    
    Reset your password
  - Authentication
    Authentication
    
    Multi-factor authentication
    
    SSO overview
  - Services
    Services
    
    Select a service
    
    Pipeline Observability
    
    Matillion ETL instance overview
    
    Data lineage
    
    Matillion Data Loader
  - Subscriptions
    Subscriptions
    
    AWS Marketplace subscription
    
    Azure Marketplace subscription
    
    Subscription Management
    Subscription Management
    
    Credit consumption dashboard
    
    Editions
    
    Manage payment details and invoices
    
    Manage subscription
- Agents
  Agents
  - Data Productivity Cloud agent
    Data Productivity Cloud agent
    
    Agent overview
    
    Create an agent in your infrastructure
    
    Restart an agent
    
    Refresh agent credentials
    
    AWS agent setup
    AWS agent setup
    
    Manual agent setup
    
    Agent installation using a CloudFormation quick-create link
    
    AWS IAM roles
    
    Add agent credentials to AWS Secrets Manager
    
    Azure agent setup
    Azure agent setup
    
    Agent installation using an ARM template
    
    Streaming agent installation on Azure
    
    ARM template permissions
    
    Configuring a key vault for Azure agent
    
    Download and amend the Azure ARM template
    
    Authenticate with Azure services
    
    Azure agent troubleshooting
    
    Additional configuration
    Additional configuration
    
    Scaling best practices
    
    How to modify access for each source system
    
    Customizing agent networking and connectivity
    
    Uploading external drivers to the agent
    
    Optional agent parameters
  - Streaming Agent
    Streaming Agent
    
    Streaming agent installation
    
    Streaming agent setup UI
    
    Sizing streaming agents
    
    Agent versions
    
    Agent update
    
    Platform keys
    
    Environment variables
    
    Lock down outbound traffic
    
    AWS installation
    AWS installation
    
    AWS installation overview
    
    Deploying a CDC agent in AWS quick guide
    
    Cloud resources
    Cloud resources
    
    Subnets and security groups
    
    CloudWatch Logs
    
    IAM Roles
    
    S3 Buckets
    
    Fargate and ECS
    
    AWS templates
    AWS templates
    
    CloudFormation
    CloudFormation
    
    CloudFormation ECS Fargate basic template
    
    CloudFormation ECS Fargate advanced template
    
    CloudFormation EC2 ECS proxy templates
    
    Kubernetes templates
    Kubernetes templates
    
    Agent (AWS) template
    
    Terraform
    Terraform
    
    Terraform ECS Fargate basic template
    
    Terraform ECS Fargate advanced template
    
    Azure installation
    Azure installation
    
    Azure installation overview
    
    Deploying a CDC agent in Azure quick guide
    
    Deploying a CDC agent in Azure using the CLI
    
    Cloud resources
    Cloud resources
    
    Managed identity
    
    Blob Storage
    
    Log analytics
    
    Container instances
    
    Azure templates
    Azure templates
    
    Azure - ARM template
    
    Azure - Terraform template
    
    Kubernetes templates
    Kubernetes templates
    
    Agent (Azure) template
  - Logs
    Logs
    
    AWS CloudWatch Logs
    
    Azure Log Analytics
    
    Google Cloud logging
- Designer
  Designer
  - Designer overview
  - Designer UI basics
  - Your Designer projects
    Your Designer projects
    
    Projects
    
    Environments
    
    Environment access
    
    Cloud provider credentials
    
    Branches
    
    Pipelines
    
    Shared pipelines
    
    Connectors overview
    
    Secret definitions
    
    OAuth
    
    Schedules
    
    Access
  - Git version control
    Git version control
    
    Git in Designer
    
    Commit changes
    
    Push local changes
    
    Pull remote changes
    
    Merge from branch
    
    Compare changes
    
    Reset branch
    
    Connect your own Git repository
    Connect your own Git repository
    
    Overview
    
    GitHub
    GitHub
    
    Overview of the Matillion GitHub app
    
    Prerequisites
    
    Installing the Matillion GitHub app
    
    Troubleshooting GitHub repository connection
    
    Azure DevOps
    Azure DevOps
    
    Overview of the Matillion Azure app
    
    Prerequisites
    
    Installing the Matillion Azure app
    
    Troubleshooting an Azure DevOps repository connection
    
    GitLab
    GitLab
    
    Overview of the Matillion GitLab app
    
    Prerequisites
    
    Troubleshooting GitLab repository connections
    
    Bitbucket
    Bitbucket
    
    Overview of the Matillion Bitbucket app
    
    Prerequisites
    
    Troubleshooting Bitbucket repository connections
  - AI
    AI
    
    Overview of AI features
    
    Data Productivity Cloud AI FAQ
    
    Copilot
    Copilot
    
    Copilot overview
    
    Copilot prerequisites
    
    Start using Copilot
    
    Example uses of Copilot
    
    AI Note
  - Pipeline components
    Pipeline components
    
    Components overview
    
    Data lineage
    
    Sampling output
    
    Variables
    
    Grid variables
    
    Organizing file storage
    
    Orchestration
    Orchestration
    
    AI
    AI
    
    Snowflake
    Snowflake
    
    Chunk Text
    
    Snowpark Container Prompt
    
    Create an app in Snowpark Container Services
    
    OpenAI
    OpenAI
    
    OpenAI Prompt
    
    AWS
    AWS
    
    Amazon Bedrock Prompt
    
    Amazon Textract Input
    
    Amazon Transcribe
    
    Azure
    Azure
    
    Azure OpenAI Prompt
    
    Azure Document Intelligence
    
    Azure Speech Transcribe
    
    Vector database operations
    Vector database operations
    
    Snowflake
    Snowflake
    
    Snowflake Vector Upsert (preview)
    
    Pinecone
    Pinecone
    
    Pinecone Vector Upsert
    
    Pinecone Vector Query
    
    Postgres
    Postgres
    
    Postgres Vector Upsert
    
    Connectors
    Connectors
    
    Load
    Load
    
    ActiveCampaign
    
    Amplitude
    
    Anaplan
    
    Asana
    
    Attio
    
    Azure Blob Storage Load
    
    Azure Cosmos DB for NoSQL
    
    Azure SQL
    
    Bing Ads Query
    
    Box
    
    Braze
    
    Brevo
    
    Bugcrowd
    
    Buildkite
    
    Chargebee
    
    CircleCI
    
    ClickUp
    
    Concord
    
    Confluence
    
    Data Productivity Cloud (Matillion)
    
    Data Transfer
    
    Database Query
    
    Datadog
    
    Databricks
    
    dbt Cloud
    
    Delighted
    
    Dixa
    
    Dynamics 365 Query
    
    DynamoDB Query
    
    Email Query
    
    Eventbrite
    
    Excel Query
    
    Facebook Ads Query
    
    Facebook Query
    
    Fanatics
    
    FireHydrant
    
    Float
    
    Freshdesk
    
    GitHub
    
    Gmail
    
    Gong
    
    Google Ads Query
    
    Google Analytics Query
    
    Google BigQuery Query
    
    Google Cloud Storage Load
    
    Google Sheets Query
    
    HubSpot Query
    
    Infobip
    
    Intercom
    
    JDBC
    
    Jira Query
    
    Jotform
    
    Kafka
    
    Klaviyo
    
    LaunchDarkly
    
    LDAP Query
    
    Lob
    
    Mailchimp
    
    Mailgun
    
    Mandrill
    
    Marketo Query
    
    Microsoft Exchange
    
    Mixpanel
    
    MongoDB Query
    
    NetSuite Query
    
    NetSuite SuiteAnalytics
    
    New Relic
    
    Notion
    
    OData Query
    
    Ongoing WMS
    
    Orbit
    
    Ortto
    
    PagerDuty
    
    PayPal
    
    Pendo
    
    Persona
    
    Pingdom
    
    Pipedrive
    
    PivotalTracker
    
    Productboard
    
    RDS Query
    
    Recurly
    
    Rootly
    
    S3 Load
    
    Salesforce Query
    
    Salesforce Marketing Cloud Query
    
    Salesloft
    
    SAP ODP
    
    SAP SuccessFactors
    
    SeatGeek
    
    SendGrid
    
    ServiceNow Query
    
    SharePoint Query
    
    Shopify
    
    Slack
    
    Smartsheet
    
    Snapchat
    
    Snyk
    
    Square
    
    SurveyMonkey
    
    Stripe Query
    
    Ticketmaster
    
    TikTok
    
    Toggl
    
    Twilio
    
    Volume to Delta Table
    
    Workday
    
    Workday Custom Reports
    
    Yelp
    
    Zendesk Talk
    
    Zendesk Ticketing
    
    Unload
    Unload
    
    Azure Blob Storage Unload
    
    Google Cloud Storage Unload
    
    Microsoft SQL Server Output
    
    RDS Bulk Output
    
    S3 Unload
    
    Salesforce Output
    
    Authentication Guides
    Authentication Guides
    
    Create an OAuth connection for a custom connector
    
    Bing Ads Query authentication guide
    
    Dynamics 365 Query authentication guide
    
    Google Query authentication guide
    
    HubSpot Query authentication guide
    
    Jira Query authentication guide
    
    Kafka authentication guide
    
    Mailchimp authentication guide
    
    Marketo Query authentication guide
    
    NetSuite Query authentication guide
    
    NetSuite SuiteAnalytics authentication guide
    
    Salesforce Query authentication guide
    
    ServiceNow Query Authentication Guide
    
    Stripe Query authentication guide
    
    Workday authentication guide
    
    DDL
    DDL
    
    Add Partition
    
    Alter Table
    
    Analyze Tables
    
    Append to Grid
    
    Create External Table
    
    Create Stream
    
    Create Table
    
    Delete Tables
    
    JDBC Table Metadata-To-Grid
    
    Optimize
    
    Refresh External Table
    
    Refresh Materialized View
    
    Refresh Table
    
    Remove from Grid
    
    Schema Copy
    
    SQL Script
    
    Table Metadata To Grid
    
    Truncate Table
    
    Action
    Action
    
    Snowflake
    Snowflake
    
    Alter Warehouse
    
    Create File Format
    
    AWS
    AWS
    
    CloudWatch Publish
    
    SNS Message
    
    SQS Message
    
    Messaging
    Messaging
    
    Azure Queue Storage Message
    
    Cloud Pub/Sub
    
    Send Email
    
    Webhook Post
    
    Testing
    Testing
    
    Assert External Table
    
    Assert Scalar Variables
    
    Assert Table
    
    Print Variables
    
    Scripting
    Scripting
    
    dbt Core
    
    Bash Pushdown
    
    Bash Script
    
    Python Pushdown
    
    Python Script
    
    Query Result to Grid
    
    Query Result to Scalar
    
    Run Notebook
    
    Update Scalar
    
    Utility
    Utility
    
    Start Compute
    
    Vacuum
    
    Run pipeline
    Run pipeline
    
    Run Orchestration
    
    Run Shared Pipeline
    
    Run Transformation
    
    Logic
    Logic
    
    Flow
    Flow
    
    And
    
    End Failure
    
    End Success
    
    If
    
    Or
    
    Retry
    
    Start
    
    Iterate
    Iterate
    
    File Iterator
    
    Fixed Iterator
    
    Grid Iterator
    
    Loop Iterator
    
    Table Iterator
    
    Stacking and detaching iterators
    
    Transformation
    Transformation
    
    AI
    AI
    
    Snowflake
    Snowflake
    
    Cortex
    Cortex
    
    Completions
    
    Embed
    
    Extract Answer
    
    Sentiment
    
    Summarize
    
    Translate
    
    Vector Search (preview)
    
    Document AI Predict
    
    Databricks AI (preview)
    Databricks AI (preview)
    
    AI Analyze Sentiment
    
    AI Classify
    
    AI Extract
    
    AI Fix Grammar
    
    AI Mask
    
    AI Query
    
    AI Similarity
    
    AI Summarize
    
    AI Translate
    
    Vector Search
    
    Join
    Join
    
    Except
    
    Intersect
    
    Join
    
    Unite
    
    Read
    Read
    
    Fixed Flow
    
    Generate Sequence
    
    Multi Table Input
    
    Stream Input
    
    Table Input
    
    Testing
    Testing
    
    Assert View
    
    Transform
    Transform
    
    Aggregate
    
    Calculator
    
    Construct Variant
    
    Convert String to Struct
    
    Convert Type
    
    Detect Changes
    
    Distinct
    
    Extract Nested Data
    
    Extract Structured Data
    
    Filter
    
    First/Last
    
    Flatten Variant
    
    Lead/Lag
    
    Map Values
    
    Pivot
    
    Rank
    
    Rename
    
    Split field
    
    SQL
    
    Table Delete Rows
    
    Transpose Columns
    
    Transpose Rows
    
    Unpivot
    
    Window Calculation
    
    Write
    Write
    
    Create View
    
    Rewrite Table
    
    Table Output
    
    Table Update
  - Examples
    Examples
    
    Incremental loading
    
    Jira Query
    
    S3 Load and transformation
    
    Data quality framework
- Streaming
  Streaming
  - Streaming pipeline overview
  - Create a streaming pipeline
  - Manage streaming pipelines
  - Signal tables
  - Schema drift
  - Snapshots
  - Source database maintenance
  - Sources
    Sources
    
    Db2 for IBM i
    Db2 for IBM i
    
    Db2 for IBM i streaming connector
    
    Db2 for IBM i database configuration
    
    Microsoft SQL Server
    Microsoft SQL Server
    
    Microsoft SQL Server streaming connector
    
    Microsoft SQL Server database configuration
    
    MySQL
    MySQL
    
    MySQL streaming connector
    
    MySQL database configuration
    
    MySQL streaming agent configuration
    
    Oracle
    Oracle
    
    Oracle streaming connector
    
    Oracle database configuration
    
    PostgreSQL
    PostgreSQL
    
    PostgreSQL streaming connector
    
    PostgreSQL database configuration
  - Destinations
    Destinations
    
    Amazon S3
    
    Azure Blob Storage
    
    Snowflake
  - Pre-built pipelines
    Pre-built pipelines
    
    Sync All Tables
- Data Loader Batch
  Data Loader Batch
  - Batch Loads Introduction
  - Getting started
  - Batch Pipeline UI
  - Batch loading overview
    Batch loading overview
    
    Schema drift
    
    Batch load replication overview
    
    Batch pipeline detailed view
  - Administration
    Administration
    
    Setup requirements
    Setup requirements
    
    Technical requirements
    
    Cloud data warehouse permissions
    
    Managing credentials
    Managing credentials
    
    Destinations
    
    Passwords
    
    Cloud credentials
    
    OAuth connections
    
    NetSuite batch pipeline authentication guide
    
    Scheduling with Quartz cron expressions
  - Sources
    Sources
    
    Custom Connector Data Loader pipeline
    
    Allowed connection properties
    
    ActiveCampaign
    
    Amazon Redshift
    
    Amplitude
    
    Brevo
    
    Chargebee
    
    Circle CI
    
    Concord
    
    Confluence
    
    Datadog
    
    DB2 for IBM i
    
    Delighted
    
    Dropbox
    
    Dynamics 365 Sales
    
    Elasticsearch
    
    Eventbrite
    
    Excel
    
    Facebook AdAccounts
    
    Facebook AdInsights
    
    Facebook Content Insights
    
    Freshdesk
    
    Google Analytics
    
    Google BigQuery
    
    Google Custom Search
    
    Google Sheets
    
    HubSpot
    
    IBM Db2
    
    Jira
    
    Klaviyo
    
    LaunchDarkly
    
    Mailchimp
    
    MariaDB
    
    Marketo
    
    Microsoft SQL Server
    
    MongoDB
    
    MySQL
    
    NetSuite
    
    OData
    
    Open Exchange Rates
    
    Oracle
    
    Oracle Eloqua
    
    PagerDuty
    
    Pendo
    
    PostgreSQL
    
    Productboard
    
    QuickBooks
    
    Recurly
    
    Sage Intacct
    
    Salesforce
    
    Salesforce Pardot
    
    Shopify
    
    Slack
    
    Smartsheet
    
    Snapchat
    
    Snowflake
    
    Snyk
    
    Stripe
    
    SugarCRM
    
    TikTok
    
    Toggl
    
    Twilio
    
    Xero
    
    Zendesk
    
    Zendesk Talk
  - Destinations
    Destinations
    
    Snowflake
    Snowflake
    
    Set up Snowflake
    
    Connect to Snowflake
    
    Delta Lake on Databricks
    Delta Lake on Databricks
    
    Set up Delta Lake on Databricks
    
    Connect to Delta Lake on Databricks
    
    Amazon Redshift
    Amazon Redshift
    
    Set up Amazon Redshift
    
    Connect to Amazon Redshift
    
    Google BigQuery
    Google BigQuery
    
    Set up Google BigQuery
    
    Connect to Google BigQuery
  - Tech notes
    Tech notes
    
    Tech note - failing pipelines
  - Troubleshooting
    Troubleshooting
    
    Destinations
- Custom Connector
  Custom Connector
  - Matillion Custom Connector Overview
  - Export your own connector to Matillion ETL
  - Create a pipeline with custom connectors
  - Creating Custom Connectors
    Creating Custom Connectors
    
    Custom connector setup
    
    Endpoints
    
    Authentication
    
    Add OAuth
    
    Pagination
    
    Script paging
    
    API parameters
    
    Custom connector example
  - Flex connectors
    Flex connectors
    
    Flex connector setup
    
    Flex connector tutorial
- Data Productivity Cloud API
  Data Productivity Cloud API
  - API overview
  - Authentication
  - API reference
    API reference
    
    API reference user guide
    
    Data Productivity Cloud public API reference
  - API Guides
    API Guides
    
    Executing and managing a pipeline
  - API changelog
- Changelogs
  Changelogs
  - 2024
    2024
    
    November
    
    October
    
    September
    
    August
    
    July
    
    June
    
    May
    
    April
    
    March
    
    February
    
    January
  - 2023
    2023
    
    December
    
    November
    
    October
    
    September
    
    August
    
    July
    
    June
    
    May
    
    April
    
    March
    
    February
    
    January
  - 2022
    2022
    
    December
    
    November
    
    October
    
    September
    
    August
    
    July
    
    June
    
    May
    
    April
Style guide
Style guide
- Style guide overview
- Markdown and formatting
  Markdown and formatting
  - Formatting text
  - Capitalization
  - Headings
  - Horizontal rules
  - Images
  - Links
  - Lists
  - Tables
  - Words
- MarkUp and reviewing
  MarkUp and reviewing
  - MarkUp.io
- Release status
  Release status
  - Release status
Data Loader CDC
Data Loader CDC
- Matillion Change Data Capture
- Getting started
- CDC shared jobs
- CDC Pipeline UI
- CDC Overview
  CDC Overview
- CDC Agent
  CDC Agent
  - CDC agent installation
  - CDC agent setup UI
  - Sizing CDC agents
  - Agent versions
  - Agent update
  - Platform keys
  - Environment variables
  - Lock down outbound traffic
  - AWS installation
    AWS installation
    
    AWS installation overview
    
    Deploying a CDC agent in AWS quick guide
    
    Cloud resources
    Cloud resources
    
    Subnets and security groups
    
    CloudWatch Logs
    
    IAM Roles
    
    S3 Buckets
    
    Fargate and ECS
    
    AWS templates
    AWS templates
    
    CloudFormation
    CloudFormation
    
    CloudFormation ECS Fargate basic template
    
    CloudFormation ECS Fargate advanced template
    
    CloudFormation EC2 ECS proxy templates
    
    Kubernetes templates
    Kubernetes templates
    
    Agent (AWS) template
    
    Terraform
    Terraform
    
    Terraform ECS Fargate basic template
    
    Terraform ECS Fargate advanced template
  - Azure installation
    Azure installation
    
    Azure installation overview
    
    Deploying a CDC agent in Azure quick guide
    
    Deploying a CDC agent in Azure using the CLI
    
    Cloud resources
    Cloud resources
    
    Managed identity
    
    Blob Storage
    
    Log analytics
    
    Container instances
    
    Azure templates
    Azure templates
    
    Azure - ARM template
    
    Azure - Terraform template
    
    Kubernetes templates
    Kubernetes templates
    
    Agent (Azure) template
  - GCP installation
    GCP installation
    
    GCP installation overview
    
    Deploying a CDC agent in GCP quick guide
    
    Cloud resources
    Cloud resources
    
    Project and Service Account
    
    GCP Cloud Storage bucket
    
    IAM permissions
    
    Workload Identity (GKE)
    
    GCP templates
    GCP templates
    
    Agent (GCE) template
    
    Agent (GKE) Kubernetes template
- Administration
  Administration
  - Technical Requirements (CDC)
  - Lifecycle management
    Lifecycle management
    
    Amazon S3 lifecycle management
    
    Azure Blob Storage lifecycle management
    
    Google Cloud Storage lifecycle management
- Sources
  Sources
  - CDC sources overview
  - Source database maintenance
  - Db2 for IBM i
    Db2 for IBM i
    
    Db2 for IBM i connector
    
    Configuring Db2 for IBM i
    
    DB2 for IBM i snapshot
  - PostgreSQL
    PostgreSQL
    
    PostgreSQL connector
    
    Configuring PostgreSQL database
    
    PostgreSQL advanced settings
    
    PostgreSQL snapshot
  - Oracle
    Oracle
    
    Oracle connector
    
    Configuring Oracle Database
    
    Oracle advanced settings
    
    Oracle snapshot
  - Microsoft SQL Server
    Microsoft SQL Server
    
    Microsoft SQL Server connector
    
    Configuring Microsoft SQL Server database
    
    Microsoft SQL advanced settings
    
    Microsoft SQL Server snapshot
  - MySQL
    MySQL
    
    MySQL connector
    
    Configuring MySQL database
    
    Configuring the CDC agent for MySQL
    
    MySQL snapshot
- Destinations
  Destinations
- Troubleshooting
  Troubleshooting

Data Productivity Cloud public API reference

Swagger UI

Note

If this page looks empty, refresh/reload the page.

Help us improve this page by giving feedback.

Products
DATA PRODUCTIVITY CLOUD
AI
MATILLION ETL
CONNECTORS
SECURITY
ALL FEATURES
PRICING
REQUEST A DEMO
FREE TRIAL

Solutions
OVERVIEW
BY CLOUD DATA PLATFORM
BY USE CASE
BY INDUSTRY
SUCCESS STORIES

Resources
ALL
BLOG
PRODUCT TOUR
HUB
ACADEMY
EXCHANGE
EVENTS
PARTNERS
GLOSSARY

About
OVERVIEW
WHY MATILLION
LEADERSHIP
CAREERS
INVESTORS
BRAND AND MEDIA KIT
CONTACT

Support & Services
SUPPORT
SERVICES & TRAINING
PARTNER PORTAL
SUPPORT PORTAL

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement