MindSphere Extract
Overview
The MindSphere Extract component calls the MindSphere API to retrieve and store data to be either referenced by an external table or loaded into a table, depending on your cloud data warehouse. You can then use transformation components to enrich and manage the data in permanent tables.
Using this component may return structured data that requires flattening. For help with flattening such data, we recommend using the following components:
- Extract Nested Data for Snowflake or Google BigQuery.
- Nested Data Load for Amazon Redshift.
Warning
The tokens loaded from the MindSphere API have a lifespan of 30 minutes. These tokens are stored within an in-memory cache that expires each entry after 25 minutes have elapsed. At this point, a new token is requested.
Properties
Name = string
A human-readable name for the component.
API = drop-down
Select the MindSphere API to make queries to.
Data Source = drop-down
Select the MindSphere data source from the available options.
API Username = string
Your API username for the chosen MindSphere app. For help locating your API
API Password = string
Your API password for the chosen MindSphere app. For help locating your API password, read MindSphere Extract authentication guide. Store the password in the component, or create a managed entry for the password using Manage Passwords (recommended).
App Name = string
The name of target MindSphere application. You can view an overview of your MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version = numeric
The version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant = string
The Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant = string
The User Tenant. For help creating a user account or assigning MindSphere permissions, read Providing Access for New Users.
Entity Id = string
Your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the data source is set to "IoT Time Series EU1".
Property Set Name = string
Specify the name of the Property Set. This property is only available when the data source is set to "IoT Time Series EU1".
Select = string
Specify the data to select from the property set. This property is only available when the data source is set to "IoT Time Series EU1".
Latest Value = boolean
When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the data source is set to "IoT Time Series EU1".
From = timestamp
Specify the start point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
To = timestamp
Specify the end point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
Limit = integer
Limit the number of results per page that will be staged. This property is only available when the data source is set to "IoT Time Series EU1".
Asset ID = string
Specify the Asset ID. This property is only available when the data source property is set to "Asset ID" or "Variables".
Filter = JSON
Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Page Limit = integer
Integer value for the limit of pages to stage.
Location = filepath
Provide an Amazon S3 bucket path, Google Cloud Storage (GCS) bucket path, or Azure Blob Storage path that will be used to store the data. The data can then be referenced by an external table. A folder will be created at this location with the same name as the target table.
Integration = drop-down
(GCP only) Choose your Google Cloud Storage Integration. Integrations are required to permit Snowflake to read data from and write to a Google Cloud Storage bucket. Integrations must be set up in advance of selecting them in Matillion ETL. To learn more about setting up a storage integration, read our Storage Integration setup guide.
Warehouse = drop-down
The Snowflake warehouse used to run the queries. The special value, [Environment Default], will use the warehouse defined in the environment. Read Overview of Warehouses to learn more.
Database = drop-down
The Snowflake database. The special value, [Environment Default], will use the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.
Schema = drop-down
The Snowflake schema. The special value, [Environment Default], will use the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.
Target Table = string
A name for the new table. Upon running the job, this table will be recreated and will drop any existing table of the same name.
Name = string
A human-readable name for the component.
Data Source = drop-down
Select the MindSphere data source from the available options.
API Username = string
Your API username for the chosen MindSphere app. For help locating your API
API Password = string
Your API password for the chosen MindSphere app. For help locating your API password, read MindSphere Extract authentication guide. Store the password in the component, or create a managed entry for the password using Manage Passwords (recommended).
App Name = string
The name of target MindSphere application. You can view an overview of your MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version = numeric
The version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant = string
The Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant = string
The User Tenant. For help creating a user account or assigning MindSphere permissions, read Providing Access for New Users.
Entity Id = string
Your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the data source is set to "IoT Time Series EU1".
Property Set Name = string
Specify the name of the Property Set. This property is only available when the data source is set to "IoT Time Series EU1".
Select = string
Specify the data to select from the property set. This property is only available when the data source is set to "IoT Time Series EU1".
Latest Value = boolean
When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the data source is set to "IoT Time Series EU1".
From = timestamp
Specify the start point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
To = timestamp
Specify the end point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
Limit = integer
Limit the number of results per page that will be staged. This property is only available when the data source is set to "IoT Time Series EU1".
Asset ID = string
Specify the Asset ID. This property is only available when the data source property is set to "Asset ID" or "Variables".
Filter = JSON
Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Page Limit = integer
Integer value for the limit of pages to stage.
Location = filepath
Provide an Amazon S3 bucket path that will be used to store the data. The data can then be referenced by an external table. A folder will be created at this location with the same name as the target table.
Type = drop-down
- External: The data will be put into your chosen S3 bucket and referenced by an external table.
- Standard: The data will be staged on your chosen S3 bucket before being loaded into a table. This is the default setting.
Standard Schema = drop-down
The Redshift schema. The special value, [Environment Default], will use the schema defined in the Matillion ETL environment.
External Schema = drop-down
The table's external schema. Read Getting Started with Amazon Redshift Spectrum to learn more.
Target Table = string
A name for the new table. Upon running the job, this table will be recreated and will drop any existing table of the same name.
Name = string
A human-readable name for the component.
Data Source = drop-down
Select the MindSphere data source from the available options.
API Username = string
Your API username for the chosen MindSphere app. For help locating your API
API Password = string
Your API password for the chosen MindSphere app. For help locating your API password, read MindSphere Extract authentication guide. Store the password in the component, or create a managed entry for the password using Manage Passwords (recommended).
App Name = string
The name of target MindSphere application. You can view an overview of your MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version = numeric
The version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant = string
The Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant = string
The User Tenant. For help creating a user account or assigning MindSphere permissions, read Providing Access for New Users.
Entity Id = string
Your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the data source is set to "IoT Time Series EU1".
Property Set Name = string
Specify the name of the Property Set. This property is only available when the data source is set to "IoT Time Series EU1".
Select = string
Specify the data to select from the property set. This property is only available when the data source is set to "IoT Time Series EU1".
Latest Value = boolean
When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the data source is set to "IoT Time Series EU1".
From = timestamp
Specify the start point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
To = timestamp
Specify the end point of the desired time range in the API call. The format is YYYY-MM-DDThh:mm:ss.mmmZ where T is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the data source is set to "IoT Time Series EU1".
Limit = integer
Limit the number of results per page that will be staged. This property is only available when the data source is set to "IoT Time Series EU1".
Asset ID = string
Specify the Asset ID. This property is only available when the data source property is set to "Asset ID" or "Variables".
Filter = JSON
Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Page Limit = integer
Integer value for the limit of pages to stage.
Table Type = drop-down
Select whether the table is Native (by default in BigQuery) or an external table.
Project = drop-down
Select the Google Cloud project. The special value, [Environment Default], will use the project defined in the environment. For more information, read Creating and managing projects.
Dataset = drop-down
Select the Google BigQuery dataset to load data into. The special value, [Environment Default], will use the dataset defined in the environment. For more information, read Introduction to datasets.
Target Table = string
A name for the new table. Upon running the job, this table will be recreated and will drop any existing table of the same name.
New Target Table = string
A name for the new external table. Only available when the table type is External.
Cloud Storage Staging Area = Google Cloud Storage bucket
The URL and path of the target Google Cloud Storage bucket to be used for staging the queried data. Only available when the table type is Native.
Location = Google Cloud Storage bucket
The URL and path of the target Google Cloud Storage bucket. Only available when the table type is External.
Load Options = multiple drop-downs
- Clean Cloud Storage Files: Destroy staged files on Google Cloud Storage after loading data. Default is On.
- Cloud Storage File Prefix: Give staged file names a prefix of your choice. The default setting is an empty field.
- Recreate Target Table: Choose whether the component recreates its target table before the data load. If Off, the component will use an existing table or create one if it does not exist. Default is On.
- Use Grid Variable: Check this checkbox to use a grid variable. This box is unchecked by default.
| Snowflake | Delta Lake on Databricks | Amazon Redshift | Google BigQuery | Azure Synapse Analytics |
|---|---|---|---|---|
| ✅ | ❌ | ✅ | ✅ | ❌ |