Schedules

Schedules allow users to run pipelines periodically at specified times.

Schedules can be viewed, created, edited, and deleted from the Schedules tab in the Project page. Not added a project yet? Read Add project. You can also add a schedule from the pipeline canvas.

Note

Each Data Productivity Cloud account can have a maximum of 500 schedules. Once you have 500 schedules, you will need to delete an existing schedule if you want to create a new one.

Understanding publishing and scheduling

To create a schedule, you must have published the pipeline that the schedule will run, using the Push local changes command from a branch in your project and ticking Publish.

Publishing creates an artifact that contains all the resources needed to run a specific version of the published pipeline. Artifacts are immutable and independent from the source code in your Git repository. The artifact created when you publish a pipeline is a copy of the pipeline, which is run by a schedule.

It's important to understand this distinction, as it has a number of implications for how schedules interact with pipelines.

When you delete a pipeline in Designer, the pipeline copy in the artifact that is run by the scheduler is not deleted. This means that scheduled pipelines will continue to run and consume credits after the pipeline is deleted. To prevent this from happening, you must also delete or disable the schedule.
When you create a schedule and select a previous artifact version, the Pipeline drop-down will show all pipelines in that version, including any pipelines you may have deleted in later versions. This is because artifacts are a snapshot of your project at the time of publishing.
When you edit a pipeline and republish it, all existing schedules running this pipeline where Always use latest is selected for the Artifact property will automatically use the new version of the pipeline.
If you edit a pipeline without republishing it, the scheduler will continue to use the previously published version.

Learn more about artifacts here.

Add a schedule

There are two ways to add a schedule.

From the project:

From the Your projects menu, select your project.
Navigate to the Schedules tab.
Click Add schedule.
Complete the properties on the Create a new schedule screen, described below, and then click Create.

From the pipeline:

On the pipeline canvas, click Add schedule in the upper-right.
In the Add schedule pop-up, click Schedule.
Complete the properties on the Create a new schedule screen, described below, and then click Create.

Note

Schedules become active 2 minutes after you create them.

Create a new schedule

Complete the following properties.

Property	Description
Name	A unique, descriptive name for the schedule.
Environment	Select the environment where the pipeline you want to schedule is published. The schedule will run the pipeline using this environment's data warehouse connection and defaults. Read Add environment to learn more about environments.
Artifact	Select the artifact to which the pipeline you want to schedule belongs. This is the name you gave the version when you published. Choose Always use latest if you want the schedule to update automatically as new versions of the pipeline are published. The Always use latest setting is ideal for non-production environments but not recommended for production.
Pipeline	Select a pipeline to run in the schedule.
Agent	A working agent. This is only required if you have a Hybrid SaaS solution. To learn how to create an agent, read Create an agent.
Timezone	Select a timezone for the schedule to adhere to.
Starts on	Select the date and time that the schedule will be enabled. This is not necessarily the time of the first schedule run. For example, if your start date and time is 1st August 2024 at 5:00 PM, with a schedule set to run every day at 8:00 AM, your pipeline will first run at 8:00 AM on 2nd August 2024.
Allow concurrent schedule runs	On by default. When switched on, scheduled pipeline runs will always start, even if the previous run is still in progress. When switched off, scheduled pipeline runs will be skipped (not queued) if a previous pipeline run is still in progress.

To complete the schedule, select either the Standard or Advanced tab.

A Standard schedule is created by selecting the schedule interval from drop-down menus:

Property	Description
Repeat every	The interval between schedule runs.
Unit	The units of the Repeat every interval. Options are Day (the default), Week, Hour, or Minute.

Use these two properties together to specify the schedule interval. Depending on the Unit selected, further properties will be available to precisely specify the scheduled time, as follows:

Day: Select the Hour and Minute of the day that the schedule will run.
Week: Select the day of the week, and then the Hour and Minute of the day that the schedule will run.
Hour: Select the Minute past the hour that the schedule will run.

Using these combinations of drop-downs allows you to specify a variety of precise schedule intervals, but there may be cases where you can't find a way to express the interval you need. In these cases, you can use the Advanced tab and specify the schedule interval using a Cron expression. Any valid Cron expression can be used. Some examples are given in the following section.

Cron expression examples

Expression	Description
`0 15,30,45 * ? * *`	Every hour at minutes 15, 30 and 45.
`0 0 12 1 * ?`	Every month on the 1st, at noon.
`0 0 12 15 * ?`	Every month on the 15th, at noon.
`0 0 12 1/4 * ?`	Every 4 days starting on the 1st of the month, at noon.
`0 0 12 L * ?`	Every month on the last day of the month, at noon.
`0 0 12 LW * ?`	Every month on the last weekday, at noon.
`0 0 12 6L * ?`	Every month on the last Friday, at noon.
`0 0 12 1W * ?`	Every month on the nearest weekday to the 1st of the month, at noon.
`0 0 12 ? * 2#1`	Every month on the first Monday of the month, at noon.
`0 0 12 ? * 5#3`	Every month on the third Thursday of the month, at noon.
`0 0 12 ? JAN *`	Every day at noon in January only.
`0 0 12 ? JAN,JUN *`	Every day at noon in January and June.
`0 0 12 ? DEC *`	Every day at noon in December only.
`0 0 12 ? JAN,FEB,MAR,APR *`	Every day at noon in January, February, March and April.
`0 0 12 ? 9-12 *`	Every day at noon between September and December.

Edit a schedule

Navigate to the Schedules tab.
Click the more button ... on the corresponding row of a schedule you want to edit.
Click Edit schedule.
Make changes to the schedule.
Click Update.

Delete a schedule

Navigate to the Schedules tab.
Click the more button ... on the corresponding row of a schedule you want to delete.
Click Delete schedule.
Click Yes, delete to confirm deletion. Otherwise, click Cancel.

Schedule metadata

The Schedules tab displays metadata about each schedule, including:

The name of the schedule.
The environment that defines the connection between your project and your cloud data warehouse.
The pipeline that the schedule will run, including its folder path.
The artifact that identifies the version of the pipeline that is run. If Always use latest is displayed, this schedule will always use the latest version of the pipeline.
The frequency of the schedule, e.g. 8:00am every day.
The status of the schedule, which is either Enabled or Disabled.

Note

If you have deleted a pipeline in Designer, remember to delete or disable any schedules that run the deleted pipeline. Otherwise, these schedules will continue to run the pipeline using the artifact.

Schedules and daylight saving time

If a schedule is set to run during the transition from standard time to daylight saving time (or the reverse), this can result in the following unexpected behaviors.

During the autumn change from daylight saving time to standard time, a task scheduled to start at 1:00 AM in the America/Los_Angeles time zone (i.e. 0 1 * * * America/Los_Angeles) will run twice because the local time shifts from 1:59 AM back to 1:00 AM. As a result, there are two points during that day when the local time is 1:00 AM, so the schedule is triggered twice.

During the spring change from standard time to daylight saving time, a task scheduled to start at 2:00 AM in the America/Los_Angeles time zone (i.e. 0 2 * * * America/Los_Angeles) will not run because the local time shifts from 1:59 AM to 3:00 AM. As a result, there is no point during that day when the local time is 2:00 AM, so the schedule is not triggered.