Using Maia for anomaly detection
Identifying unexpected changes in pipeline behavior, such as sudden spikes in execution time, can be difficult to spot manually. Maia's anomaly detection capabilities can help you. Maia automatically analyzes past scheduled pipeline runs and visually flags any significant deviations from expected behavior.
This feature means you can detect potential issues before they become failures, saving time and reducing downtime in production environments.
What is anomaly detection?
Maia's anomaly detection uses statistical modeling to compare the most recent scheduled pipeline run against a historical trend of past scheduled runs. Maia identifies anomalies when the execution time is significantly higher or lower than expected.
To ensure accuracy, Maia requires at least 10 prior scheduled runs of the same pipeline to establish a trend.
Note
- Anomaly detection currently only applies to scheduled pipeline runs and evaluates execution time as the sole metric.
- Manual or API-triggered runs, as well as metrics such as row count or data volume, are not yet included in the analysis.
How it works
Maia performs continuous analysis using metrics that include:
- Start and end time.
- Total execution duration.
With these data, Maia calculates an expected range for runtime and sets upper and lower thresholds. If a new scheduled run falls outside these bounds, the run is visually flagged in the Pipeline Runs dashboard.
Example: a pipeline took 7.8% longer than its expected duration based on the last 10 runs.
What to look for
To view anomaly indicators, follow these steps:
- Log in to the Data Productivity Cloud.
- Click the Activity icon
to open the Activity menu. - Click Pipeline Runs.
Anomaly detection in Pipeline Runs includes the following indicators and information:
- Red arrows (up/down) highlight anomalies next to row count or duration.
- Hover over the icon for a tooltip with details:
- Example: "Unusual row volume: Typically ~100,000 rows, but only 80,000 rows processed in this run."
- Anomalies This section lists all detected anomalies.
Use cases
| Scenario | Example | Anomaly triggered by |
|---|---|---|
| Source data changes | Upstream table missing rows | Row volume lower than normal |
| Schema or filter updates | Business logic change drops valid rows | Row volume lower than normal |
| Infrastructure performance | Load times spike due to backend slowness | Execution time higher |
| Optimization or refactoring | Code change makes execution faster than usual | Execution time lower |
Limitations
- Only start time, end time, and execution duration time are currently analyzed.
- Only scheduled pipeline runs are considered.
- No alerts are triggered (such as via email notification). Anomaly detection only provides visual indicators for now.