Skip to content

Streaming agent logs

The activity of your streaming agent can be logged in your cloud provider's logging service, either AWS CloudWatch or Azure Log Analytics, to help you address any issues you encounter. Analyzing the logs generated by your Streaming agent enables you to identify patterns, troubleshoot issues, and gain insights into the behavior of your deployment.

AWS CloudWatch

You can retrieve, inspect, and analyze log data using AWS CloudWatch in the AWS console.

By default, logs are kept for an indefinite time, but you can set up a retention period based on your requirements.

Prerequisites

  • You must have a valid AWS account to access and use AWS CloudWatch.
  • Your Streaming agent must be deployed in the instance for which you want to generate logs. The Streaming agent captures and forwards log data to AWS CloudWatch for monitoring and analysis.
  • Your AWS account must have the necessary IAM (Identity and Access Management) permissions to access CloudWatch and retrieve logs. At a minimum, the account should have the cloudwatch:DescribeLogGroups and cloudwatch:FilterLogEvents permissions for the specific log group and log streams you want to access.
  • Configure a log group in CloudWatch to store the logs generated by the Streaming agent.
  • Configure the log retention settings for the log group to ensure that logs are retained for the desired period.

How to generate logs in AWS

To generate logs from AWS CloudWatch for a deployed Streaming agent in an instance, follow these steps.

  1. In the AWS Management Console, navigate to the CloudWatch service from the services menu or from the search bar at the top of the console screen.
  2. In the CloudWatch dashboard, locate and click Log Groups in the left-hand navigation menu under the Logs category.

  3. In the Log Groups page, search for the log group associated with your Streaming agent's logs. The log group should have a name or pattern specific to your deployment, which you will have set when you installed the agent.

    Note

    If a log group for the Streaming agent does not already exist, create a new log group by following the process in Log groups.

  4. Click on the log group to access its details. You will see a list of log streams within that group.

  5. Identify the log stream that corresponds to the specific instance where your Streaming agent is deployed. Click on the log stream to view its logs.
  6. Within the log stream, you can locate specific logs by applying filters, searching for keywords, or adjusting the time range.

Azure Log Analytics

In the Azure console, you can retrieve, inspect, and analyze log data using the Azure Log Analytics service. Azure Log Analytics provides a centralized location for log storage, analysis, and visualization, enabling you to monitor and troubleshoot your Azure environment effectively.

Azure Log Analytics also allows you to create custom dashboards and set up alerts based on log data. These features can help you monitor and receive notifications for specific events or log conditions.

Prerequisites

  • You must have a valid Azure subscription to access and use Azure Log Analytics.
  • You must have Azure Container Instances (ACI) or an Azure Kubernetes Service (AKS) cluster where the Streaming agent is deployed in a container. The container must be running and generating logs.
  • Ensure that an Azure Log Analytics workspace has been created. This should have been created when you installed the agent.
  • You must enable container monitoring for the Azure Container Instances or Azure Kubernetes Service cluster where the Streaming agent is deployed. This ensures that the logs generated by the container are collected and forwarded to Azure Log Analytics.

How to generate logs in Azure

To generate logs from Azure Log Analytics for a deployed Streaming agent in an instance, follow these steps.

  1. In the Azure Portal, navigate to the Log Analytics service from the services menu or from the search bar at the top of the portal screen.
  2. In the Log Analytics dashboard, search for the workspace associated with your Streaming agent's logs. The workspace should have a name or pattern specific to your deployment. Click the workspace name to enter that workspace.
  3. In the Log Analytics workspace, click Activity Log. By default, you will see the Logs section where you can run queries and view log data.
  4. In the Logs section, you can write and execute queries using the Azure Monitor Query Language (KQL) to retrieve specific logs. You can filter logs based on time range, specific instance, or other relevant parameters. See the example below.

Query example

// Retrieve logs with specific table
  ContainerInstanceLog_CL
  | project TimeGenerated, Message
Where TimeGenerated and Message are fields from the ContainerInstanceLog_CL table.

This query will retrieve the timestamp (TimeGenerated) and the log message (Message) from the logs generated by container instances.

© Copyright 2024 Matillion Ltd | Legal | Privacy Policy | Disclaimer | Modern Slavery Statement