This is related to our first issue with SageMaker Metrics here.
-------------------------------------------------------------------
We recently finished setting up and testing AWS CloudWatch Metric Streams to use for our SageMaker Endpoints Metrics.
We've encountered one small issue that I hope someone can help us understand.
We created 2 AWS Metric Streams in CloudWatch for SageMaker Metrics. One was used as a POC (proof of concept) and the other as our final stream.
We also created 2 corresponding Amazon Firehose Streams that are using IAM roles with the exact same permissions. The only differences between the Firehose streams is one stream has SSE (Server-side encryption) enabled (and the other does not), and they are using 2 different destination URLs.
- Firehose Stream 1 = https://aws.cloud.dynatrace.com/
-
When we use Firehose Stream 2, the SageMaker CloudWatch metrics that live in the AWS Custom Namespace (/aws/sagemaker/Endpoints) DO NOT stream to Dynatrace. These are the CPU, Disk, and Memory Utilization metrics. The other SageMaker metrics appear to be streaming just fine.
When we use Firehose Stream 1, the SageMaker CloudWatch metrics that live in the AWS Custom Namespace (/aws/sagemaker/Endpoints) DO stream to Dynatrace. The other SageMaker metrics also appear to be streaming just fine.
Note: We are currently using AWS Metric Streams for RDS, ElastiCache, and Step Functions. None of these streams are experiencing this issue. It's only SageMaker.
Is there a technical/functional difference between these 2 destinations URLs?
And if so, which URL should we be using?
And could the SSE setting be causing this?
Thank you.
