- last edited on by
MaciejNeumann
Hi,
We have Logs and Metrics streams integration via Firehose. According to the documentation topology should be supported via https://dt-url.net/x6038p6. Also according to https://docs.dynatrace.com/docs/ingest-from/amazon-web-services/integrate-with-aws/aws-logs-ingest/l... entity linking should be supported for some of the objects for example RDS. What we noticed is that logs are not listed in the Clouds App:
From the logs app:
dt.source_entity
["CUSTOM_DEVICE-XXX","RELATIONAL_DATABASE_SERVICE-XXX"] and the clouds app this entity has id CUSTOM_DEVICE_YYY
Also if create management zone by aws account id - "display entities" button shows nothing. In the plugin release notes I saw:
changed idPattern from arn
{aws.account.id} to arnaccount::{aws.account.id}:account
I checked the arn metadata and it is the new format, so I expect that the id is extracted correctly.
I search in the documentation about topology and it saw that RELATIONAL_DATABASE_SERVICE-XXX kind ids are Classic, but didn't find information what will replace them and currently what happens with these entity linking.
Can you help me with that?
Regards, Deni
Hi Deni,
I think, you are correct in your observations. When using the Firehose integration for logs and metrics, entity linking is currently limited and there is still a distinction between the classic entity model and the new entity types used in the Cloud app.
The IDs you see like RELATIONAL_DATABASE_SERVICE-XXX come from the classic entity model. They are still created for backward compatibility (for example, in the Logs app), but the Cloud app only shows the new model (CUSTOM_DEVICE_YYY).
The idPattern change you mentioned (arnaccount::...) is part of the ongoing migration to unify IDs across AWS entities. While ARN extraction is correct, linking is not yet fully consistent across Logs, Metrics, and Cloud topology.
That’s why logs can still point to a classic entity, while the Cloud app only lists the new one. Similarly, management zones by AWS account ID won’t show entities if they are still represented in the classic model.
At the moment:
Firehose logs and metrics ingest works and provides enrichment, but topology in Clouds is only partially supported.
RDS and some other AWS services are on the path to being fully represented with new IDs, but the migration is still ongoing.
Eventually the classic IDs (like RELATIONAL_DATABASE_SERVICE-*) will be replaced by the new entity model, so linking between Logs/Clouds/Management zones will be consistent.
So in short: you’re not misconfigured – this is expected behavior due to the model transition. If you need stable filtering/zoning today, it’s best to rely on the new ARN-based IDs.
Stream logs via Amazon Data Firehose
Hi @t_pawlak ,
Thanks for your reply!
Could you please clarify what you mean by "rely on the new ARN-based IDs"?
Is there any documentation about the new entity model — including its current state and any planned release dates?
I followed exactly Stream logs via Amazon Data Firehose document. However, in the “Supported services” table, Amazon RDS is listed as supporting linking — but its logs still aren’t linked.
I understand that the RELATIONAL_DATABASE_SERVICE-* is only for backward compatibility and looking in the environment seem that the current ids are in format CUSTOM_DEVICE-*.
What’s confusing is that for the same Amazon RDS instance, I see 3 different IDs across the apps:
RELATIONAL_DATABASE_SERVICE-XXX in Logs app (ok this is for backward comparability)
CUSTOM_DEVICE-YYY in the Logs app
CUSTOM_DEVICE-ZZZ in the Clouds app
About the id pattern - I looked at the metadata of the log message for a specific RDS and Cloud apps line for the same RDS (entity.name for both is the same) - they are in the following format aws_arn: "arnrds:(zone):(account_id):db:(entity.name)" and the entity for the AWS account itself has the same format as the new one in the plugin - aws_arn: "arnaccount::(account_id):account".
Regards, Deni
Hi,
i think that
New ARN-based IDs: by this I mean the entity identifiers that are derived directly from the AWS ARN (e.g. arnrds:... or arnaccount::...). These are what the new plugin and the Cloud app use internally, and they are the most reliable basis today for management zone rules or filtering.
Documentation: unfortunately there isn’t a single public document that fully maps the “classic” entity model to the “new” one or provides a release roadmap. The transition is ongoing and that’s why you still see multiple IDs for the same RDS instance.
Featured Posts
