on - edited on by
MaciejNeumann
Summary
Out of the box, the Apache Kafka extension never exposes more than 500 entities per Apache Kafka component. However, the actual Kafka cluster might have more than those, and they are all detected. The limitation is placed on the UI, and it can be re-configured.
Problem
You might notice that the number of custom entities coming up after enabling the Apache Kafka extension is limited to 500. This doesn’t mean that there’s a limit to how many entities can be detected. It means that, for performance reasons, there needs to be a restriction on how many are going to be listed in the UI. There's an easy way to work around this, which also proves that the entities are indeed detected: you can get the Topics you’re missing if you filter the list by name (it works with a “contains” logic):
Resolution
Changing the limit
The limit is configurable, but it requires that you change the extension definition. The way to go about this is to download the full package via the public Hub, edit the extension.yaml file, then upload the new version to the cluster. The implication here is that the extension then becomes a custom one, meaning you are now responsible for its maintenance.
What you need to change is the entitiesLimit field within the entities list card of the yaml, under the corresponding entity screen. For kafka:topic, it currently looks like so:
entitiesListCards:
- key: kafkatopic_list_self
pageSize: 15
displayName: List of Kafka Topics
displayCharts: true
enableDetailsExpandability: true
numberOfVisibleCharts: 1
Notice how there’s no entry for the entity limits. That’s cause the latter uses its default value, 500. You can get the exhaustive list of possible fields and values for extensions 2.0 via the /api/v2/extensions/schemas/ API. Specifically for the entities list card, you’d have to make a request to:
https://<your_tenant>/api/v2/extensions/schemas/1.303/screen.card.entities.list.schema.json
The field we’re interested in here is this one:
"entitiesLimit": {
"description": "The limit of entities quantity fetched from the server.",
"type": "integer",
"default": 500,
"minimum": 100,
"maximum": 5000
}
It follows that if you want to maximize the size of the Topics list, you’d have to add this line to the entitiesListCards:
entitiesLimit: 5000
Entities API
Another way to list all custom entities created and monitored by the extension is to use the monitored entities api. To access all topics, use the entity selector type(kafka:topic), then select a pageSize value higher than 500. For example, if you pass the following parameters, you will get up to 1000 topics:
{
"entitySelector": "type(kafka:topic)",
"pageSize": 1000
}
What's next
If the above doesn't work for you, you should open a support ticket. Make sure to include:
- Links to the entities lists on the Dynatrace side.
- Some kind of proof from the Apache Kafka side (screenshots etc) that entities are missing from the above list.
- A link to this article and proof that you've already taken the suggested steps.
