Extensions 2.0 - How to use dimension filters

[Radu]

Use case: add dimension filters to an Extension 2.0 YAML so that unwanted data can be filtered out. This can be implemented in the "dimensions" section of each data source definition (under "group" or "subgroup"). This post will use "snmp" as the data source, but the implementation is the same across all other data sources.

Generic concept and syntax:

You can add a filter to a dimension in one of two ways:

• filter: const:<filter-definition>
  • const: must be followed by a literal filter definition

  • <filter-definition> follows this syntax:

    • $eq(<str>) – check if <str> matches what you're filtering
    • $prefix(...) – must begin with …
    • $suffix(...) - must end with …
    • $contains(...) – must contain …
    • $and(<expr1>, <expr2>) – can be used to chain two or more of the above expressions with AND operator
    • $or(<expr1>, <expr2>) – can be used to chain two or more of the above expressions with OR operator
    • $not(<expr>) – can be used to negate an expression
• filter: var:<var-id>
  • var: must be followed by the ID of a defined variable (in the vars section)

Example 1 - I'm collecting a sensor.value metric (oid 1.0.0.0.0.3) from a table of sensors of my snmp device. The table exposes a dimension called sensor.type (oid 1.0.0.0.0.2) which is an integer value for the type of sensor. I know sensor.type 3 is a temperature sensor, so I only want to collect the value of this type of sensor. I will have to use filters to only select this dimension value.
YAML definition:

snmp:
  - group: metrics
    interval:
      minutes: 1
    subgroups:
      - subgroup: sensor-data
        table: true
        dimensions:
          - key: sensor.type
            value: oid:1.0.0.0.0.2
            filter: const:$eq(3)
        metrics:
          - key: sensor.value
            value: oid:1.0.0.0.0.3

 
Example 2 - The F5 device has a lot of Traffic Profiles that may not be needed for monitoring. I want users to be able to filter out Traffic Profiles that they don't want monitored. I must create a variable where users can enter syntax to filter out this data.

YAML definition:

vars:
  id: profile-name-filter
  type: pattern
  displayName: Pattern to filter which Profies are monitored

snmp:
  - group: traffic-profiles
    interval:
      minutes: 1
    subgroups:
      - subgroup: profile-metrics
        table: true
        dimensions:
          - key: profile.name
            value: oid:2.0.0.0.0.1
            filter: var:profile-name-filter
        metrics:
          - key: some.profile.metric
            value: oid:2.0.0.0.0.8

Note: the variable field in the Monitoring configuration will take the same syntax as the literal filter definition explained earlier. The users see this:

[tarjei]

So this filtering should work?

(It doesnt)

- subgroup: ResourceGroup
  wmiNamespace: "root\\mscluster"
  query: SELECT State,Name FROM MSCluster_ResourceGroup WHERE Name!='Available Storage'
  metrics:
    - key: com.kanari.extension.wmi-cl.host.clusterresourcegroup.state
      value: column:State
  dimensions:
    - key: resourcegroup.name
      value: Name
      filter: const:$contains("Clu")

[Radu]

Please remove the quotes around Clu. 

[Radu]

And also remember that value extraction in WMI always happens from Columns.

You correct dimension definition is:

- key: resourcegroup.name
  value: column:Name
  filter: const:$contains(Clu)

[tarjei]

It doesn't work.

Should I open a support case then?

[Radu]

Hi @tarjei 

After some discussions I found out that for WMI, due to performance reasons, the filtering within the dimensions definition is disabled. Instead, filtering should be applied directly within the query. You can still use variables to expose the filter as a configuration parameter.

This means that for WMI, the filtering takes WQL syntax and not the syntax I documented eariler.

The example from your extension would be:

query: SELECT State,Name FROM MSCluster_ResourceGroup WHERE Name!='Available Storage' var:resourceGroupFilter
metrics:
  - key: com.kanari.extension.wmi-cl.host.clusterresourcegroup.state
    value: column:State
dimensions:
  - key: resourcegroup.name
    value: column:Name

 Where resourceGroupFilter is a variable defined as before. Within the monitoring configuration you will have to add the additional syntax. For example, "Name starts with Clu" would be "AND Name LIKE 'Clu%'"

Could you try this approach and let me know if it fixes your issue?

[tarjei]

Ok,

Thank you! 🙂 I will try this. But you do understand my frustration with this?

Would you happen to know when the documentation for ext 2.0 will get the desired TLC?

[tarjei]

Hi!

I have now tried multiple variants of this, but it doesnt work.

[tarjei]

I tried it, but it did not work.

[Radu]

Could you try one final variation please. Change the query to

query: SELECT State,Name FROM MSCluster_ResourceGroup WHERE var:resourceGroupFilter

 And add the whole Name filter in the var field of the UI. 

[tarjei]

Hi @Radu this does not work. Is this because I have to use a specific schema version?

[Radu]

Please open a support ticket so we can troubleshoot this. The query looks correct but there may be other things happening that we cannot tell from the YAML alone.

[tarjei]

Hi!

I have an ongoing discussion for this, will make them open a support case for it. Thank you!

[tarjei]

Hi!

I see that in Dynatrace IIS Extension, there is dimension filtering. That plugin is using 1.245 schema.

Microsoft IIS | Dynatrace Hub

[tarjei]

This is the entire yaml:

name: "custom:com.kanari.extension.wmi-cl"
version: "1.0.48"
minDynatraceVersion: "1.225"
author:
  name: "Kanari AS"

vars:
  - id: resourceGroupFilter
    displayName: Resource Group Filter
    type: pattern
  - id: resourceFilter
    displayName: Resource Filter
    type: pattern
  - id: nodeFilter
    displayName: Node Filter
    type: pattern
  - id: networkinterfaceFilter
    displayName: Network Interface Filter
    type: pattern

metrics:
  - key: com.kanari.extension.wmi-cl.host.clusterresourcegroup.state
    metadata:
      displayName: Cluster ResourceGroup State
      description: State of the cluster resource group
      unit: Count
  - key: com.kanari.extension.wmi-cl.host.clusterresource.state
    metadata:
      displayName: Cluster Resource State
      description: State of the cluster resource
      unit: Count
  - key: com.kanari.extension.wmi-cl.host.clusternode.state
    metadata:
      displayName: Cluster Node State
      description: State of the cluster node
      unit: Count
  - key: com.kanari.extension.wmi-cl.host.clusternetworkinterface.state
    metadata:
      displayName: Cluster Network Interface State
      description: State of the cluster network interface
      unit: Count



wmi:
  - group: Host
    interval: 1m
    subgroups:
      - subgroup: Resource
        wmiNamespace: "root\\mscluster"
        query: SELECT State,Name FROM MSCluster_Resource
        metrics:
          - key: com.kanari.extension.wmi-cl.host.clusterresource.state
            value: column:State
        dimensions:
          - key: resource.name
            value: column:Name

      - subgroup: Node
        wmiNamespace: "root\\mscluster"
        query: SELECT State,Name FROM MSCluster_Node
        metrics:
          - key: com.kanari.extension.wmi-cl.host.clusternode.state
            value: column:State
        dimensions:
          - key: node.name
            value: column:Name

      - subgroup: ResourceGroup
        wmiNamespace: "root\\mscluster"
        query: SELECT State,Name FROM MSCluster_ResourceGroup WHERE Name!='Available Storage'
        metrics:
          - key: com.kanari.extension.wmi-cl.host.clusterresourcegroup.state
            value: column:State
        dimensions:
          - key: resourcegroup.name
            value: column:Name
            filter: var:resourceGroupFilter
      - subgroup: Node
        wmiNamespace: "root\\mscluster"
        query: SELECT State,Name FROM MSCluster_NetworkInterface
        metrics:
          - key: com.kanari.extension.wmi-cl.host.clusternetworkinterface.state
            value: column:State
        dimensions:
          - key: networkinterface.name
            value: column:Name

#vars:
#  - id: resourceGroupFilter
#    displayName: Pattern matching Resource Groups which should be ignored
#    type: pattern

dashboards:
  - path: "dashboards/windows_failover_clustering.json"

alerts:
  - path: "alerts/node_high.json"
  - path: "alerts/node_low.json"
  - path: "alerts/cluster_resource_high.json"
  - path: "alerts/cluster_resource_low.json"
  - path: "alerts/cluster_resourcegroup_high.json"
  - path: "alerts/cluster_resourcegroup_low.json"
  - path: "alerts/nic_low.json"

screens:
  - entityType: wmi:com_kanari_extension_wmi_cluster_instance
    detailsSettings:
      staticContent:
        showProblems: true
        showProperties: true
        showTags: true
        showGlobalFilter: true
        showAddTag: true
      layout:
        autoGenerate: false
        cards:
          - type: "CHART_GROUP"
            key: "host-clusterresources"
    propertiesCard:
      properties:
        - type: RELATION
          relation:
            entitySelectorTemplate: type(host), toRelationships.runsOn($(entityConditions))
            displayName: Host
    chartsCards:
      - key: "host-clusterresources"
        displayName: "Cluster Resources"
        numberOfVisibleCharts: 3
        charts:
          - displayName: "Cluster ResourceGroup State"
            metrics:
              - metricSelector: "com.kanari.extension.wmi-cl.host.clusterresourcegroup.state:SplitBy(\"resourcegroup.name\")"
          - displayName: "Cluster Resource State"
            metrics:
              - metricSelector: "com.kanari.extension.wmi-cl.host.clusterresource.state:SplitBy(\"resource.name\")"
          - displayName: "Cluster Network Interface State"
            metrics:
            - metricSelector: "com.kanari.extension.wmi-cl.host.clusternetworkinterface.state:SplitBy(\"networkinterface.name\")"
#    entitiesListCards:
#      - key: "cluster-nic-list"
#        displayName: "Network Interfaces List"
#        entitySelectorTemplate: "type(wmi:com_kanari_extension_wmi_cluster_network_interface),fromRelationships.runsOn($(entityConditions))"
#        displayCharts: false
#        displayIcons: true
#        enableDetailsExpandability: tru
topology:
  types:
    - name: wmi:com_kanari_extension_wmi_cluster_instance
      displayName: Cluster Instance
      enabled: true
      rules:
        - idPattern: wmi_cluster_instance_{dt.entity.host}
          sources:
            - sourceType: Metrics
              condition: $prefix(com.kanari.extension.wmi-cl.host)
          attributes: []
          requiredDimensions: []
          instanceNamePattern: Cluster Instance on {dt.entity.host}
        - idPattern: wmi_cluster_instance_{host}
          sources:
            - sourceType: Metrics
              condition: $prefix(com.kanari.extension.wmi-cl.host)
          attributes: [ ]
          requiredDimensions: [ ]
          instanceNamePattern: Cluster Instance on {host}
    - name: wmi:com_kanari_extension_wmi_cluster_network_interface
      displayName: Cluster Network Interface
      enabled: true
      rules:
        - idPattern: wmi_cluster_network_interface_{dt.entity.host}
          sources:
            - sourceType: Metrics
              condition: $prefix(com.kanari.extension.wmi-cl.host.clusternetworkinterface)
          attributes: [ ]
          requiredDimensions: [ ]
          instanceNamePattern: Cluster Network Interface on {dt.entity.host}

[jegron]

Hi @Radu !

I believe there might not be any documentation available on this subject currently. Would it be possible to reach out to the documentation team for further information on this?

Additionally, I'm looking to filter only interfaces specifically with the prefix '#' :

vars:
- id: if_alias
type: text
displayName: Pattern to filter capturing of Interfaces by Interface Alias

dimensions:
- key: if.alias
value: oid:1.3.6.1.2.1.31.1.1.1.18
filter: var:if_alias

I tried this syntax :

It works but I also obtain all the interfaces which do not have the if alias filled in. How can I exclude them?