Inject Custom Metadata into host record, without oneagentctl

[ewoelkerling]

Hi All, 

  We are monitoring an environment with a number of legacy windows servers, and are picking up details for these using the Remote Windows Host Monitoring extension via WMI.  Whilst this part works fine, in order to facilitate a number of downstream integrations we need to augment the data for the created host records with additional information (eg. Serial Number).  

  For hosts that have oneagents installed, this information is being provided on the hosts themselves and is appearing in the customHostMetadata are for each of the host entries, however this is obviously not viable for hosts that do not have oneagent installed. 

  Our intention was to implement this augmentation using an ActiveGate extension to detect these 'remote' hosts, then query WMI for the missing details and augment in Dyna, however I have searched high and low on the API and through the SDK, but unfortunately don't seem to be able to find the appropriate methods/examples to do this either way. 

  Is anyone able to offer some guidance please, or even better example code of how this can be done in this context?  

  All help would be greatly appreciated!  Thanks! 

[Julius_Loman · Accepted solution]

Since you are using the Extension 2.0, I believe the only method is to extend the existing topology model for the extension. You can do that in the Settings > Generic topology model for the wmi:com_dynatrace_extension_host-observability_instance type.

You need to add another source of data (I'd choose either events or logs, depending on which fits you best) and create appropriate attribute extraction rule. You need to send the data regularly to keep the properties "alive". So sending an event with the device name and the serial number (and other properties you also need). 

However, any future updates of this extension from the HUB will overwrite your topology modifications.

[ewoelkerling]

HI Julius - thanks for the info, I'll give it a try.  FYI - we're not necessarily tied to Extension 2.0 (apart from the Remote Windows part) - the active gate extension can use V1 if there's appropriate methods/capability  - does that change your thoughts? 

[Julius_Loman]

I'd stick with Extension 2.0 - as it definitely provides mode value along with the universal analysis screens. With Extension 1.0 it was possible to add metadata by using SmartScape API and updating the custom device,, but definitely I'd recommend to stay on Extension 2.0 version you have now.

[ewoelkerling]

Thanks Julius - very much appreciate the insight & assistance - thanks again!  

[ewoelkerling]

HI Julius - given your excellent answer before, I'd like to pose a follow on question (I didn't make it a new forum question as it relies on the context above)..  

I've tested your suggestion and it works really well and has no problems, however I've run into a downstream issue..  

In order to feed in the appropriate information (SerialNumber in this case) I setup the appropriate rules in the custom topology, then quickly created a V2 WMI extension to retrieve the serial via WMI.  Whilst the WMI extension is reporting as running OK, no metrics are coming through.  When I investigate the ActiveGate on which it's running, it's throwing a MINT error (7336][err][2023-02-01 06:33:01.965Z] [ds:wmi] [error] [EEC_CONNECTION_ERROR] HTTP error, dropping MINT packet (1 lines, 255 bytes): [HTTP Status Code = 400 - Bad Request (http://127.0.0.1:14599/mint/4962476800840004540)] EEC REST API call failed [status code=30]).  

Investigating further, I believe that it's probably due to the WMI query (SELECT * FROM Win32_BIOS) as there's nothing in this that can return a 'metric' (they're all text fields - see below for example output).  

Ultimately, the metric line that works for manual ingestion is like "com.dynatrace.extension.host-observability.info,,host.serialnumber=xyzqr,device.name=ip-10-0-0-214.ec2.internal 100" - which I believe works as it's a proper metric send.. whereas the V2 extension WMI I've defined is: 

wmi:
- group: augments
interval:
minutes: 1
dimensions:
- key: host
value: this:device.host
subgroups:
- subgroup: BIOS
query: SELECT * FROM Win32_BIOS
metrics:
- key: com.dynatrace.extension.host-observability.info
value: column:Version
dimensions:
- key: host.serialnumber
value: column:SerialNumber

WMI Output: 

SMBIOSBIOSVersion :
Manufacturer :
Name : T95 Ver. 01.11.00
SerialNumber : CND2303LMY
Version : HPQOEM - 0

Obviously a mismatch..  any suggestions on a workaround that might allow me to turn this into a proper metric send, but still have the dimension I need for augmentation? 

[Julius_Loman]

As you have chosen to use metric as the data source, then it must definitely contain a number, your Version output is not a number.
So your WMI query must return (any) numeric number.  Any chance you can do some join with another WMI table and just return some random number? It can be any random number.

[Mike_L]

You can’t join tables in WQL, so that won’t work alas. There is an easier solution though which is to use a constant for the value of the metric. You’d do that by setting the value as const:1 for example.

[ewoelkerling]

Thanks guys - although I should have mentioned, I tried that and ran into what looks like a schema constraint problem and/or a mint-metric issue:  (apologies for the lack of detail on the previous) 

For example:  If I run the following plugin configuration: 

wmi:
  - group: augments
    interval:
      minutes: 1
    dimensions:
      - key: host
        value: this:device.host
    subgroups:
      - subgroup: BIOS
        query: SELECT * FROM Win32_BIOS
        metrics:
          - key: com.dynatrace.extension.host-observability.info
            value: 100
        dimensions:
          - key: host.serialnumber
            value: column:SerialNumber

I get an error on the plugin upload step: 

raise dtcliutils.ExtensionValidationError(response.text)
dtcli.utils.ExtensionValidationError: {"error":{"code":400,"message":"Extension validation error","constraintViolations":[{"path":"$.wmi[0].subgroups[0].metrics[0].value","message":"$.wmi[0].subgroups[0].metrics[0].value: integer found, string expected","parameterLocation":"PAYLOAD_BODY","location":null}]}}

If I alter it so that the metric is quoted, but still an integer like below:

wmi:
  - group: augments
    interval:
      minutes: 1
    dimensions:
      - key: host
        value: this:device.host
    subgroups:
      - subgroup: BIOS
        query: SELECT * FROM Win32_BIOS
        metrics:
          - key: com.dynatrace.extension.host-observability.info
            value: "100"
        dimensions:
          - key: host.serialnumber
            value: column:SerialNumber

It uploads OK, but throws a different error when trying to run on the ActiveGate

Failed to assign monitoring configuration to ActiveGate. Reason: GENERIC_ERROR:Status GENERIC_ERROR (27) returned from 1 agents (10.0.179.98: Cannot gather mint metrics from query BIOS: Fault retrieving metric com.dynatrace.extension.host-observability.info: Unknown name. [status code=27] [status code=27]); 

which leaves me stuck.. 

Any other thoughts on how to get around this one?  Personally I'm not a huge fan of the schema constraint, although I can understand possible thoughts behind it.. 

[ewoelkerling]

Apologies, read Mike's post more carefully and added the const:..  but unfortunately I'm still throwing an error.. 

So running the following configuration the plugin now uploads: 

wmi:
  - group: augments
    interval:
      minutes: 1
    dimensions:
      - key: host
        value: this:device.host
    subgroups:
      - subgroup: BIOS
        query: SELECT * FROM Win32_BIOS
        metrics:
          - key: com.dynatrace.extension.host-observability.info
            value: const:100
        dimensions:
          - key: host.serialnumber
            value: column:SerialNumber

but throws the following: 

Failed to assign monitoring configuration to ActiveGate. Reason: GENERIC_ERROR:Status GENERIC_ERROR (27) returned from 1 agents (10.0.179.98: Cannot gather mint metrics from query BIOS: Fault retrieving metric com.dynatrace.extension.host-observability.info: Unknown name. [status code=27] [status code=27]);

Any ideas?   Metric definition perhaps? 

[Mike_L]

It might be that the WMI datasource forgot to implement the const function. Can you open a support ticket?

[ewoelkerling]

Can do and have..  case #130044 created.   Will update this forum as things progress.  Thanks! 

[Mike_L]

Thanks, I name dropped some R&D people in that support ticket who might be able to help out.

[ewoelkerling]

Thanks Mike - much appreciated.  Please take an extra donut out of petty cash 😁

[ewoelkerling · Accepted solution]

FYI - For anyone reading this - the CONST: modifier isn't necessary for the BIOS Serial retrieval - there are a number of hidden metrics available in the WMI branch for Win32_BIOS that can be used.  

I have addressed this issue, and am now adding the serial_number as a dimension using the following query & configuration: 

     - subgroup: bios
        wmiNamespace: root\cimv2
        query: SELECT SerialNumber, SMBiosMajorVersion FROM Win32_BIOS
        dimensions:
          - key: serial_number
            value: column:SerialNumber
        metrics:
          - key: com.dynatrace.extension.host-observability.info.bios.biosversion
            value: column:SMBiosMajorVersion
            type: gauge
            featureSet: standard

Would still be nice to get the const: element sorted, but at least there's a workaround 😀

[Ana_Kuzmenchuk]

Hi @ewoelkerling thank you very much for sharing a workaround with the Community! Much appreciated 😊 Happy you were able to find a solution!