Cron like schedule for 2.0 Extensions?

[sivart_89]

The EF 1.0 custom db queries extension allows for use of a cron like schedule for when the queries will be executed. I have only seen the ability to specific how often (every 5 minutes, 10 minutes, etc) to run the query for EF 2.0 extensions.

Does 2.0 have the ability to specify a cron like schedule? We use that pretty heavily today with the 1.0 extension. With the sunset of EF 1.0 in Oct next year I've started looking at replacing the one we use today with 2.0.

[Julius_Loman]

Please see this thread. Although documention does not (yet) reflect the availability. It's just not possible to have this configured as configurable variable of the extension configuration and must be "hardcoded" in the extension yaml.

[sivart_89]

What is referenced here is a standard interval such as every 5 minutes. I am aware of being able to change that to something else (10, 30, whatever) but that is not what I am referring to. As an example we have the 1.0 extension running a query on a cron schedule such as the below to run at a very specific time Monday - Friday. I don't see that being possible yet in the 2.0 framework.

0-5 12 * * MON-FRI

[Julius_Loman]

Yes, my fault. You can right now enable/disable extensions configuration using API. So, this can be accomplished for example by calling it from a workflow (if you are on SaaS and you have the new Dynatrace platform available) or triggering the API call remotely - for example from a cron scheduler.

[sivart_89]

Thanks for the information @Julius_Loman but I could see this still being a huge headache. If I recall correctly the query does not show up in a 'separate config object', or something you can individually manipulate. This means we would have to have a new extension for each query that has a unique schedule that cannot be put into the standard interval format.

While yes API calls are great, it just doesn't make much sense to not have this ability added to the new framework as it causes a lot more maintenance, unless I am missing something.

@Mike_L glad to see this is being brought in, thank you! 🙂

[Mike_L · Accepted solution]

This is scheduled to be added to the SQL datasource somewhere in the first half of 2024 to bridge the most important gap with the Custom DB Query extension.

[itamarmesa]

Hi Mike_L!

Are there any news on this? Thanks!

[Mike_L]

It has been postponed for now. It might be that this functionality is not present before the removal of extension framework 1.0.

[sivart_89]

I'm sad to hear this but I do appreciate the update here. I'm not quite ready to start moving everything over to the 2.0 extension I've developed but when ready I will work with the app team to adjust the scheduling.

[itamarmesa]

Is there any way to leave the admins some time between you adding the cron functionality to the 2.0 extensions in case that it is not ready in time before the framework 1.0 removal and the framework 1.0 extensions removal?

It will make our monitoring unavailable only dued to this.

[Mike_L]

I'm working on the priority of this. I'll point the product manager to your comment about unavailable monitoring.

[sivart_89]

@Mike_L just checking in on this being it has been 2 months. Is there any more definitive timeframe of when cron like scheduling will be available for this?

[Mike_L]

It's currently in active development for the SQL data source and we expect it to be there before the removal of EF1 🙂 

[ASE]

When will EF1 be removed, @Mike_L?

We need to know exactly when our custom DB query extension 1.0 will stop working.

[Mike_L]

Python extensions built in EF1 will be removed in October. More precisely ActiveGate and OneAgent 1.301.

JMX will keep working until March.

[sivart_89]

Just to make sure I am clear here, we are using the sqlOracle datasource to query Oracle databases. This upcoming feature applies to this right? Not sure if there is a separate datasource for something like MSSQL but just wanted to be clear on what we are hoping for (this new feature to work for queries we are running against Oracle databases)

[Mike_L]

It is indeed planned to work with all the SQL data sources, including Oracle.

[Mike_L]

This will be added to all SQL data sources in AG 1.299. Example YAML:

 

[sivart_89]

Great news here Mike. Thank you to all involved with this effort!

[Mike_L]

EDIT: cron support for the SQL data sources have been delayed to AG 1.301.

[sivart_89]

@Mike_L is this still planned for 1.301? I see this is our next release. I'm wanting to try it out as soon as its available.

[Mike_L]

Yes, it was completed in 1.301. 

[Vidhyasagar]

@Mike_L -- Please help to share the documentation to refer and update our extension via VSCODE with schedule instead of interval

[Mike_L]

VScode will support it as soon as you can select the 1.301 schema.

[AntonioSousa]

@Mike_L 
Will the the Custom Extensions Creator support this?

[david_lopes]

Yes

[ASE]

We noticed our ActiveGates were updated recently.

We have a custom DB 2.0 extension that we need to add cron scheduling to ASAP, but I'm not seeing the 1.301 schema mentioned above in VS Code.

Which one of these schemas should we be selecting?

All of our current 2.0 extensions are set to: minDynatraceVersion: "1.279.0"

[Mike_L]

1.301 or higher. I am not sure why you don’t have 1.301 on your server though.

[ASE]

Thank you, @Mike_L.

Are there any concerns if our extensions are not all using the same minDynatraceVersion:?

At the moment, we only want to add cron to one of our extensions.

Which means, the rest of them will remain on minDynatraceVersion: "1.279.0".

When updating schemas, is it necessary to do it to ALL extensions?

[Mike_L]

There's no concern. It just means that the one with 1.301/1.303 can only run on AG's of at least that version.

[ASE]

Perfect.

Does the time of day used in the cron expression still need to be in UTC, @Mike_L?

In Extensions 1.0, the cron schedule was set to execute using UTC time.

We need Dynatrace to execute the query at 12:00PM CST on only specific days, but we had to enter 17:00PM CST because it needed to be entered using UTC.

schedule: "0 17 3,10,17,24 * *"

[ASE]

Has anyone confirmed that cron works?

We replaced interval with schedule (using a simple cron expression) and our custom DB Extension now fails for bad JSON/YAML config. 😐

There are no JSON errors highlighted in the Dynatrace VS Code Extension.

Does the schedule need to go in a specific place in the YAML?

[Mike_L]

We have it working on our side. I can recommend to use the vscode extension and use schema version 1.300 or later. It will make sure that you put the attribute on the correct place. Be a bit careful with the format, I put the example further down as there are some specialities such as 0/5 for every 5 minutes.

[ASE]

Thank you, @Mike_L .

I'm guessing it does not like the cron expression?

Here's the cron expression we were using in Extensions 1.0:

Here's what I have in the new 2.0 extension.yaml:

schedule: "0 17 2,9,16,23 * *"

I placed this under the query like you did in your example.

Am I just missing a question mark in place of the 5th index?

This is not something that was required in Extensions 1.0:

[Mike_L]

It's indeed missing the ?, it needs to follow this format: https://www.javainuse.com/cron

[ASE]

Got it working... maybe. 😅

Still need to see if it executes the query on the days and time specified in our cron expression.

Here's the final cron expression.

I needed to replace the last * with a ?.

It would be nice if the Dynatrace VS Code Extension actually flagged my original expression as an error, as the extension's logs recorded this as an error.

Before:

schedule: "0 17 2,9,16,23 * *"

After:

schedule: "0 17 2,9,16,23 * ?"

[ASE]

Good news, @Mike_L.

The query is executing on the specified day and time, as expected.

However, it is not returning any metric data.

The query was working fine when we were executing it daily.

But since we went from interval to the new cron schedule, Dynatrace says the query is now "failing".

It's a valid query, as I can execute it over and over against our database without issue.

What's the query execution timeout threshold? How long can a query run before Dynatrace declares it failed?

[Mike_L]

You can add for example:

timeout: 120

It should be on the same indentation as the query to set the timeout to 2 minutes.

[ASE]

Could someone update the Dynatrace VS Code Extension so it provides a better example of how to use the cron schedule?

This messaging is a little confusing:

I suggest having this info box include a link to the official documentation:

https://docs.dynatrace.com/docs/ingest-from/extensions20/data-sources/sql/sql-reference#schedule

This messaging is a lot more clear:

[Mike_L]

Feel free to contribute to it here, or raise an issue. It's an open source project: https://github.com/dynatrace-extensions/dynatrace-extensions-vscode

[Nicky23]

Hi @Mike_L 

I have an issue with the metrics on my Informix dashboard not being displayed correctly because the extension I am using is still on version 1.0. According to the documentation, migrating to version 2.0 is necessary for the metrics to be displayed properly.

Is there any alternative solution or a temporary workaround to view the metrics while we migrate to version 2.0?

Thank you in advance for your assistance.

Best regards,

Nicole

[Mike_L]

You can keep the ActiveGate that runs the extension on 1.299 or before while doing the migration.

[Nicky23]

If my ActiveGate was updated before I realized the issue that Informix would no longer be monitored with extension 1.0, would it be possible to install another ActiveGate with the previous version? Also, I was informed that the Informix extension in the Hub would be available by the end of December. Do you have any information about this?