Custom Extension Creator - SQL Extension - Ingesting large data into logs

[chris_cho]

Hello,

I got an initiative ingesting data from a SQL server into logs.

For this, I checked the features provided by the Custom Extension Creator, selecting and extracting SQL data.
It worked quite well, but I think that it doesn't handle pretty well large data retrieved from a server at once, or based on checkpoints, such as: iterative batches and based on the last ingested timestamp.
Considering this, I was thinking to use Dynatrace Python Extension SDK, where I intend using a field for SQL query...
But, the inconvenience with this approach would be that there is no way having a multiline field, for more complex SQL queries, considering that the monitoring configuration is stored in a JSON structure, and not in a YAML file that supports new lines. In case a complex SQL query contains comments, whatever comes after comments "--" will be ignored 🤔
I will see up to where I can get implementing the data ingestion from a SQL server, using Dynatrace Python Extension SDK.

Please, let me know if you have any other suggestions, or any workarounds.

Thanks,
Chris

[AntonPineiro]

Hi,

Do you want to ingest extension logs? Or which logs?

My first recommenation would be from OneAgent, if you have OneAgent, in inframode, installed in SQL host.

Best regards

[chris_cho]

Thank you for your reply.
It's not about SQL logs, but specific data from particular tables.

[michiel_otten]

What happens if you hard-set it in the .py file? It's not the most beautifull thing but guess it could work?

[chris_cho]

Thank you for your reply.

Hardcoded SQL statement would not be a viable and flexible option.
The main idea is to provide a generic implementation for self-serve, thus anyone could use this Dynatrace Extension based on their needs, by providing all elements extracting the SQL data and ingesting it into logs, events, or bizevents.
Another option than using one line field for SQL query, would be using external files stored into a specific location, such as Nexus folder for example, which will be uploaded and executed by the Dynatrace Extension. Work in progress....