Python Extension SDK - Why pyodbc doesn't get installed on ActiveGates Python runtime environment.

[chris_cho]

Hello,

I am using Dynatrace Python Extension SDK to create a custom extension ingesting data from a SQL server, by using pyodbc Python library. For this I have the configuration:

• development workstation - Python 3.10
• use pyodbc 5.2.0 library - this requires Python >= 3.9
• tenant ActiveGates Python 3.6

I added in the ./setup.py file the pyodbc:
install_requires=["dt-extensions-sdk", "requests", "pyodbc==5.2.0"],

Built successfully the extension distribution package using:
dt-sdk build
and uploaded it to the tenant.
I set a monitoring configuration, and running it I am getting the exception:
"Failed to assign monitoring configuration to ActiveGate. Reason:" -- no reason mentioned.

I checked the log file python3.log in ActiveGates, related to this extension and it looks like
[err] import pyodbc
[err]ModuleNotFoundError: No module named 'pyodbc'
[err]Traceback (most recent call last):
[err] File "/opt/dynatrace/remotepluginmodule/agent/res/dsruntime/python3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[err] return _run_code(code, main_globals, None,
[err] File "/opt/dynatrace/remotepluginmodule/agent/res/dsruntime/python3.10/lib/python3.10/runpy.py", line 86, in _run_code
[err] exec(code, run_globals)
. . .   . . .   . . .
I checked whether the pyodbc was installed as supposed to be in the Python runtime environment
$ cd /opt/dynatrace/remotepluginmodule/agent/res/dsruntime/python3.10/bin
$ pip show pyodbc

but nothing came up.
Not sure why the pyodbc library did not get installed in the the Python runtime environment.

Please let me know if you have any suggestions.

Thank you,
Chris

[AntonioSousa]

@chris_cho ,

It seems OK. Please check in extension.zip if the pyodbc wheel is included?

[chris_cho]

It looks like the pyodbc doesn't work as expected.

Instead, I tried pypyodbc that was distributed to ActiveGates, but it raised other exception about not finding the files for {ODBC Driver 17 for SQL Server} and {SQL Server Native Client 11.0}.

Not sure if I am getting anywhere using the Dynatrace Python Extensions SDK interacting with any SQL data (SQL Server, Oracle, MySQL, PostgreSQL, others).

[david_lopes · Accepted solution]

You can get anywhere you want, we do it for plenty of extensions.

As you can see on the screenshot that you highlighted you have pyodbc for windows, that will never work on a linux AG.

You need to either build on the same architecture or use the --extra-platform flag when building

And of course you don't really need to mess with this because we have a SQL datasource, that can connect to any database using JDBC.

[AntonioSousa]

@david_lopes .

From @chris_cho details, it's clear he has a Linux AG. But, if it were a Windows AG, it would work, right?

[david_lopes]

Yes sorry, good catch!

I forgot the word linux there, fixing it.

[chris_cho]

Unfortunately, I cannot get the Python libraries from an external repo, than the organization's repo.
For now, The only ODBC package available for me is pypyodbc, for both Windows and Linux.

[chris_cho]

Hello @david_lopes and @AntonioSousa,
Thank you both for your reply.

Indeed, I thought it will be way easier using the Python ODBC libraries, such as pyodbc and pypyodbc, but for distributing it for Linux ActiveGates is not that simple and straightforward.
Additionally, there is a need installing ODBC drivers "unixODBC", "unixODBC-devel".
Besides, connecting to MS SQL Server, pypyodbc needs to install FreeTDS (Tabular Data Stream), included in /etc/odbcinst.ini

./setup.py

I used it, but now I am getting another exception message: 
('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/usr/lib64/libtdsodbc.so' : file not found")

Perhaps I need to move FreeTDS to extras_require in the ./setup.py file.

[chris_cho]

Hello @david_lopes and @AntonioSousa,

Definitely, I would prefer an out-of-the-box solution, than reinventing the wheel.
So far, by using SQL extension, I couldn't find the way to extract a potential large data based on batches, neither on checkpoints. Are there other ways to use the SQL data sources, through JDBC, than the SQL extensions?

On the other hand, exploring Python Extensions SDK, I am a step ahead, but still not there yet.
For this stage, I created another question
Python Extension SDK fails connecting to MS SQL Server by using Windows Authentication - Linux AGs