topic Re: Unzipping a file with newest DT capability in Developer Q&A Forum

Unzipping a file with newest DT capability

y_buccellato — Fri, 17 Nov 2023 11:09:41 GMT

I currently am asked to download some trace log from SAP CPI API.

The called API gets back a log file in a .zip format.

Is it possible to download and unpack the zip in the same code snipped via some Dynatrace feature to then ingest those log data?

Regards

Re: Unzipping a file with newest DT capability

stefan_eggersto — Mon, 20 Nov 2023 07:50:17 GMT

Hi @y_buccellato ,

Yes, this is possible. Let me give you a rough outline of what you need:

access the external API from an app function: https://developer.dynatrace.com/develop/functions/
download the ZIP with fetch("https://...")
extract the info you want from the ZIP file
ingest the logs via https://developer.dynatrace.com/reference/sdks/client-classic-environment-v2/#storelog

The handling of the ZIP file is the most tricky part. I just tried zip.js and came up with the following snippet for an app function:

import { BlobReader, TextWriter, ZipReader } from "@zip.js/zip.js";

export default async function () {
  const zipFile = await fetch("https://developer.dynatrace.com/zips/gettingStarted/finalproject.zip");
  const blob = await zipFile.blob();

  const reader = new ZipReader(new BlobReader(blob));
  const entries = await reader.getEntries();
  for (const entry of entries) {
    if (entry.filename === "finalproject/package.json") {
      const textWriter = new TextWriter();
      await entry.getData(textWriter);

      console.log("This is the content of the package.json file:");
      console.log(await textWriter.getData());
    }
  }
}

Just adapt the filename check, and you should be able to extract the data you need.

Also, remember that you don't have access to the file system within an app function, so everything you do needs to be in memory.

Let me know if this helps!

Re: Unzipping a file with newest DT capability

y_buccellato — Mon, 20 Nov 2023 08:52:48 GMT

On the function documentation I'm reading that there are few limitation.

I'm particular interest in this one:

Function inputs/outputs can't be larger than 5 MB, respectively.

Suppose the first API call will give me back a 15 MB .zip file, am I already breaching the current function limit?

Thank for the previous reply, I appreciated it

Re: Unzipping a file with newest DT capability

stefan_eggersto — Mon, 20 Nov 2023 09:13:32 GMT

The input refers to the payload you can send to the function, and the output is what you return from the function.

A fetch request to an external service is not impacted by these limits.

Re: Unzipping a file with newest DT capability

y_buccellato — Mon, 20 Nov 2023 09:15:34 GMT

Thank you very much - a lot of learning and nuance (also due to language barrier) for me.

I'll do my best

Re: Unzipping a file with newest DT capability

y_buccellato — Tue, 21 Nov 2023 10:31:15 GMT

Is the javascript code you posted applicable in a workflow step?

When I select a "Run Javascript" and paste/execute the code I have a reply back

An error occurred: {"code": 541, "message": "Execution crashed.", "details": {"type": "BAD_REQUEST", "message": "<stripped secret anyhow::Error>"}}

Which is leading me to think that I need use another approach.

PS from work flow I'm able to download the first zip file but then I don't know how to reference the previous downloaded file to unzip it 🙂

Re: Unzipping a file with newest DT capability

stefan_eggersto — Tue, 21 Nov 2023 12:34:46 GMT

No, it won't work in a Workflow step. You currently can't use third-party libraries in Workflows.

However, you can create an app with this logic in an app function, and then call it from a Workflow step.

Re: Unzipping a file with newest DT capability

y_buccellato — Wed, 22 Nov 2023 07:37:11 GMT

Just an update to give you my impression: when I became aware that I didn't exactly get you the first time, I decided to start from scratch and followed the DT tutorial on some of the documentation page.

Doing it this way I better understood the fundamentals and also where an app function must be placed to execute the logic (though I don't fully understand why should I create an entire app if I only need an app Function to execute some specific logic likely to be "Take some data, manipulate it, push some data to DT as logs").
In the end I think I managed to configure the first part of the logic which was to do a Get, retrieve a zip and unzip it - definitely also thanks to your original answer - but I think I will stop here and look for an alternative solution.

In my prev. experience I was going with bash and recently I went with Power Automate and I thought I could find a similar friendly environment for my case. Here it appears I need to have a low to mid experience in the developing field which I can grow in time but I don't have immediately at my disposal.
For time and daily tasks reason I will put an hold on this but overall it was a good experience.

Thank you for your help and time,

regards