cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

How do you manage nextPageKey automatically when extracting lots of data from Dynatrace?

y_buccellato
Advisor

Hello Community people!

 

more and more often I'm asked about big enough data extraction from Dynatrace.
I usually work with Dynatrace api v2 and Power Query (M) from Microsoft.

When I hit the limit for the data extracted I try to segment the api query itself in order to not have the nextPageKey cursor but sometime it is inevitable.

So my question is: how do you usally manage to extract data when nextPageKey is showed in your response load?

Are you scripting your costum solution in order to continue the extraction automatically? (without everytime re-typing the uri with the nextPageKey cursor?)

 

What other tools do you usually use in order to extract the data with Dynatrace apiv2?


Thanks in advance to everybody who will join this conversation and will contribute,

Regards

7 REPLIES 7

ChadTurner
DynaMight Guru
DynaMight Guru

we just leverage the clearing of the API and provide the key for the next page and so on. Its simple enough but im sure you could automate it with a postman script. 

-Chad

AntonioSousa
DynaMight Guru
DynaMight Guru

Same as @ChadTurner , we have done it in Linux shell, python and even MSDOS .bat scripts 😂😂😂

Have had some problems, but it was in our coding...

Antonio Sousa

Julius_Loman
DynaMight Guru
DynaMight Guru

I mostly use Python or cURL to fetch the data. But I'm really interested in how the paging should be used in the Power Query feature of Excel. 

Certified Dynatrace Master | Alanata a.s., Slovakia, Dynatrace Master Partner

Mike_L
Dynatrace Champion
Dynatrace Champion

I usually use the Python API client, that one handles the next page key automatically so I don't have to care about it 😀

https://github.com/dynatrace-oss/api-client-python

To get all hosts and even handle the too many concurrent request message automatically is as simple as:

dt = Dynatrace("environment_url", "api_token", too_many_requests_strategy=TOO_MANY_REQUESTS_WAIT )
for entity in dt.entities.list('type("HOST")', fields="properties.memoryTotal,properties.monitoringMode"):
    print(entity.entity_id, entity.display_name, entity.properties)

y_buccellato
Advisor

Thank you to everybody who answered.

I'm taking my time to go trought the process of learning, thinking, practicing. Especially thank you for pointing me to the DT git repo. direction.

ct_27
DynaMight Advisor
DynaMight Advisor

I have the same problem and have yet to find a simple solution as well.  Mike_L's link is honestly extremely helpful but I'd rather not rely on it since it doesn't support all API's and it's not logic i can apply elsewhere.  I've tried similar approaches using postman and insomnia REST Clients but they handle nextpages in a way different than DT approaches it.

 

I've coded solutions in Python and PHP but they're clunky and honestly my code is embarassing to get it working.

 

I'm posting in response because the "Next Page" thing is actually a cause sometimes for me to just not want to use the API.  I know I'll only get a partial result set and it's too time consuming to put all the JSON responses together. This is even an issue with PowerBI and Excel reporting because they too can't handle the API's NextPage thing.

 

Is DT considering any solutions for this?  Being that DT is all API First it would be helpful to the clients if the APIs can be a bit more client friendly.  (don't get me start on inconsistent FROM string defaults)

HigherEd

satit_dpm
Contributor

Hi,

 

I have created a PowerQuery to go through pagination and get the whole list. 
What you need to do is to create Function and call it recursively.

In my case, I need to query all problems from last quarter.
So I have created this function named getProblemAPIResult(), and from the code below you can see it is calling itself passing nextPageKey.

 

(api as text, headers as record, parameters as text, nextPageKey as text, currentList as list) => 
let
    apiResult = if nextPageKey = ""
    then Json.Document(Web.Contents(api & parameters, headers))
    else Json.Document(Web.Contents(api & "?nextPageKey=" & nextPageKey, headers)),
    newList = List.Combine({currentList, apiResult[problems]}),
    hasNext = try apiResult[nextPageKey],
    returnList = if hasNext[HasError] 
    then newList
    else getProblemAPIResult(api, headers, parameters, apiResult[nextPageKey], newList)
in
    returnList

 
Then the rest is easy, you call this function passing API, all necessary parameters and keep last 2 parameters empty.
Only the function calling matters, the rest are just table transformation.

let
    problems = getProblemAPIResult(APIURL & "/v2/problems", [Headers=[Accept="application/json; charset=utf-8", Authorization="Api-Token " & APIToken]], "?pageSize=500&from=" & fromTime & "&to=" & toTime, "", {}),
    #"Converted to Table" = Table.FromList(problems, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"problemId", "displayId", "title", "impactLevel", "severityLevel", "status", "affectedEntities", "impactedEntities", "rootCauseEntity", "managementZones", "entityTags", "problemFilters", "startTime", "endTime"}, {"Column1.problemId", "Column1.displayId", "Column1.title", "Column1.impactLevel", "Column1.severityLevel", "Column1.status", "Column1.affectedEntities", "Column1.impactedEntities", "Column1.rootCauseEntity", "Column1.managementZones", "Column1.entityTags", "Column1.problemFilters", "Column1.startTime", "Column1.endTime"}),
    #"Expanded Column1.managementZones" = Table.ExpandListColumn(#"Expanded Column1", "Column1.managementZones"),
    #"Expanded Column1.managementZones1" = Table.ExpandRecordColumn(#"Expanded Column1.managementZones", "Column1.managementZones", {"name"}, {"Column1.managementZones.name"}),
    #"Invoked Custom Function" = Table.AddColumn(#"Expanded Column1.managementZones1", "problemStart", each EpochToICTDateTime([Column1.startTime])),
    #"Invoked Custom Function1" = Table.AddColumn(#"Invoked Custom Function", "problemEnd", each EpochToICTDateTime([Column1.endTime])),
    #"Added Custom" = Table.AddColumn(#"Invoked Custom Function1", "Duration (min)", each ([Column1.endTime]-[Column1.startTime])/60000)
in
    #"Added Custom"

 
Hope it helps.

Regards,
Satit