Best practices of managing large environment

[waikeat_chan]

We might be about to manage a customer environments where the number of host is in thousands, instead of few hundreds.

Anyone has any experience or advice on how to best manage them? We definitely already has ways to automate the deployment of OneAgent at scale, the interesting thing for me now is how to slice-and-dice data after they are captured.

For example, my colleague and me are pondering upon what to do if the management zone is too many that, even the dropdown list won't be able to show all (or takes time to scroll to the bottom). We just go ahead and ask customer to type a few letter to filter out the long list of management zone? or we plan properly beforehand to make sure they fit correctly into the multiple environments that we will be creating in the DT Managed?

Another example, what would you guys do if the tags also has too many of them?

Dynatrace Monaco is definitely a thing we're studying now. But that mostly address about the configuration-management, not so much about address the data-management/data-analysis.

Any insight is highlight appreciated. Thanks!

Best Regards,

Wai Keat 

[Radoslaw_Szulgo · Accepted solution]

There are many aspects you'd need to think of - mainly performance (aka technical limitations) and enablement challenges (aka management at scale). 

For technical limitations it's the best to reach out to our Enterprise Solution Architect services - 

With them you go through a sizing exercise and plan a target deployment architecture - cluster nodes, active gates, environment distribution, management zones etc. 

Some technical limitations:

• 24 cluster nodes and 40k agents (max 20k agents in one environment); soon we will release 30 nodes and 55k agents (25k in 1 environment). 

• There is currently a total of 100.000 conditions allowed, across all management zones, rules & configs.

• Kubernetes:
  • 50 Kubernetes clusters per Dynatrace environment
  • 500 nodes per Kubernetes cluster
  • 50,000 pods per Kubernetes cluster

For management configuration and enablement at scale - definitely REST APIs are the way to go. Monaco can simplify a lot with Gitops approach. I've forked Monaco to also support user accounts provisioning, environments and in general cluster config. 

Hope this helps to start.