Highly available ActiveGate deployment on AWS using autoscaling group and prebaked AMI

[seppan]

I'm setting up an ActiveGate deployment to monitor AWS services and to act as a proxy for OneAgent Lambda extensions and OTEL instrumented containerised applications. The AG deployment should be highly available and have multiple AGs behind a load balancer so that loosing one instance does not cause problems for any of the applications and that AG updates can be made as rolling upgrades. The deployment is also going to be fully automated and hooked to a CICD pipeline that deploys changes automatically to multiple Dynatrace environments (dev/CI, Staging & Production). 

Now, typically the AG is deployed using an installer, but it would be preferred to have everything prebaked in an AMI to make deployments more reliable. Does anyone have experience in building a reusable AMI for ActiveGate? The instances launched using the AMI should be reconfigurable to work against any Dynatrace environment. Environment specific configuration would be applied at instance launch time.

I'm new to using AG and not familiar with all the configuration files etc. so I'm unsure if making such a reusable AMI is feasible or not. There's a container based deployment option available so I assume it is possible. Tbh, I'd really prefer the container based approach but due to its limitations (not possible to monitor AWS services or collect OTEL logs) it cannot be used.

Does anyone have experience in building a reusable AMI for ActiveGate? Any comments about the feasibility of this plan or real world experience would be highly appreciated. Thanks!

[radek_jasinski]

Hi @seppan 

Automating the implementation of Dynatrace ActiveGate in the CI/CD process is as possible and is not dependent on the chosen platform. What is most important is:

1. preparing the appropriate environments on the Dynatrace cluster side (e.g. Dev/Test/Prod) to which you will hook up the relevant AGs.
2. generate your own access token for each environment, which you then use in the CI/CD tool. Remember the appropriate scope.
3. the next thing is to parameterise the stages with the appropriate token and a link to the Dynatrace Cluster (SaaS) so that the installer can be downloaded.
4. In the step within the stage, you need to include the commands to download and install AG on the selected AMIs.
5. in addition, you can parameterise the installation with: https://docs.dynatrace.com/docs/setup-and-configuration/dynatrace-activegate/configuration/configure...

In my case, this is how it works on many environments. There is no problem with it 🙂

[seppan]

Hi @radek_jasinski  and thanks for the reply.

The points you mentioned are clear and I understand that this would be the typical way to install AG. However, I don't want to run installers at instance launch time, instead I want to have the AG installed in a prebaked AMI so that installation cannot fail at launch time. I.e. have as immutable infrastructure as possible with as little changes as possible during the instance lifetime.

Is this possible and would there be something specific parameters I should choose when I run the installer to build a reusable and prebaked AG AMI? And what would be the config files I need to overwrite during launch time to hook the AG with the specific Dynatrace cluster?

I've also failed to get AG installer to work with a http proxy and I can't find instructions for it. The networks where AG will be deployed do not have Internet access, but there's a proxy. 

[radek_jasinski]

Here you have instruction how to configure proxy for ActiveGate: link

If you create a machine with AG installed and later appoint a clone of it then AG should connect to Dynatrace Cluster. However, you may need to verify some parameters, but I have not done such an exercise yet. Have you tried to invoke a previously prepared AMI with AG

The most important thing is to maintain the correct access token and network pass-through to the DT server.

[seppan]

Or are you suggesting to build separate AMIs for each Dynatrace environment?

[radek_jasinski]

Generally if you want to connect the AG to a particular environment then it must be a separate instance. If you want to monitor a Test environment and you create a test environment in Dynatrace then it requires a different AG than for monitoring a production environment.

[seppan]

I know AG needs to be a separate per environment, but the question is that can you reconfigure the AG only by modifying configuration files (reusable AMI) or is it possible to choose the target environment only at installation time (AMI per environment)?

If its not possible to have a reusable AMI it makes testing the AMI harder and would require additional steps. It would be preferred to have a reusable AMI so that it can be rolled to first dev/CI, if it works there, then to Staging and finally to Production. If there's a separate AMI per environment, then its functionality would have to be tested separately per environment which would add (unnecessary) complexity to the deployment pipeline.

[radek_jasinski]

You can change the configuration as you point it to a different access token and environment ID

[radek_jasinski]

Take another look at this document: https://docs.dynatrace.com/docs/setup-and-configuration/dynatrace-activegate/configuration/configure...

[Kor01 · Accepted solution]

We use Terraform for AG deployments. We download an installer per environment we manage, and point the install script on the AGs to download the installer from there. Configuration is set in the install script that runs in userdata, so we have a repo per environment. But that environment specific installer takes care of all the little stuff.

This is working well for us in 12 environments with over 150 AGs. They're setup once per environment, and the autoupdates take it from there. They're pretty painless for the most part.

While this works well for us, I thought this was an interesting question! No real world experience with it, but I found some interesting docs which may help

The ActiveGate docs here have a list of the the default configuration settings with their default locations. Check these locations, but it seems like the main ones that would need to altered with each new environment would be:

1. Update/replace "authorization.properties" to alter the environment id, auth tokens, etc: https://docs.dynatrace.com/docs/setup-and-configuration/dynatrace-activegate/configuration/configure...
2. Update/replace "custom.properties" to address environment specific network zones and module settings. This overrides "config.properties" as well: https://docs.dynatrace.com/docs/setup-and-configuration/dynatrace-activegate/configuration/configure...

One config that I'm not sure would cause you an issue is the "cluster.properties". I can't find much information regarding this file.

Good luck!