Hi Folks,
During a SaaS migration, we had to solve a license consumption-related challenge.
In Managed DPS, we could offer our clients the option to reduce their DEV and UAT license consumption and use the released capacity to cover more IT infrastructure.
There were several options, but we always chose to switch off the environments at the environment level, instead of, for example, switching off OneAgents on the hosts. It was the simplest solution.
Update specific environment — Dynatrace Managed Docs
Example call form the documentation:
curl -X PUT "https://myManaged.cluster.com/api/cluster/v2/environments/19a963a7-b19f-4382-964a-4df674c8eb8e?createToken=false" -H "accept: application/json; charset=utf-8" -H "Authorization: Api-Token fSRCdB7PQDSdFVANvNfSF" -H "Content-Type: application/json; charset=utf-8" -d "
{\"name\":\"MyNewTeam\",\"id\":\"19a963a7-b19f-4382-964a-4df674c8eb8e\",\"trial\":false,\"state\":\"DISABLED\",\"tags\":[\"owner:[email protected]\",\"department:finance\"],\"storage\":{\"userSessionRetention\":{\"maxLimitInDays\":90},\"sessionReplayRetention\":{\"maxLimitInDays\":35}}}For this, we used the documented Cluster API v2 Environments API calls. We created environment disable and environment enable curl requests for each environment and scheduled them using cron jobs.
The schedule looked like this:
- Switch off DEV/UAT environments at 20:00 on weekdays.
- Switch them back on at 06:30 on weekdays.
- Keep them switched off for the entire weekend.
This way, many Host Unit / GiB-hours could be freed up and used to monitor other infrastructure entities.
We did not observe any issues with this solution.
ADVANTAGE: The OneAgent recognizes when the environment/ActiveGate is in an "off" state and stops generating traffic. When the environment/ActiveGate becomes available again, the OneAgent recognizes the "on" state and resumes traffic generation.
Switch off
Switch on
There is no need to restart processes or pods because the OneAgent remains up and running the whole time, so code module injection remains active.
At the SaaS environment level, however, there is no equivalent API call. But we still had to provide a solution for our clients.
We evaluated several options. The final two candidates were:
- Switch off the ActiveGates.
- Switch off the OneAgents.
Switching off the ActiveGates
We did not experience any problems with "normal" VMs + OneAgent (Linux/Windows) after switching the Linux/Windows ActiveGates off and back on.
However, in Kubernetes environments, we encountered OneAgent injection issues after the containerized ActiveGates "started" again.
Client created solution for "swith on/off" cron jobs of AG: (remove of AG capabilities...)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: dynakube-scaler
namespace: dynatrace
rules:
- apiGroups: ["dynatrace.com"]
resources: ["dynakubes"]
verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dynakube-scaler-binding
namespace: dynatrace
subjects:
- kind: ServiceAccount
name: scaler-sa
namespace: dynatrace
roleRef:
kind: Role
name: dynakube-scaler
apiGroup: rbac.authorization.k8s.io
kind: CronJob
apiVersion: batch/v1
metadata:
name: stop-activegate
namespace: dynatrace
spec:
schedule: "0 19 * * *"
timeZone: Europe/Budapest
concurrencyPolicy: Forbid
suspend: false
jobTemplate:
spec:
template:
spec:
containers:
- name: oc
image: xxx/openshift4/ose-cli
command:
- oc
args:
- patch
- dynakube/ocpapgsb
- -n
- dynatrace
- --type=merge
- -p
- '{"spec":{"activeGate":{"capabilities":[]}}}'
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Never
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: scaler-sa
serviceAccount: scaler-sa
securityContext: {}
schedulerName: default-scheduler
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
kind: CronJob
apiVersion: batch/v1
metadata:
name: start-activegate
namespace: dynatrace
spec:
schedule: "0 18 * * *"
timeZone: Europe/Budapest
concurrencyPolicy: Forbid
suspend: false
jobTemplate:
spec:
template:
spec:
containers:
- name: oc
image: xxx/openshift4/ose-cli
command:
- oc
args:
- patch
- dynakube/ocpapgsb
- -n
- dynatrace
- --type=merge
- -p
- '{"spec":{"activeGate":{"capabilities":["routing","kubernetes-monitoring","dynatrace-api"]}}}'
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Never
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: scaler-sa
serviceAccount: scaler-sa
securityContext: {}
schedulerName: default-scheduler
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1So, switching off the OneAgents was the winner.
But how???
Of course, we used the latest Dynatrace Workflow functionality. Once again, we aimed for the simplest possible solution.
We created four workflows:
- Switch off OneAgents on the hosts at a scheduled time.
- Switch on OneAgents on the hosts at a scheduled time.
- Switch off OneAgents on the hosts on demand, for example, in emergency situations.
- Switch on OneAgents on the hosts on demand, for example, in emergency situations.
The workflow itself is very simple.
The first main step collects the host list using an API call. The second main step switches the OneAgents off or on using API calls from a script created with DT Assist.
Here is an example workflow for switch off.
First main step
host_list is a GET API call using the appropriate API token.
https://xxx.live.dynatrace.com/api/v2/entities?entitySelector=type%28%22HOST%22%29&from=now-3d now-3d can cause a logical problem. For example, you might switch on hosts that you don't actually want to switch on. You can adjust this time range according to your requirements.
Second main step
Use the results from host_list in a script and switch the OneAgents on or off on the selected hosts.
import { execution } from '@dynatrace-sdk/automation-utils';
import { credentialVaultClient } from '@dynatrace-sdk/client-classic-environment-v2';
export default async function () {
const ex = await execution();
const data = await ex.result('host_list');
const parsed = JSON.parse(data.body);
const creds = await credentialVaultClient.getCredentialsDetails({
id: 'CREDENTIALS_VAULT-7936DA7F9F7FBEBF',
});
const token = creds.token;
const url = "https://xxx.live.dynatrace.com/api/v2/";
//------functions
async function hostmodification(url, token, data) {
const res = await fetch(`${url}settings/objects`, {
method: "POST",
headers: {
"Authorization": `Api-Token ${token}`,
"Content-Type": "application/json",
"Accept": "application/json"
},
body: data
});
if (!res.ok) throw new Error(`Failed apply host modifications: ${res.status}`);
return await res.json();
}
//----------------
let body = [{"schemaId":"builtin:host.monitoring","schemaVersion":"1.4","scope":"TEST","value":{"enabled":false}}];
for (var i = 0; i < parsed.entities.length; i++) {
var counter = parsed.entities[i];
console.log(counter.entityId);
body = `[{"schemaId":"builtin:host.monitoring","schemaVersion":"1.4","scope":"${counter.entityId}","value":{"enabled":false}}]`
//console.log(JSON.stringify(body));
await hostmodification(url, token, body);
}
}DISADVANTAGE
There is one important disadvantage to this solution.
If a process on a host or a pod in a containerized environment is restarted while the OneAgent is in the "switched off" state, for example, because of process or pod maintenance, you will lose code module injection for that process or pod.
Therefore, you need to make sure that the affected process or pod is restarted after the OneAgent has been switched back on.
For example, if you have a performance test pipeline, you could add a process or pod restart step after the OneAgent has been switched back on.
In these situations, Fleet Management functionality can also help.
I hope this solution is useful for other members also.
Best regards,
János
Great write-up @Mizső , I have developed a similar, more advanced solution for SaaS using workflows that helps eliminate the restart issues to a significant degree. It is also connected to deployment events from the CI/CD pipelines, sending SDLC events on deployments. So, for example, if part of the environment has to be shut down, deployments enable Dynatrace during deployment and turn it off after deployment is finished. This ensures that the Dynatrace OneAgents are enabled during restarts. This, of course, does not resolve situations of manual restarts of processes when OneAgent is disabled.
- I have a common workflow for disabling / enabling part of the environment. Groups (scopes) are defined in lookup tables so we can define multiple parts (dev / test / preprod / uat) independently, currently by host groups and k8s clusters. It takes a few arguments on input:
- enabled (boolean) - if the workflow should enable or disable
- group (string) - name of the group to manage
- trigger (string) - trigger type
- "manual" - for manual enable/disable (triggers always)
- "schedule" - for scheduled enable/disable (does not disable if deployment is running)
- "deployment" - for enable/disable before/after deployment (does not disable after deployment is during business hours)
- check_restarts (boolean) - check for restarts
- send_restart_events (boolean) - if CUSTOM_ALERT events need to be send
- Workflow does multiple tasks
- Gets host groups to disable from Grail lookup tables
- Gets host to be disabled from these groups
- Gets k8s clusters for the groups (disable Kubernetes API ingest)
- Check if deployment is running (do not disable hosts if deployment is running, another workflow run will disable the environment after deployment finishes)
- Disable or enable hosts if conditions match
- Check for processes needed to be restarted and raise custom alerts for these processes - depending on the argument.
This workflow is triggered by several other workflows using the API (not a call other workflow type) to optimise workflow cost (otherwise there would be 7 full workflows instead of one):
Featured Posts
