# Known Issue: Kubernetes Dynatrace Operator v0.9.0 upgrade to newer version might get stuck

**URL:** <https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605>\
**Category:** Container platforms\
**Tags:** openshift, dynatrace-operator, kubernetes\
**Created:** [December 7, 2022, 11:01am UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605 "2022-12-07T11:01:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![stefan\_penner](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/stefan_penner/32/10145_2.png) [@stefan\_penner](https://community.dynatrace.com/u/stefan_penner)\
**Post date:** [December 7, 2022, 11:01am UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605/1 "2022-12-07T11:01:46Z")

</div>

## TL;DR

While upgrading&nbsp;from Dynatrace Operator v0.9.0 to a newer version (for example, v0.9.1), **after** applying the new manifests, you may need to execute an additional command&nbsp;to [force-delete “old” Dynatrace CSI Driver pods](https://www.dynatrace.com/support/help/shortlink/kubernetes-troubleshoot#stuck-pods):

```markup
kubectl delete pod -n dynatrace --selector=app.kubernetes.io/component=csi-driver,app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/version=0.9.0 --force --grace-period=0

```

## What is the Issue?

Dynatrace CSI Driver pods are deployed as DaemonSets on all nodes to provide a writeable volume storage for OneAgent and OneAgent binaries to pods. If a Dynatrace CSI driver pod is stuck in “_Terminating_”, OneAgents on this node can’t be injected anymore into any Pods which are scheduled on this node.

Starting with Dynatrace Operator v0.9.0, Dynatrace CSI driver pods can get stuck in “_Terminating_” if the CSI driver mounted volumes recently and then gets restarted. This has been fixed with Dynatrace Operator v0.9.1 ([#1252](https://github.com/Dynatrace/dynatrace-operator/pull/1252)). Unfortunately, this issue affects the upgrade from Operator v0.9.0 itself to any newer version.

## Who is impacted?

Any Kubernetes/OpenShift clusters which use Dynatrace Operator v0.9.0 in combination with Dynatrace CSI drivers and want to upgrade to a newer version (for example, v0.9.1). There is no action needed in case you configured your Dynakube(s) to use classicFullstack injection or application-only monitoring without CSI drivers.

Here’s how you can check which version you run:

```markup
kubectl get pod -l name=dynatrace-operator -n dynatrace -L app.kubernetes.io/version

```

_Note: If you don’t get a version/value when executing this command, you’re using Dynatrace Operator \< 0.6.0 and are not impacted._

## How can I fix the issue?

Kubernetes allows to force-delete pods in case pods are stuck in termination. In order to fix the upgrade issue, all CSI driver pods need to be force deleted **after** applying the new operator manifests:

Kubernetes:

```markup
kubectl delete pod -n dynatrace --selector=app.kubernetes.io/component=csi-driver,app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/version=0.9.0 --force --grace-period=0

```

OpenShift:

```markup
oc delete pod -n dynatrace --selector=app.kubernetes.io/component=csi-driver,app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/version=0.9.0 --force --grace-period=0

```

_Note: Please adapt the name of the namespace in case you don’t use the dynatrace-defaults_

## Can I still use Dynatrace Operator Version 0.9.0?

Yes, you can still use Dynatrace Operator v0.9.0 and there is no immediate need for an upgrade. There shouldn’t be any issues related to CSI driver pods unless you upgrade or uninstall the Dynatrace Operator.

For all customers who plan to upgrade from an older Dynatrace Operator version (\<v0.9), we do not recommend upgrading to v0.9.0. Please directly upgrade to v0.9.1 instead.

We hope this community post helps to raise awareness and avoid troubleshooting within the Dynatrace Operator upgrade procedure.

---

<div class="post-metadata">

**Author:** ![mizso74](https://avatars.discourse-cdn.com/v4/letter/m/f05b48/32.png) [@mizso74](https://community.dynatrace.com/u/mizso74)\
**Post date:** [December 7, 2022, 7:36pm UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605/2 "2022-12-07T19:36:07Z")

</div>

Hi&nbsp;[@stefan\_penner](https://community.dynatrace.com/user/47074),

Thank you very much to share this important information.

Best regards,

Mizső

---

<div class="post-metadata">

**Author:** ![haris](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/haris/32/38293_2.png) [@haris](https://community.dynatrace.com/u/haris)\
**Post date:** [February 3, 2023, 12:29pm UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605/3 "2023-02-03T12:29:01Z")

</div>

Hi&nbsp;[@stefan\_penner](https://community.dynatrace.com/user/47074), is it possible that this could lead to this:  
**The customresourcedefinitions “[dynakubes.dynatrace.com](http://dynakubes.dynatrace.com)” is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update**  
I have checked the yaml file and it seems like the version is specified, so I’m not sure.

PS. I have found that in some cases this was solved by using **kubectl replace** &nbsp;instead of&nbsp; **kubectl apply** , but I’m not that familiar with it so no idea if it helps. Thanks.

---

<div class="post-metadata">

**Author:** ![DanielS](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@DanielS](https://community.dynatrace.com/u/DanielS)\
**Post date:** [February 3, 2023, 3:09pm UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605/4 "2023-02-03T15:09:48Z")

</div>

Thanks [@stefan\_penner](https://community.dynatrace.com/user/47074)

---

<div class="post-metadata">

**Author:** ![stefan\_penner](https://dub1.discourse-cdn.com/dynatrace/user_avatar/community.dynatrace.com/stefan_penner/32/10145_2.png) [@stefan\_penner](https://community.dynatrace.com/u/stefan_penner)\
**Post date:** [February 8, 2023, 8:37am UTC](https://community.dynatrace.com/t/known-issue-kubernetes-dynatrace-operator-v0-9-0-upgrade-to-newer-version-might-get-stuck/37605/5 "2023-02-08T08:37:34Z")

</div>

Hi&nbsp;[@haris](https://community.dynatrace.com/user/57428)&nbsp;,

we have not seen and would not expect any error message related to the resource version of the Dynakube CRD caused by Operator 0.9.0 - so no, this should not be caused by the upgrade issue mentioned above.

In your case, it sounds like there has been an invalid version of the CRD applied. In this case, you could either use _kubectl replace_ or delete the Dynakube and re-apply it again. _Kubectl apply_ does a [three-way-merge](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-apply-calculates-differences-and-merges-changes)&nbsp;between your local file, the live kubernetes object manifest&nbsp;as well as the last-applied-configuration annotation.

Hope that helps!