Yes, the certificate and private key the AMD uses for secure communications with the CAS/RUM Console/etc can be replaced
The certificate and private key are stored in /usr/adlex/config/gate_ssl.pem on the AMD. Back up this file just in case, and replace it with the certificate and key of your choosing, then restart rtmgate for the change to take effect. Please note that for a drop in replacement without any additional changes, the private key in your new set can't be encrypted, as the current one is not encrypted.
We have documentation for configuring custom-issues SSL certificates for all of the main DC RUM components (CAS/ADS, RUM Console and AMD) as part of the DC RUM Deployment Security Guide (link for 12.4, link for DC RUM 2017).
I'm going to point out that the folder that Erik listed (/usr/adlex/config) only applies to DC RUM releases up to and including 12.4.10. For releases starting with version 12.4.11 (this includes all DC RUM 2017 releases), the folder where the key should be stored is now /var/lib/tomcats/rtmgate/conf/.