cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

This product reached the end of support date on March 31, 2021.

NDSTART fails : cannot start RTM Probe

sandrine-extern
Advisor

Hello !

We encounter an issue since we installed an AMD on a virtual RHEL6.2.

While starting (ndstart), RTM Probe fails to start, whatever I do (restarting the VM, trying to reinstall the AMD...).

I had a look at the /var/log/adlex/rtm.log and here are the lines that show errors :

L0 2014-08-17 23:01:26 rtm.setup:855: Found proper driver: /usr/adlex/rtm/drivers/linux-2.6/linux-2.6.32-220.17.1.el6.x86_64/rtmmem_drv.ko
L0 2014-08-17 23:01:26 rtm.setup:1213: WARNING: rtmmem_drv module has not been loaded. Trying to load...
2014-08-17 23:01:26 rc.modules : Loading rtmmem_drv module
2014-08-17 23:01:26 rc.modules : module params: mem_size=2564M docoredump=1 prealloc_limit=67
insmod: error inserting '/usr/adlex/rtm/drivers/linux-2.6/linux-2.6.32-220.17.1.el6.x86_64/rtmmem_drv.ko': -1 Cannot allocate memory
2014-08-17 23:01:26 rc.modules : FATAL ERROR: Cannot load rtmmem_drv module !!!
2014-08-17 23:01:26 rc.modules : Please restart the probe to load the rtmmem_drv
L0 2014-08-17 23:01:25 rtm.setup:1241: ERROR: rtmmem_drv driver loading failed! RTMMEM driver is necessary to start rtm! Please try to reboot the system or decrease the amount of memory for RTMMEM driver.
L0 2014-08-17 23:01:27 rtm.setup:560: WARNING: Communication interface eth0 network driver not present; trying load e1000 module
L0 2014-08-17 23:01:27 rtm.setup:951: /sbin/modprobe -q e1000
L0 2014-08-17 23:01:26 rtm.setup:1241: ERROR: rtmmem_drv driver loading failed! RTMMEM driver is necessary to start rtm! Please try to reboot the system or decrease the amount of memory for RTMMEM driver.
L0 2014-08-17 23:01:28 rtm.setup:720: Prepare comm devices
L0 2014-08-17 23:01:29 rtm.setup:720: Prepare comm devices
L0 2014-08-17 23:01:29 rtm.run: rtm.setup returned with 113
L0 2014-08-17 23:01:29 rtm.run: ERROR: could not setup receive interfaces
L0 2014-08-17 23:01:31 rtm.run: rtm.setup returned with 113
L0 2014-08-17 23:01:31 rtm.run: ERROR: could not setup receive interfaces
rtm.start: ERROR: rtm.run process has not been found after 0 sec. - exit 1
rtm.start: ERROR: rtm.run process has not been found after 0 sec. - exit 1

Do you have any hint to help me?

Thanks in advance !

Sandrine

10 REPLIES 10

grzegorz_sulest
Dynatrace Helper
Dynatrace Helper

Hi Sandrine,

Seems that the driver cannot allocate memory.

Currently you have 2564M assigned for rtm.

Check the amount of memory assigned to the VM.

Either increase the memory size assigned to the VM or decrease the amount for the rtm driver.

If that won't help open a support ticket with https://support.compuwareapm.com.

Regards,

Grzegorz

sandrine-extern
Advisor

Hi Grzegorz,

Thank you for your answer!

I compared the memory configuration between 2 probes that were set recently and it is exactly the same.

I tried to reduce at maximum the RTM memory and it still fails while I have enought free memory on the machine.

I'll ask the support.

Thanks again!

Regards,

Sandrine

Hi Sandrine,

Try reducing the rtm.shm.buffer.size in the rtm.config to a lower value, something like 1500M.

Please restart and let us know if that resolves the issue.

Regards,

Harshal.

 

the custom drivers are not currently supported in a virtual machine; you will need to disable them entirely and use the native drivers

 

--- Erik

sandrine-extern
Advisor

Hi guys,

My rtm.shm.buffer.size is at 256M.

rtmmem.min.shm.packet.buf.count=1500000

rtmmem.shm.buffer.size=2564M

rtmmem.shm.docoredump=true

rtmmem.shm.packet.buffer.percentage=80

rtmmem.shm.prealloc.limit.perc=67

rtmmem.shm.returnqueues=0

Do you think I should reduce it even more ?

Erik, which drivers are you talking about?

 

My main issue here is that I haven't change much at this machine that was working pretty fine with RHEL 5.7, then I had to reinstall it and passed to a RHEL 6.2 version and it is not working anymore... 'sigh'

ulf_thorn222
Inactive

Hi Sandrine

Have a check here Using Network Interfaces with Native Drivers

Else you go to the RTMINST on the AMD and somewhere (smile) in the menues there's supposed to be a choice.

THere is a manual way also but it includings a bit of tampering with files and manual restarts and onlhy do it after having tried the above:

Check the following entries in rtm.config:

  • *driver.shm=true* and *force.native.drv=false* >>> allows the use the custom driver.
  • *driver.shm=false *and *force.native.drv=true* >>> allows the use the native driver.

So to use custom driver follow the steps below:

  1. Open /usr/adlex/config/rtm.config
  2. Find force.native.drv=true and change it to force.native.drv=false
  3. Find driver.shm=false and check it's still set to driver.shm=true
  4. Save the changes
  5. Then restart the whole box.

 

sandrine-extern
Advisor

Hi Ulf,

 

Thank you! Demba from the support gave me the answer too.

It is now working fine!

I just think weird that everything works just fine on another probe that has the "old settings" while they are just the same machines...It misteries...

 

Thank you all again!

przemyslaw_tafe
Dynatrace Pro
Dynatrace Pro

regarding this part:

2014-08-17 23:01:26 rc.modules : FATAL ERROR: Cannot load rtmmem_drv module !!!
2014-08-17 23:01:26 rc.modules : Please restart the probe to load the rtmmem_drv
L0 2014-08-17 23:01:25 rtm.setup:1241: ERROR: rtmmem_drv driver loading failed! RTMMEM driver is necessary to start rtm! Please try to reboot the system or decrease the amount of memory for RTMMEM driver.

FYI : AMD needs a continuous (non fragmented) memory block to load rtmmem_drv into it. If it cannot find one it won't load the driver. That's why the OS reboot is recommended to allow non-fragmented memory available right after OS restart to be used by the rtmmem driver before other OS services & apps use up the memory.

sandrine-extern
Advisor

Thank you for this additional information.

I rebooted the machine several times, the issue seems to have been due to the custom drivers.

Regards,

Sandrine

smunawar
Inactive

Hello,

I am currently doing upgrade for AMD 12.4.6 and encounter similar issue on RHEL6.4 kernel patch 2.6.32-573.22.1.el6.x86_64

While starting (ndstart), RTM Probe fails to start as captured.

Anyone can help, please.