cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Central Analysis Server Alert Engine

smunawar
Inactive

Hi,

Currently in our email we receive a lot of notification from CAS Alert engine, example as provide below.

During Microsoft Patching in our environment seems this alert more coming, kindly advise how to pinpoint the problem. Thanks

- 0 of 7844 finished sessions were not decrypted due to no private key found, 0% of finished sessions not decrypted due to incompleted SSL handshake, 10.046% of finished sessions not decrypted ot partially decrypted .

Alert generated for the time interval from (Tue Jul 28 00:43:00 SGT 2015) to (Tue Jul 28 00:47:59 SGT 2015) based on 5 amdstats file(s), To see details for AMD go to Reference 

8 REPLIES 8

brett_b2
Inactive

Hello Munawar,

 

Are you curious as to how to solve your 10.046% finished sessions not decrypted or partially decrypted or would you like to modify the alert and its threshold?

 

Regards,

Brett Barrett

Hello Brett,

Thank you for reply, In our dynatrace environments there is a principal "there is no data loss in the first place (it's apply to Dynatrace - DCRUM - Synthetic) unless you have details information to justified it". 

First, is it possible to solve the 10.046% of finished sessions not decrypted of partially decrypted? if yes, then kindly inform me what need to be done, what necessary thing need to be put in place (example: new SSL certs or new Private keys or anything).

Second, If not possible to solved the issue then we can thing of how to change the threshold, for this I'll do another research how to change it.

BR//Munawar

 

chris_v
Dynatrace Pro
Dynatrace Pro

To track SSL decryption failures, you need to use rcon on the AMD to get the details to troubleshoot.

log on to the AMD, start

rcon

and use the command

show ssldecr status

The results detail the reasons for partial or failed decryption of sessions.

eg.

        SESSIONS:
             Total number of sessions=59545 (inProgress=7 Finished=59538)
             SSL protocol version breakdown per number of sessions:
                       supported versions: ssl3.0=0 tls1.0=55914 tls1.1=0 tls1.2=0
                       unsupported versions: ssl2.0=0 other versions=0 no version info=3631
             Long handshakes=5062 Short handshakes=49856 Compressed sessions=0 SessionTkt reused=0 SessionId reused=99618
             Finished sessions decrypted with no errors=54521 (91% of all finished sessions)
             Sessions in progress decrypting with no errors=7 (100% of all sessions in progress)
             Finished sessions decrypted partially=31 (0% of all finished sessions)
                       with a packet lost during payload data exchange=31
                       with a corrupted payload data packet=0
                       with decryption failed during payload data exchange=0
                       terminated by alert during payload data exchange=0
             Finished sessions not decrypted=4946 (8% of all finished sessions)
                       with no private key found=280 (new sessions=224 reused sessions=56)
                       with a packet lost during handshake=25 (new sessions=5 reused sessions=20)
                       with a corrupted handshake packet or incorrect handshake sequence=40 (new sessions=40 reused sessions=0)
                       with decryption broken during handshake=0 (new sessions=0 reused sessions=0)
                       with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
                       with unsupported SSL feature=996 (unsupported cipher=996 server key exchange=0)
                       with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
                       with RSA decryption failed=0, RSA invocations blocked=0 (new sessions=0 reused sessions=0)
                       reused sessions with no matching master session seen before=15
                       with incomplete SSL handshake=0 (new sessions=0 reused sessions=0)
                       closed without data=23
                       with invalid 'Hello' packet client=0, server=0
                       terminated by alert during handshake=0
                       reuse errors when PMS identified with session id=86, with session ticket=0
                       session not seen from the beginning=3567
                       with other errors=0
             Supplemental Data detected, server=0 client=0

From this dump, we can see about 8% of sessions aren't decrypted properly, there are numerous reasons:

no private key (280) - easy to fix, get the private key
packets lost - not easy to fix
corrupted/incomplete handshake - not easy to fix
unsupported SSL feature - unsupported cipher (easy to fix, change server configuration)
reused sessions with no master - not easy to fix
closed without data - no fix
reuse errors - not easy to fix
session not seen from beginning - no easy to fix

A lot of the above are probably due to packets being missed by the AMD, either at the start or mid session. Fix would involve SPANs aren't oversubscribed/dropping packets, AMD isn't overloaded etc.  I know this AMD suffers packet loss (it's a virtual machine in my test lab, and is low on resources), which is the root cause for the majority of the decryption failures.

unsupported cipher, fixable.  TLS introduces DH/DHE/ECE (Diffie Hellman, Elliptic Curve) cipher suites, these are not decryptable - it's mathematically impossible, not something that can be fixed.  Reconfigure the application servers to use a RSA based cipher suite.

 

Hi Chris,

Super explanation from the expert, appreciate your help.

I know that there is a documentation, but from your details information given can pinpoint the issue and we can use this as reference for further checking in our environments, determine what is the next possible steps and refer to cross check with DCRUM documentation.

Thank you,

Munawar

smunawar
Inactive

After checking the SESSIONS in the AMD, here is the result. Any comment?

 

SSL DECRYPTION STATUS:
CONFIGURATION: Engine:openssl(thread) status:OK
Keys recognized=83 not recognized=0
SESSIONS:
Total number of sessions=89745613 (inProgress=391 Finished=89745222)
SSL protocol version breakdown per number of sessions:
supported versions: ssl3.0=0 tls1.0=88043404 tls1.1=147 tls1.2=257529
unsupported versions: ssl2.0=0 other versions=0 no version info=1444533
Long handshakes=13964648 Short handshakes=74336420 Compressed sessions=0 SessionTkt reused=0 SessionId reused=148650229
Finished sessions decrypted with no errors=88272072 (98% of all finished sessions)
Sessions in progress decrypting with no errors=391 (100% of all sessions in progress)
Finished sessions decrypted partially=913 (0% of all finished sessions)
with a packet lost during payload data exchange=909
with a corrupted payload data packet=3
with decryption failed during payload data exchange=0
terminated by alert during payload data exchange=1
Finished sessions not decrypted=1469448 (1% of all finished sessions)
with no private key found=0 (new sessions=0 reused sessions=0)
with a packet lost during handshake=119 (new sessions=119 reused sessions=0)
with a corrupted handshake packet or incorrect handshake sequence=25 (new sessions=25 reused sessions=0)
with decryption broken during handshake=37 (new sessions=37 reused sessions=0)
with unsupported SSL version=0 (ssl2.0=0 otherVersions=0)
with unsupported SSL feature=12 (unsupported cipher=12 server key exchange=0)
with compression errors=0 (unsupported compression=0, cannot decompress control records=0 data records=0)
reused sessions with no matching master session seen before=22107
with incomplete SSL handshake=2814 (new sessions=2811 reused sessions=3)
closed without data=1442818
with invalid 'Hello' packet client=0, server=0
terminated by alert during handshake=82
reuse errors when PMS identified with session id=20841, with session ticket=0
session not seen from the beginning=1434
with other errors=0
Supplemental Data detected, server=0 client=0
CERTIFICATES:
total server-certificate pairs=79
parsed properly=79 (matched=79 matching failed=0 not used=0)
parsing errors=0 (decode=0 extract=0 RSAerror=0)
RSA DECRYPTOR INTERNAL DIAGNOSTICS:
init/init errors (i=)13962416/0
finalize/finalize errors (f=)13962409/0
cancel/cancel errors (c=)3638/0
parallel curr/avg/max (p=)4294963665/70/4294967257
find key for cert init/fini/cancel/matched(f=)86/85/1/85
PMS CACHE INTERNAL DIAGNOSTICS:
entries added (a=)13964278 (asInitialized=13960867 asUninitialized=1161 withErrorCode=2250)
entries changed (c=)23650 (toInitialized=1542 toUninitialized=0 toError=22108)
entries deleted (d=)13959404
total entries in cache (n=)83180
SESSIONS ON HOLD DIAGNOSTICS:
total: 1437 max: 8 current: 0
PMS found: 168 not found: 1269

chris_v
Dynatrace Pro
Dynatrace Pro

That's actually a pretty good result (1% failures).

The majority of your failures are due to sessions being closed before any data is transmitted.

closed without data=1442818

Without knowing more, I'd say this is some behavior of the client/application that establishes a connection needlessly.

I'd also check for lost packets, the other errors you're seeing are probably all related to packets the AMD missed seeing.

with a packet lost during payload data exchange=909
with incomplete SSL handshake=2814
reused sessions with no matching master session seen before=22107
session not seen from the beginning=1434

 

smunawar
Inactive

Hi Chris,

Thanks for the comment, any further documentation that I can read for "closed without data=1442818" due the numbers is quite big or any recommended steps required to reduce the number.

As mentioned, We are trying to make our AMD as zero loss as possible.

Appreciate your guidance.

BR//Munawar

chris_v
Dynatrace Pro
Dynatrace Pro

You'll have to figure out why the client/application is opening connections that aren't used.  That's not an AMD problem.